# Efficient Zero-shot Visual Search via Target and Context-aware Transformer

Zhiwei Ding1,2,*, Xuezhe Ren3,*, Erwan David4, Melissa Vo4, Gabriel
Kreiman5,6, Mengmi Zhang3
* Equal contribution
1 Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA,
2 Center for Neuroscience and Artificial Intelligence, Baylor College of
Medicine, Houston, TX, USA,
3 CFAR and I2R, Agency for Science, Technology and Research, Singapore,
4 Department of Psychology, Goethe Univerisity Frankfurt, Frankfurt, Germany,
5 Children’s Hospital, Harvard Medical School, Boston, MA, USA,
6 Center for Brains, Minds and Machines, Cambridge, MA, USA,
Address correspondence to<EMAIL_ADDRESS>
###### Abstract

Visual search is a ubiquitous challenge in natural vision, including daily
tasks such as finding a friend in a crowd or searching for a car in a parking
lot. Human rely heavily on relevant target features to perform goal-directed
visual search. Meanwhile, context is of critical importance for locating a
target object in complex scenes as it helps narrow down the search area and
makes the search process more efficient. However, few works have combined both
target and context information in visual search computational models. Here we
propose a zero-shot deep learning architecture, TCT (Target and Context-aware
Transformer), that modulates self attention in the Vision Transformer with
target and contextual relevant information to enable human-like zero-shot
visual search performance. Target modulation is computed as patch-wise local
relevance between the target and search images, whereas contextual modulation
is applied in a global fashion. We conduct visual search experiments on TCT
and other competitive visual search models on three natural scene datasets
with varying levels of difficulty. TCT demonstrates human-like performance in
terms of search efficiency and beats the SOTA models in challenging visual
search tasks. Importantly, TCT generalizes well across datasets with novel
objects without retraining or fine-tuning. Furthermore, we also introduce a
new dataset to benchmark models for invariant visual search under incongruent
contexts. TCT manages to search flexibly via target and context modulation,
even under incongruent contexts.

##  1 Introduction

When searching for toilet paper, we usually expect a bathroom scene and will
likely search through regions around the toilet or the sink for a white and
short cylinder-shaped object. If provided with a kitchen scene that is
irrelevant to our object of interest, we might feel lost about where to focus
on and simply search randomly throughout the scene; nonetheless, our
expectation of toilet paper’s visual appearance will still lead us to find it
without too much difficulty (Fig. 1). Humans rely heavily on knowledge about
the target object and the surrounding context in daily visual search tasks.
Target information provides reference of specific visual features to guide
attention during the search, whereas context information helps narrow down the
search area and guide attention towards more relevant locations. Both sources
of information work coherently to make visual search more efficient.

Figure 1: Searching for targets in semantically incongruent scenes is
inefficient. The image describes a kitchen scene photograph with a red
bounding box indicating the location of target object (toilet paper, shown
also in the bottom left corner). The image has a semantic object-scene
violation: the target object does not fit into the global meaning of the scene
(i.e. toilet paper inside a dishwasher in kitchen).

Visual search constitutes a ubiquitous challenge in daily life and localizing
a target object in a complex scene is important in many applications. Visual
search must fulfill four key properties: (1) selectivity (to distinguish the
target from distractors in a cluttered scene), (2) invariance (to localize the
target despite differences in its appearance or in cases where the target is
partially occluded), (3) efficiency (to localize the target with a few eye
movements instead of exhaustive sampling) and (4) zero-shot (to generalize to
novel target objects despite minimal or zero prior exposure to them). Given a
search image and a target object, it remains unclear how to build a
computational model that capitalizes on both target and contextual cues to
efficiently localize the target object and generalize to various search
conditions without additional training. Furthermore, investigation on the
impact of incongruent context on invariant visual search performance is
limited, largely due to the lack of such dataset. Synthetic out-of-context
images have been introduced previously [7], but those images are not
realistic.

Here we propose a zero-shot deep learning architecture, the Target and
Context-aware Transformer (TCT), that integrates both target and contextual
information to modulate self-attention via a multi-head transformer encoder as
in the Vision Transformer (ViT) [14]. Target modulation is computed as the
patch-wise local relevance between the target and search images, whereas
contextual modulation is applied in a global fashion. Unlike the existing
visual search models that learn to predict fixation sequences based on human
eye tracking data [44], our model is zero-shot where it directly transfers
knowledge from contextual reasoning in object recognition tasks to visual
search.

We test our model on three image datasets spanning various levels of
difficulty. TCT beats the SOTA models in highly difficult datasets, handling
complex scenes with clutter distractors and incongruent contexts without task-
specific fine-tuning. With built-in target and context modulation, TCT can
flexibly use both sources of information to strategize for high search
efficiency.

In summary, our key contributions include:

[model] We propose a biologically inspired search model with a hierarchical
deep architecture based on the pre-trained ViT. Without additional training,
the model can efficiently find novel target objects in cluttered natural
scenes, despite changes in the target’s color, scale, rotation or even when
the search scene contains a different exemplar belonging to the same category
as the target.

[dataset] We introduce a new image dataset to benchmark invariant visual
search under natural incongruent contexts.

[zero-shot and human-like] Our model generalizes to search for targets
efficiently in various context conditions, beats baseline models by a large
margin, and demonstrates human-like visual search performance.

##  2 Related Works

Visual search in daily life is particularly challenging since the instance of
the target object in the search scene could vary dramatically due to
unexpected transformations like rotation, scale, color, occlusion, etc.
Previously, a large body of behavioral[42, 32, 36, 37] and
neurophysiological[9, 12, 4, 38] studies have focused on _identical target_
visual search, deviating largely from the complexities of real-world scenes.
Recent works extend these efforts to establish human visual search benchmarks
in approximately naturalistic conditions, where the search scenes were highly
cluttered and the target information were provided as the text of the object
category [44] or an image instance different from the one in the search scene
[48].

###  2.1 Bottom-up Saliency Prediction

Certain parts of an image automatically attract attention due to color,
contrast, or spatiotemporal changes. Several models have been proposed to
capture such bottom-up saliency [25, 21, 16, 46, 8, 19, 45], recently using
deep convolution networks [20, 30, 27, 28]. These models do not include
information about the sought target.

###  2.2 Object Detection

Many works[15, 34] use object detection algorithms[18] to perform visual
search. Object detection models generate bounding boxes around several objects
of interest. Generally, models for object detection include a convolutional
neural network backbone with additional engineered components that use the
output of the network to produce candidate bounding boxes. Compared to human
visual search, object detection models employ brute-force strategies which are
not driven by top-down cues. Critically, object detection algorithms are not
able to generalize to search for novel objects, but rather require extensive
training with the target objects.

###  2.3 Goal-directed Invariant Visual Search

Humans rely heavily on the provided target as the task goal during visual
search; such target-dependent modulation is likely originated from the
prefrontal cortex[4, 33] and projected onto lower-level visual cortical
structures[5, 31]. There have been only a few attempts in using deep learning
models to predict human fixation sequences in visual search tasks [41, 1, 48,
44], all of which were inspired by top-down modulation in humans. [48]
developed a convolutional neural network (CNN) that processes search and
target image independently, where the attention map for visual search is
computed by convolving the top-level search and target feature representation.
While this model demonstrates high resemblance to human behavior in terms of
search efficiency and scanpath history, it solely focuses on target-based
modulation without considering other sources of information present in the
search scene such as the surrounding context. Another work[44] uses Inverse
Reinforcement Learning (IRL) to learn human scanpaths by treating each
fixation as a potential source of reward and trains a fixation sequence
generator to maximize the reward. This model, however, requires supervised
learning with human eye tracking data, which is labor-intensive and
economically costly to obtain. Additionally, since the model was trained to
perform the task using 18 pre-defined object categories, it could hardly
generalize to all visual search conditions, such as novel categories outside
of the training domain or objects without category labels.

While CNNs have been widely adopted in computer vision algorithms,
Transformer-based architectures – particularly, the Vision Transformer
(ViT)[14] – has recently outperformed CNNs in object classification.
Importantly, while ViT has not been constructed with explicit invariance
properties like CNNs (with built-in translation invariance), it demonstrates
stronger robustness against input perturbations including natural corruptions,
real-world distribution shifts, and natural adversarial examples [3] when
trained on sufficient amount of data. Such robustness against a broad range of
perturbations is essential for invariant visual search.

The proposed TCT is more biologically plausible than other search models, as
it integrates two essential properties that guide human searching behavior:
target and context. These human-inspired attention modulators are incorporated
into the pre-trained VIT architecture. Moreover, TCT does not require
additional training or fine-tuning on human eye tracking data or ground truth
labels for visual search tasks.

###  2.4 Contextual Modulation

A few psychophysics experiments have been carried out to establish human
benchmarks in in- and out-of-context object recognition and showed that
incongruent contextual cues impair recognition performance [49, 7]. Computer
vision models in object recognition can learn co-occurrence statistics between
an object’s label and visual appearance but also its label and surrounding
context[13, 39, 2]. The multi-head self-attention blocks in ViT [14] compute
long-range interactions between local image patches, by definition carrying
statistical information between the object of interest and the surrounding
context. A recent work[7] introduced a Context-aware Recognition Transformer
Network (CRTNet) that processes object and context information in two
independent streams before integrating them in subsequent decoder modules.
With an additional confidence estimator that balances the weights of the two
streams, CRTNet could recognize out-of-context objects robustly. However, we
still lack such computational models for visual search tasks.

The aforementioned IRL model [44] attempts to integrate contextual information
in visual search by including features of the target as well as other “anchor
objects” [40, 6] and background scenes to model the internal belief state. The
belief state representation is computed based on “contextual beliefs” of a
limited number of object and scene categories extracted from a panoptic
segmentation model[22, 23] and gets updated dynamically throughout the search
process. The segmentation model, however, requires additional supervised
training if the search task is to be generalized to novel object categories.
Overall, such an approach assumes that humans have an internal scene parser
that segments the search scene into object and scene classes. Our model, on
the other hand, does not make any presumption upon the mechanism of the
contextual reasoning process and does not require segmentation. Instead, TCT
transfers the contextual knowledge of the search scene obtained implicitly
during object recognition tasks to visual search tasks.

##  3 Our TCT Model

We propose a Target and Context-aware Transformer (TCT) to perform
naturalistic visual search tasks with high efficiency. TCT takes in a target
object $\mathbf{I}_{T}$ and a search scene $\mathbf{I}_{S}$ and extracts
feature representation of the target and the context independently. The
extracted features are then applied onto each individual transformer self-
attention layer $l$ in the form of target modulation $\mathbf{M}_{T,l}$ and
context modulation $\mathbf{M}_{C,l}$ to guide self-attention map
$\mathbf{SA}_{l}$. We call each of these modulated self-attention layers a
Target and Context-aware Attention Block (TCAB). The visual search task is
performed on the attention map extracted from the final TCAB.

We design the TCT architecture closely following that of the original Vision
Transformer (ViT). This setup allows us to borrow the pre-trained ViT directly
and easily re-purpose it into a visual search model without additional
training. A schematic of TCT is shown in Fig. 2. Each of the self-attention
layers in the original ViT is converted into a TCAB, while the remaining
architecture stays completely intact and thus is not be elaborated here.

Figure 2: Architecture for Target and Context-aware Transformer (TCT). TCT
takes in a target object $\mathbf{I}_{T}$ and a search scene $\mathbf{I}_{S}$
and extracts feature representation of the target and the context
independently. The extracted features are applied onto each Target and
Context-aware Attention Block (TCAB) in the form of target modulation
$\mathbf{M}_{T,l}$ and context modulation $\mathbf{M}_{C,l}$ to guide
attention and in turn produce a final attention map. The model produces
fixations by orderly selecting the maxima of the attention map. If the fixated
area overlaps with the target bounding box, the search process ends.
Otherwise, inhibition-of-return (IOR) is applied to the attention map by
reducing the activation response within the fixated area to zero permanently
in an area of pre-defined size. The process is repeated until the target is
found. Red dots denote predicted eye fixations.

###  3.1 Feature Extraction for Target

We take the Vision Transformer(ViT) pre-trained on ImageNet-21K and fine-tuned
on ImageNet-1K [11] with the official checkpoints from Google. We further
fine-tune the ViT on CIFAR-100[26] while explicitly removing the position
embeddings in order to exclude the spatial information in feature
representation that is irrelevant for visual search tasks.

The transformer reshapes an image of the target object
$\mathbf{I}_{T}\in\mathbb{R}^{C\times W\times H}$ into a sequence of image
patches $\mathbf{p}_{T}\in\mathbb{R}^{C\times N\times P^{2}}$, where $H$ and
$W$ is the resolution of the original image, $C$ is the number of channels,
$N$ is the number of patches given a pre-defined patch size $P\times P$. The
patches are then converted into a sequence of patch embeddings
$\mathbf{h}_{T,1}\in\mathbb{R}^{N\times D}$, where $D$ is the size of the
hidden states in each transformer layer. In each layer $l$, the hidden states
$\mathbf{h}_{T,l}$ are mapped through a linear projection $\mathbf{U}_{QKV,l}$
into query $\mathbf{Q}_{T,l}$, key $\mathbf{K}_{T,l}$, and value
$\mathbf{V}_{T,l}$ representations,

$[\mathbf{Q}_{T,l},\mathbf{K}_{T,l},\mathbf{V}_{T,l}]=\mathbf{h}_{T}\mathbf{U}_{QKV,l}$
(1)

each one of which is then reshaped into $N\times D_{h}$ for computing
multihead self-attention, where $D_{h}$ is the hidden state size of each
attention head. We retain the target query from each layer $\mathbf{Q}_{T,l}$
to represent the target feature and apply it in the corresponding layer to
modulate attention when passing the search image $\mathbf{I}_{S}$ through the
transformer (Sec. 3.3).

###  3.2 Feature Extraction for Context

We extract contextual feature representations from a modified version of the
CRTNet [7], an object classification transformer that integrates object
information and contextual reasoning with cross-attention mechanisms. The
original CRTNet architecture remains intact except that the confidence
weighting system is pruned away, enforcing the network to learn more context
information that benefits object recognition. For each pair of context image
$\mathbf{I}_{S}$ (i.e. the search image in visual search settings) and target
image $\mathbf{I}_{T}$, the modified CRTNet extracts context feature
embeddings from a stack of transformer decoder layers. Subsequently, a context
classifier, which consists of fully-connected layers and a softmax layer,
takes the feature embedding from the last decoder layer and outputs the
predicted class label probabilities $\mathbf{y}_{T,C}$ among $\mathbf{C}$
object classes. The modified CRTNet is trained end-to-end using cross-entropy
loss with respect to $\mathbf{y}_{T,C}$.

To further boost the contextual reasoning ability during object recognition,
we introduced two changes for CRTNet training: (1) We expand the COCO-18
training dataset (Sec. 4.1) with additional images from corresponding 18
object categories in MSCOCO[29], generating a total of 201,383 target-context
image pairs. (2) The original CRTNet was trained on target-context image pairs
where the target image was simply cropped out from the context image. We
perform 4 augmentations on the target-context pairs, combining 2 augmentations
on target image $I_{T}$ and 2 augmentations on context image $I_{S}$.
Augmentations on target images $I_{T}$ include: (1) the original target image
$I_{T}$ is replaced with a random target image belonging to the same target
category; (2) the original target image $I_{T}$ is blacked out with pixel
value 0. Augmentations on context images $I_{S}$ include: (1) the target image
$I_{T}$ within the original context image $I_{S}$ is cropped out and replaced
with a random target belonging to the same category as $I_{T}$; and (2) the
target image $I_{T}$ within the original context image $I_{S}$ is blacked out
with pixel value 0.

To explicitly test whether the contextual feature extracted from the modified
CRTNet contains useful information for object recognition, we check the top-1
recognition accuracy for the augmented target-context pairs where both the
target itself and the target within the context images are blacked out (Tab.
S1). Although the average accuracy for such inputs is much lower compared to
the original intact target-context pairs (85% vs 99%), it is still
significantly higher than chance level ($1/18=5.5\%$). This suggests that the
surrounding context by itself carries statistical information that enables
CRTNet to make reasonable inferences about the target object identity. Here we
transfer such useful contextual information directly to our TCT model to aid
visual search tasks.

###  3.3 TCTB

The transformer first takes in a search scene $\mathbf{I}_{S}$ and extracts
its query $\mathbf{Q}_{S,l}$, key $\mathbf{K}_{S,l}$, and value
$\mathbf{V}_{S,l}$ representations similarly as for the target image (Sec.
3.1, Eqn. 1). Within each TCAB, the unmodulated self-attention
$\mathbf{SA}_{l}$ is computed as in the original ViT:

$\mathbf{SA}_{l}=\operatorname{softmax}\Bigl{(}\mathbf{Q}_{S,l}\mathbf{K}_{S,l}^{\intercal}/\sqrt{D_{h}}\Bigr{)}.$
(2)

Intuitively, target modulation is computed as patch-wise local relevance
between the target and search image, whereas contextual modulation is applied
onto the attention output in a global fashion. For target modulation, we aim
to locate the search patches that contain similar visual features as the
target patches. This is implemented by first computing the cross attention
between the target query $\mathbf{Q}_{T,l}$ and the search query
$\mathbf{Q}_{S,l}$ and then voting for the most relevant search patch for each
target patch $\mathbf{p}_{T}$. Voting is implemented as assigning the top one
most relevant search patch with relevance $1$ and all others with relevance
$0$ (denoted as $\operatorname{TOP1}(\cdot)$).

$\mathbf{M}_{T,l}=\operatorname{TOP1}\biggl{(}\operatorname{softmax}\Bigl{(}\mathbf{Q}_{T,l}\mathbf{Q}_{S,l}^{\intercal}/\sqrt{D_{h}}\Bigr{)}\biggr{)}$
(3)

The resultant binary target-search relevance map serves as our target-driven
attention modulator $\mathbf{M}_{T,l}$, which is applied onto the original
search image self-attention weights $\mathbf{SA}_{l}$ in an element-wise
fashion (denoted as $\odot$) to achieve target-modulated attention weights
$\mathbf{A}_{T,l}$.

$\mathbf{A}_{T,l}=\mathbf{M}_{T,l}\odot\mathbf{SA}_{l}$ (4)

The attention output computed based on target-modulated attention weights
$\mathbf{A}_{T,l}$ and the search value $\mathbf{V}_{S,l}$ is then further
modulated by the contextual modulator $\mathbf{M}_{C,l}$ (detailed in Sec.
3.2) to achieve the updated hidden states $\mathbf{h}_{S,l+1}$ that serve as
the input for the next TCAB.

$\mathbf{h}_{S,l+1}=(\mathbf{M}_{C,l}+\mathbf{1})\odot\left(\mathbf{h}_{S,l}+\mathbf{A}_{T,l}\mathbf{V}_{S,l}\right)$
(5)

While both target and contextual modulation happen within each TCAB, their
influence propagates through the entire multi-block hierarchy of the
transformer. Importantly, the class token is exempt from the target and
contextual modulation operations within each block but does inherit the
modulation effects from previous blocks. We use the class token averaged
across all attention heads from the last block as the final attention map to
perform visual search tasks.

###  3.4 Fixation Sequence Generation

The ground truth location is defined as the bounding box of the target object,
i.e., the smallest rectangle encompassing all pixels of the object. A fixation
sequence is generated by iteratively selecting the maximal point in the
attention map as the fixation center. For each fixation, we check whether the
fixated area – a rectangle of pre-defined size centered at the fixation center
– overlaps with the target bounding box. If so, the search process ends;
otherwise, inhibition-of-return (IOR)[24] is applied to the attention map by
reducing the activation response within the fixated area to zero. This
reduction is permanent, in other words, infinite memory is assumed for
inhibition of return. The search continues until the target is found. The
oracle approach of judging whether the target has been located is a
simplification; in the future, the model could include a recognition module to
check whether that target can be detected at a certain fixation [48].

###  3.5 Implementation Details and Code Availability

Both target and contextual modulation in TCT are implemented on individual
layers of the ViT and therefore it is easy to investigate the contribution of
each type of modulation in different combinations of layer configurations (see
layer ablation results in Sec. 5.4). TCT achieves highest search efficiency
when target modulation is implemented at $l\in[6,12]$ and context modulation
solely at $l=3$. The target or context modulator ($\mathbf{M}_{T,l}$ or
$\mathbf{M}_{C,l}$) is set to a matrix of all ones $\mathbf{1}^{N_{S}\times
N_{S}}$ or $\mathbf{1}^{N_{S}\times D}$, respectively, if no such modulation
is implemented at the corresponding layer $l$, where ${N_{S}}$ is number of
patches in $I_{S}$ and $D$ is the hidden state size. We perform the visual
search task using search image resolution and IOR size consistent with
previous works that introduced the image datasets [44, 48]: $512\times 320$
pixels as search image and $48\times 48$ pixels as IOR for the COCO-18 and
SCEGRAM datasets, and $1280\times 1024$ as search image and $200\times 200$ as
IOR for the NatClutter dataset, respectively (Sec. 4.1). All the source code
is available at here.

(a)

(b)

(c)

(d)

Figure 3: Cumulative Performance of TCT and other baseline models on COCO-18
(a), NatClutter (b), SEAGRAM (c). Cumulative search performance is calculated
as the cumulative probability $p(n)$ to find the target in $\leq n$ fixations
for humans (red), TCT (blue) and other competitive methods (see Sec. 4.2).
Human fixation data is not available in the SCEGRAM dataset.

##  4 Experiments

###  4.1 Datasets

Attributes | # Subjects | # Trials | Target Ratio | Incongruency
---|---|---|---|---
COCO-18[44] | 10 | 612 | 0.04 | No
NatClutter[48] | 15 | 240 | 0.01 | No
SCEGRAM[35] | N/A | 187 | 0.01 | Yes
Table 1: Dataset summary. Characteristics of the three image datasets used for
performing visual search tasks: number of human subject with eye tracking data
(“# Subjects”), number of search trials (“# Trials”), average ratio between
the target and search image size (“Target ratio”), existence of search scenes
with incongruent contexts (“Incongruency”).

COCO-18[10] is a visual search dataset consisting of about 300,000 eye
movements from 10 human subjects in searching for one of 18 target object
categories in 6,202 natural indoor scenes from MSCOCO dataset [29]. The target
object is congruent with the context in each natural image. As the target
images are not provided in the original dataset, we paire each search image
with a randomly selected target image from the same object category as the one
present in the original search image.

NatClutter[48] consists of 240 invariant target-search image pairs in complex
scenes with congruent contexts. The sought target and the given target differ
in geometric transformations. The target object category could be novel to all
models of interest in the current study.

SCEGRAM[35] consists of 62 indoor target objects, each contextualized in 1
congruent search scene and 5 additional scenes with semantic and/or syntactic
violations. The target rendered within the search images and the provided
target image originate from the identical objects. To enable invariant visual
search, we collect a novel set of 62 target objects varying in appearances as
well as geometric transformations from the sought targets in the search
images. Additionally, since context modulation is more important for smaller
objects [49, 7], we sort the search images according to the ratio of target
versus search image size in ascending order and only consider the bottom 50
percentile in the current study. In the modified SCEGRAM dataset, we have 187
images in total, with 32 congruent conditions and 155 incongruent conditions.

###  4.2 Baselines and metrics

IVSN[48] is a zero-shot visual search CNN with top-down target-driven
modulation and outputs an attention map in each search trial. IRL[44]. is an
inverse reinforcement learning model trained on human fixation sequences on
COCO-18 to predict visual search scanpaths. When using IRL to perform search
on objects without explicit category labels such as in NatClutter and SCEGRAM,
we iterate through the 18 training categories to compute a category-average
attention map. Detectron2[43] has a collection of object detection algorithms.
We used ResNet+FPN pre-trained on MSCOCO [29] as the backbone for visual
search. The attention map is computed as the average of sigmoid objectness
maps extracted from the RPN component. GBVS (Graph-based Visual Saliency)[17]
is a bottom-up saliency model predicting attention maps based on low-level
statistics of the search images. DeepgazeIII[28] is a bottom-up saliency model
trained with human eye fixations in free-viewing tasks and outputs bottom-up
attention maps for search images. We also consider a Random model that
populates attention maps with values randomly drawn from a uniform
distribution. This randomly sampling process is repeated 100 times.

For all the baseline models above, we generate a sequence of fixations by
iterating through the maxima of the attention maps while applying IOR
throughout the search process (see Sec. 3.4). We evaluate all computational
models with the following evaluation metrics: Cumulative performance across
fixations is calculated as the cumulative probability $p(n)$ that a human
subjects or a computational model finds the target object in $\leq n$
fixations. We also compute Average number of fixations until the target object
has been found.

##  5 Results

Figure 4: TCT performs visual search tasks efficiently in multiple datasets
and across different context conditions. Visual search scanpaths of TCT
(column 2) and other baseline models (column 3 to 7) on COCO-18 (row 1),
NatClutter (row 2), SCEGRAM Congruent (row 3) and SCEGRAM Incongruent (row 4)
datasets. The red bounding box shows the target object. Red dots denote
predicted eye fixations. Red arrows indicate predicted eye movement
directions.

###  5.1 Visual Search on Natural Images

We first evaluate the visual search performance of TCT against the human
benchmark as well as other baseline visual search models on natural images in
the COCO-18 dataset (Fig. 3(a)). As shown in the example scanpaths in Fig. 4
row 1, TCT is able to locate the target object rapidly despite the object is
far away from the initial fixation position (the center of the image). TCT
manages to find the object using only 1 fixation for 38% of all trials and no
more than 10 fixations for 92% of all trials.

While TCT behaves worse than human in the initial fixations, its cumulative
performance increases rapidly across fixations and reaches close to human
performance within 10 fixations. IRL serves as another upper bound performance
for this task, since it was trained specifically on this dataset using human
fixation sequence to mimic human search behavior.

Among all of the zero-shot visual search models, our TCT achieves the highest
search efficiency (Fig 3(a)). In the example kitchen scene, TCT efficiently
locates the microwave with 1 fixation (Fig 4), while other baseline models
took more fixations to find the target. The outstanding performance can be
most likely attributed to its human-inspired integration of both target and
context information in guiding search attention. In contrast to TCT, IVSN is
ignorant of any contextual cues and solely relies on target information to
generate attention, leading to lower search efficiency. TCT is also notably
better than Detectron2 since the latter utilizes object detection algorithms
and is not aware of top-down modulation from the object of interest. GBVS and
DeepgazeIII also exhibit evidently lower performance, both of which are
bottom-up saliency models that do not take the target object information into
account.

###  5.2 Visual Search in Cluttered Environments

To evaluate visual search performance on more complex scenes where the target
objects are smaller in size and possibly occluded by multiple distractors, we
test TCT and other baseline models on the NatClutter dataset. Example
scanpaths in row 2 of Fig 4 demonstrate a challenging search trial with a
small teddy bear embedded in a highly cluttered scene with several
distractors. Although the target object appears much less salient than in
COCO-18, our TCT is still able to pinpoint the target position within a few
fixations. TCT successfully locates the object using only 1 fixation for 28%
of all trials and no more than 20 fixations for 92% of all trials.

While human subjects outperform TCT in the first 8 fixations, TCT surpasses
human in subsequent fixations. One reason for this difference is that humans
do not have infinite memory and potentially revisit the same fixation areas
[47], whereas permanent IOR has been applied in all computational models,
avoiding exhaustive search with repeated visits. Meanwhile, The cumulative
search performance of IRL dropped dramatically in NatClutter compared to
COCO-18, since IRL has been implemented to perform the search only within its
limited training categories within COCO-18 and therefore fails to generalize
to novel object categories or unlabelled objects. TCT manages to outperform
IRL by a large margin due to its ability to generalize to novel target objects
with minimal or even zero prior exposure.

In general, TCT performed better than other baseline models. IVSN,
particularly, achieves performance similar to TCT. Both TCT and IVSN use
target modulation and generalize well to novel target objects. Nonetheless,
the cumulative performance of IVSN was still lower than TCT possibly due to
the lack of context modulation.

###  5.3 Visual Search in Incongruent Contexts

Avg Fixations | Overall | Congruent | Incongruent
---|---|---|---
TCT(ours) | 4.2 | 3.6 | 4.9
IRL | 12.8 | 8.9 | 16.8
IVSN | 6.6 | 5.2 | 8.1
GBVS | 15.3 | 10.7 | 19.8
Detectron2 | 11.7 | 5.6 | 17.8
Deepgaze3 | 24.1 | 18.3 | 29.8
Table 2: Average number of fixations for TCT and baseline models under
congruent and incongruent conditions in the modified SCEGRAM dataset. Best is
in bold.

So far, we have focused on searching in natural scenes with contextual cues
congruent with the target objects. Next, we assess whether target-context
incongruency diminishes the search efficiency of visual search models using
the modified SCEGRAM dataset (Sec. 4.1). Search trials in SCEGRAM are split
into congruent and incongruent conditions and performance of each model is
computed for the two different conditions separately (Table 2). As
demonstrated in the example scanpaths (Fig. 4, rows 3-4), both TCT and IVSN
exhibit small performance difference across congruent and incongruent
conditions, whereas other baseline models suffer severely from incongruent
contexts. These results agree with our expectation because both TCT and IVSN
modulate attention with target information and therefore are somewhat
protected against misleading contextual information, enabling efficient search
even under incongruent conditions.

Figure 5: COCO-18 search performance for TCT with ablated components or with
modulation applied on different layers. Performance is evaluated as the
average number of fixations and the dash line indicates the best TCT
performance out of all different ablations of TCT. Left 4 columns: our default
TCT, TCT with target modulation alone (TargetAlone), TCT with context
modulation alone (ContextAlone), the original ViT with no modulation (ViT).
Mid 3 columns: TCT with target modulation applied starting from early, mid,
and late TCAB blocks onwards, respectively. Right 3 columns: our TCT with
context modulation applied at individual early, mid, and late TCAB blocks,
respectively.

###  5.4 Ablation

To further dissect individual contribution of target and contextual modulation
on search efficiency, we repeat the identical experiments on COCO-18 using
different ablated versions of TCT and compute their average number of
fixations (Fig. 5 left). When both target and context modulation are removed,
TCT becomes the original ViT (“ViT”) and performs poorly as it simply
functions as a bottom-up feature extractor and self-attention does not contain
information relevant for the visual search task. To evaluate the importance of
context modulation, we remove the global context modulation while keeping the
local target modulation intact (“TargetAlone”). The TargetAlone model is still
able to provide goal-directed information in a top-down fashion similar to its
CNN counterpart IVSN [48], yet the performance drops in comparison to TCT due
to lack of the contextual guidance. Similarly, we remove the local target
modulation while leaving the global context modulation intact
(“ContextAlone”). The ContextAlone model performs significantly worse than
TCT, indicating that target-driven modulation is essential in guiding search
attention. While context information guides attention on a global scale to
emphasize regions of interest for contextual reasoning given a target object,
by itself it does not provide information that is high resolution enough to
pinpoint the object location.

We further investigate the layer specificity of target and contextual
modulation by applying the modulation on different subsets of TCABs for each
modulation type independently. The target modulation is applied starting from
a certain block and continues across subsequent blocks. We group the blocks
where the modulation happens into early ($l\in[1,4]$), middle ($l\in[5,8]$),
and late ($l\in[9,12]$) groups. For each group, we average the search
performance. The best performance is achieved when target modulation is
applied from the middle blocks onward (Fig. 5 middle), potentially because the
early blocks in transformer mainly extract low-level nuisance visual
properties such as contrast or color that do not serve as essential cues in
visual search. Similarly, we apply the contextual modulation onto different
individual layers and group the modulation blocks in the same fashion. TCT
with contextual modulation in earlier blocks significantly outperforms than
that in later layers (Fig. 5 right), aligning with the intuition that context
functions as global guidance that redirects attention at the initial stage of
visual processing.

##  6 Conclusion

We propose a biologically plausible deep learning architecture for visual
search, TCT (Target and Context-aware Transformer). TCT integrates target
object and contextual information via a multi-head transformer encoder to
guide attention during visual search tasks. TCT transfers contextual knowledge
learnt from object recognition to visual search in a zero-shot manner. TCT
manages to approximate human search efficiency in invariant zero-shot visual
search tasks with various levels of difficulty. Remarkably, in contrast with
standard object detection approaches, TCT generalizes across tasks with novel
target object search _without retraining or fine-tuning_. Furthermore, TCT
outperforms the SOTA visual search models, especially in highly complex
datasets and across congruent and incongruent contexts in a newly introduced
dataset.

## Supplementary Materials

| Top-1 Accuracy(%)
---|---
Original Context, Original Target | 98.8
Original Context, Random Target | 98.6
Zero Context, Zero Target | 84.9
Table S1: Top-1 object recognition accuracy of the context feature extractor
on COCO-18. Classification accuracy of the context feature extractor (see Sec.
3.2) under different context conditions in COCO-18 test dataset: (1) the
target image $I_{T}$ is cropped out directly from the context image $I_{S}$
and the context image remains intact (“Original context, Original target”);
(2) the original target image $I_{T}$ is replaced with a randomly selected
target image belonging to the same target category as the original $I_{T}$,
while the context image $I_{S}$ remains intact (“Original context, Random
target”); (3) both the original target image $I_{T}$ and the bounding box
encompassing the target region on the original context image $I_{S}$ is
blacked out with pixel value 0 (“Zero context, Zero target”).

## References

  * [1] Hossein Adeli and Gregory Zelinsky.  Deep-bcn: Deep networks meet biased competition to create a brain-inspired model of attention control.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1932–1942, 2018.
  * [2] Sara Beery, Grant Van Horn, and Pietro Perona.  Recognition in terra incognita.  In Proceedings of the European conference on computer vision (ECCV), pages 456–473, 2018.
  * [3] Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit.  Understanding robustness of transformers for image classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10231–10241, 2021.
  * [4] Narcisse P Bichot, Matthew T Heard, Ellen M DeGennaro, and Robert Desimone.  A source for feature-based attention in the prefrontal cortex.  Neuron, 88(4):832–844, 2015.
  * [5] James W Bisley.  The neural basis of visual attention.  The Journal of physiology, 589(1):49–57, 2011.
  * [6] Sage EP Boettcher, Dejan Draschkow, Eric Dienhart, and Melissa L-H Võ.  Anchoring visual search in scenes: Assessing the role of anchor objects on eye movements during visual search.  Journal of vision, 18(13):11–11, 2018.
  * [7] Philipp Bomatter, Mengmi Zhang, Dimitar Karev, Spandan Madan, Claire Tseng, and Gabriel Kreiman.  When pigs fly: Contextual reasoning in synthetic and natural scenes.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 255–264, 2021.
  * [8] Neil DB Bruce and John K Tsotsos.  Saliency, attention, and visual search: An information theoretic approach.  Journal of vision, 9(3):5–5, 2009.
  * [9] Timothy J Buschman and Earl K Miller.  Serial, covert shifts of attention during visual search are reflected by the frontal eye fields and correlated with population oscillations.  Neuron, 63(3):386–396, 2009.
  * [10] Yupei Chen, Zhibo Yang, Seoyoung Ahn, Dimitris Samaras, Minh Hoai, and Gregory Zelinsky.  Coco-search18 fixation dataset for predicting goal-directed attention control.  Scientific reports, 11(1):1–11, 2021.
  * [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [12] Robert Desimone, John Duncan, et al.  Neural mechanisms of selective visual attention.  Annual review of neuroscience, 18(1):193–222, 1995.
  * [13] Santosh K Divvala, Derek Hoiem, James H Hays, Alexei A Efros, and Martial Hebert.  An empirical study of context in object detection.  In 2009 IEEE Conference on computer vision and Pattern Recognition, pages 1271–1278. IEEE, 2009.
  * [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [15] Miguel P Eckstein, Kathryn Koehler, Lauren E Welbourne, and Emre Akbas.  Humans, but not deep neural networks, often miss giant targets in scenes.  Current Biology, 27(18):2827–2832, 2017.
  * [16] Jonathan Harel, Christof Koch, and Pietro Perona.  Graph-based visual saliency.  In NIPS, pages 545–552, 2006.
  * [17] Jonathan Harel, Christof Koch, and Pietro Perona.  Graph-based visual saliency.  Advances in neural information processing systems, 19, 2006.
  * [18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [19] Xiaodi Hou, Jonathan Harel, and Christof Koch.  Image signature: Highlighting sparse salient regions.  TPAMI, 34(1):194–201, 2012.
  * [20] Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao.  Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks.  In IEEE ICCV, pages 262–270, 2015.
  * [21] Laurent Itti, Christof Koch, and Ernst Niebur.  A model of saliency-based visual attention for rapid scene analysis.  TPAMI, (11):1254–1259, 1998.
  * [22] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár.  Panoptic feature pyramid networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6399–6408, 2019.
  * [23] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár.  Panoptic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9404–9413, 2019.
  * [24] Raymond M Klein.  Inhibition of return.  Trends in cognitive sciences, 4(4):138–147, 2000.
  * [25] Christof Koch and Shimon Ullman.  Shifts in selective visual attention: towards the underlying neural circuitry.  In Matters of intelligence, pages 115–141. Springer, 1987.
  * [26] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [27] Srinivas SS Kruthiventi, Kumar Ayush, and R Venkatesh Babu.  Deepfix: A fully convolutional neural network for predicting human eye fixations.  IEEE Transactions on Image Processing, 26(9):4446–4456, 2017.
  * [28] Matthias Kümmerer, Matthias Bethge, and Thomas SA Wallis.  Deepgaze iii: Modeling free-viewing human scanpaths with deep learning.  Journal of Vision, 22(5):7–7, 2022.
  * [29] TY Lin, M Maire, S Belongie, J Hays, P Perona, D Ramanan, P Dollár, CL Zitnick, et al.  Microsoft coco: Common objects in context. ineuropean conference on computer vision 2014 sep 6 (pp. 740-755).
  * [30] Yuetan Lin, Shu Kong, Donghui Wang, and Yueting Zhuang.  Saliency detection within a deep convolutional architecture.
  * [31] Julio Martinez-Trujillo.  Searching for the neural mechanisms of feature-based attention in the primate brain.  Neuron, 70(6):1025–1028, 2011.
  * [32] Thomas Miconi, Laura Groomes, and Gabriel Kreiman.  There’s waldo! a normalization model of visual search predicts single-trial human fixations in an object search task.  Cerebral cortex, 26(7):3064–3082, 2015.
  * [33] Earl K Miller, Jonathan D Cohen, et al.  An integrative theory of prefrontal cortex function.  Annual review of neuroscience, 24(1):167–202, 2001.
  * [34] David A Nicholson and Astrid A Prinz.  Could simplified stimuli change how the brain performs visual search tasks? a deep neural network study.  Journal of Vision, 22(7):3–3, 2022.
  * [35] Sabine Öhlschläger and Melissa Le-Hoa Võ.  Scegram: An image database for semantic and syntactic inconsistencies in scenes.  Behavior research methods, 49(5):1780–1791, 2017.
  * [36] Rajesh PN Rao, Gregory J Zelinsky, Mary M Hayhoe, and Dana H Ballard.  Eye movements in iconic visual search.  Vision research, 42(11):1447–1463, 2002.
  * [37] Antonio J Rodriguez-Sanchez, Evgueni Simine, and John K Tsotsos.  Attention and visual search.  International Journal of Neural Systems, 17(04):275–288, 2007.
  * [38] David L Sheinberg and Nikos K Logothetis.  Noticing familiar objects in real world scenes: the role of temporal cortical neurons in natural vision.  Journal of Neuroscience, 21(4):1340–1350, 2001.
  * [39] Jin Sun and David W Jacobs.  Seeing what is not there: Learning context to determine where objects are missing.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5716–5724, 2017.
  * [40] Melissa Le-Hoa Võ, Sage EP Boettcher, and Dejan Draschkow.  Reading scenes: How scene grammar guides attention and aids perception in real-world environments.  Current opinion in psychology, 29:205–210, 2019.
  * [41] Zijun Wei, Hossein Adeli, Minh Hoai Nguyen, Greg Zelinsky, and Dimitris Samaras.  Learned region sparsity and diversity also predicts visual attention.  Advances in Neural Information Processing Systems, 29, 2016.
  * [42] Jeremy M Wolfe and Todd S Horowitz.  Five factors that guide attention in visual search.  Nature Human Behaviour, 1(3):1–8, 2017.
  * [43] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [44] Zhibo Yang, Lihan Huang, Yupei Chen, Zijun Wei, Seoyoung Ahn, Gregory Zelinsky, Dimitris Samaras, and Minh Hoai.  Predicting goal-directed human attention using inverse reinforcement learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 193–202, 2020.
  * [45] Jianming Zhang and Stan Sclaroff.  Saliency detection: A boolean map approach.  In Proceedings of the IEEE International Conference on Computer Vision, pages 153–160, 2013.
  * [46] Lingyun Zhang, Matthew H Tong, Tim K Marks, Honghao Shan, and Garrison W Cottrell.  Sun: A bayesian framework for saliency using natural statistics.  Journal of vision, 8(7):32–32, 2008.
  * [47] Mengmi Zhang, Marcelo Armendariz, Will Xiao, Olivia Rose, Katarina Bendtz, Margaret Livingstone, Carlos Ponce, and Gabriel Kreiman.  Look twice: A generalist model predicts return fixations across tasks and species.  PLoS Computational Biology, page In Press, 2022.
  * [48] Mengmi Zhang, Jiashi Feng, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Gabriel Kreiman.  Finding any waldo with zero-shot invariant and efficient visual search.  Nature communications, 9(1):1–15, 2018.
  * [49] Mengmi Zhang, Claire Tseng, and Gabriel Kreiman.  Putting visual object recognition in context.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12985–12994, 2020.


# unsupervised machine learning for physical concepts

Ruyu Yang  Graduate School of China Academy of Engineering Physics, Beijing
100193, China

###### Abstract

In recent years, machine learning methods have been used to assist scientists
in scientific research. Human scientific theories are based on a series of
concepts. How machine learns the concepts from experimental data will be an
important first step. We propose a hybrid method to extract interpretable
physical concepts through unsupervised machine learning. This method consists
of two stages. At first, we need to find the Betti numbers of experimental
data. Secondly, given the Betti numbers, we use a variational autoencoder
network to extract meaningful physical variables. We test our protocol on toy
models and show how it works.

##  I introduction

Physics is built on many different concepts, such as force, entropy, and
Hamiltonian. Scientists derive meaningful concepts from observations by their
ingenuity and then use formulas to connect them, constituting the theory. As a
principle, this theory is always as simple as possible. Since the beginning of
the 21st century, machine learning has developed rapidly Liu  _et al._ (2018)
and has been widely used in various fields Teichert  _et al._ (2019);
Bourilkov (2019); Mishra  _et al._ (2016), including machine-assisted
scientific research Vamathevan  _et al._ (2019); Valletta  _et al._ (2017);
Schmidt  _et al._ (2019); Carleo  _et al._ (2019). A natural question arises:
can machines propose scientific theories by themselves? It is undoubtedly a
fundamental and challenging goal. Many works have studied this from different
aspects Ha and Jeong (2021); Zhang and Lin (2018); Boullé  _et al._ (2021);
Zobeiry and Humfeld (2019); Farina  _et al._ (2020); Wu and Tegmark (2019);
Zheng  _et al._ (2018); D’Agnolo and Wulzer (2019). In general, the
establishment of the theory can be divided into two steps: identify the
critical variables from observations and connect them by formula. A technique
known as symbolic regression has been developed for the second step Udrescu
and Tegmark (2020); Udrescu  _et al._ (2020). The authors propose a network
named AI Feymann to automatically find the formula the system obeys. To
increase the success rate of symbolic regression, the critical variables
identified by the first step should be as few as possible.

In this work, we focus on the first step of establishing a theory. We suggest
using both Topological Data Analysis(TDA)Wasserman (2018);
chazal2017introduction; Murugan and Robertson (2019) and variational
autoencoder(VAE) Kingma and Welling (2019, 2013); An and Cho (2015); Khobahi
and Soltanalian (2019); Rezende  _et al._ (2014); Higgins  _et al._ (2016);
Burgess  _et al._ (2018) to extract meaningful variables from extensive
observations. TDA is a recent and fast-growing tool to analyze and exploit the
complex topological and geometric structures underlying data. This tool is
necessary for our protocol if specific structures such as circles and sphere
are in experimental data. VAE is a deep learning technique for learning latent
representations. This technique has been widely used in many problemsLuchnikov
_et al._ (2019); Cerri  _et al._ (2019) as a generative model. It can also be
seen as a manifold learning network for dimensionality reduction and
unsupervised learning tasks.

Our protocol has two stages. Firstly, we use TDA to infer relevant topological
features for experimental data. In the simplest case, where the manifold has a
low dimension, an essential feature for us is the Betti numbers, topology
invariants. Once we get the topological features, we can design the proper
architecture and loss function of VAE. As we will show later, the latent
variables of VAE need to form a manifold homeomorphic to the manifold composed
of observations. After the training of the VAE network, the latent variables
represent the key variables discovered by this machine. Thanks to the
structure of the VAE network, the latent variables, and the observations are
in one-to-one correspondence. Using the trained VAE network, one can calculate
the latent variables and the observations from each other. That means the
formula derived by symbolic regression connecting the latent variables can
predict the experimental phenomena.

We test our protocol on three toy models. They have different topological
features. The first is a classical coupled harmonic oscillator, where the
observations constitute a circle embedded in three-dimensional Euclidean
space. Another example is two balls rotating around the same center, with
different radius and angular velocities. With the other ball as the reference
system, the observations are the Cartesian coordinates of the ball. The
observations constitute a lemniscate curve. The third is a two-level system,
and the observations are the expected value of some physical quantity,i.e.,
some hermitian matrices. The observations constitute a sphere.

This paper is organized as follows. In section II, we describe the works been
done before and the problem we want to solve in more detail. In section III,
we introduce the architecture of neuron networks and argue why we need the
manifold of latent variants should be homeomorphic to that of observations. In
section IV, we show the performance of this protocol on three toy models. We
compare the observations and latent variables to show the relation between
them.

##  II Related work and our goal

Data-driven scientific discoveries are not new ideas. It follows from the
revolutionary work of Johannes Kepler and Sir Isaac Newton Roscher  _et al._
(2020). Unlike the 17th century, we now have higher quality data and better
calculation tools. People have done a lot of research on how to make machines
help people make scientific discoveries in different contexts Carleo  _et al._
(2019); Cichos  _et al._ (2020); Karpatne  _et al._ (2018); Wetzel (2017);
Rodriguez-Nieva and Scheurer (2019); Wu and Tegmark (2019). In the early days,
people paid more attention to the symbolic regression Kim  _et al._ (2020);
Udrescu and Tegmark (2020); Udrescu  _et al._ (2020); Lample and Charton
(2019). Another challenging direction is to let the machine design
experiments. In Melnikov  _et al._ (2018); Krenn  _et al._ (2016); Fösel  _et
al._ (2018); Bukov  _et al._ (2018), authors designed automated search
techniques and a reinforcement-learning-based network to generate new
experimental setups. In condensed physics, machine learning has been used to
characterize phase transitions Van Nieuwenburg  _et al._ (2017); Uvarov  _et
al._ (2020); Carrasquilla and Melko (2017); Wang (2016); Ch’Ng  _et al._
(2017); Yu and Deng (2021).

In recent years, some works have contributed to letting the machine search for
key physical concepts. They used the different networks to extract key
physical parameters from the experimental data Iten  _et al._ (2020); Zheng
_et al._ (2018); Lu  _et al._ (2020). In Zheng  _et al._ (2018), authors
propose an unsupervised method to extract physical parameters by interaction
networks Battaglia  _et al._ (2016). Another helpful tool is the VAE network,
which has been widely used for similar goal Lu  _et al._ (2020); Iten  _et
al._ (2020).

VAE network is a powerful tool, by which one can minimize the numbers of
extracted variables by making the variables independent of each other Higgins
_et al._ (2016); Burgess  _et al._ (2018). To do this, one can choose the
prior distribution $P(z)$ as latent variables $z$. However, this method fails
when the manifold of observations have some topological features. This paper
aims to solve this problem.

Here we describe this goal in more detail. Suppose we have an experimental
system $\mathcal{S}$. In this system $\mathcal{S}$, some physical variables
change as the experiment progresses. We use $\mathcal{P}$ to denote the set
consists of the value of these physical variables. We use $\mathcal{P}^{(k)}$
to represent the value of these variables in time $k$. From the system
$\mathcal{S}$ we can derive an experimental data set $\mathcal{E}$. Every data
point is a vector $\mathcal{E}^{(k)}\in\mathcal{E}$ where $\mathcal{E}^{(k)}$
denotes the data point belongs to time $k$. From the perspective of physics,
the change of experimental data must be attributed to the change of key
physical variables. Therefore there is a function
$f:\mathcal{P}\rightarrow\mathcal{E}$ such that
$f(\mathcal{E}^{(k)})=\mathcal{P}^{(k)}$ for any $k$. In this paper we aim to
find the function $\tilde{f}$ and $\tilde{f}^{-1}$ such that
$\tilde{\mathcal{P}}^{(k)}=\tilde{f}^{-1}(\mathcal{S}^{(k)})$ where
$\tilde{\mathcal{P}}^{(k)}$ constitute a set $\tilde{\mathcal{P}}$. We call
$\tilde{\mathcal{P}}^{(k)}$ the effective physical variables. We remark that
$\tilde{\mathcal{P}}^{(k)}$ is not necessary to equal to $\mathcal{P}^{(k)}$,
but their dimensions should be the same. Effective physical variables are
enough to describe the experimental system. In fact, one can redefine the
physical quantity can get a theory that looks very different but the predicted
results are completely consistent with existing theories.

The problem arise if we try to use neuron network to find proper function
$\tilde{f}$ and $\tilde{f}^{-1}$. All the functions neuron network can
simulate is continuousLABEL:, so $\tilde{f}$ and $\tilde{f}^{-1}$ must be
continuous. In a real physical system, $f$ and $f^{-1}$ don’t have to be like
this. Therefore, in some cases, we can never find a
$\tilde{\mathcal{P}}^{(k)}$ which has the same dimension with
$\mathcal{P}^{(k)}$. For example, suppose we have a ball rotating around a
center. The observable data is the location of the ball, denoted by Cartesian
coordinates $\mathcal{E}^{(k)}=(x^{(k)},y^{(k)})$, which form a circle, and
the simplest physical variable is the angle
$\mathcal{P}^{(k)}=\theta^{(k)}\in[0,2\pi)$. In this case,
$f:\theta\rightarrow(x,y)$ is continuous while $f^{-1}$ is not, which cannot
be approximated by neural network. This is because $f$ is a periodic function
of $\theta$. More generally, as long as the manifold composed of $\mathcal{E}$
has holes with any dimension, this problem arises. We call these physical
variables topological physical variables(TPVs). Back to the last example, we
can find the Betti numbers of the ball’s location is $[1,1,0]$, where the
second $1$ means that it has one TPVs. Due to this reason, we suggest using
TDA to identify the topological features firstly. After knowing the numbers of
TPVs, we can design proper latent variables and the corresponding loss
function $\mathcal{L}$. For the case we have two PPVs, we need two latent
variables, named $x$ and $y$, and we add the topological term
$|x^{2}+y^{2}-1|$ to $\mathcal{L}$ to restrict them.

In at least two cases, the manifold will have holes. One is the $\mathcal{E}$
of a conversed system,e.g., the classical harmonic oscillator. Another
situation stems from the limits of the physical quantity itself, such as the
single-qubit state, which forms a sphere. In addition to these two categories,
sometimes the choice of observations and references also affects their
topological properties.

##  III architecture of neuron network

observationslatent vaxriablesEncoder Figure 1:  The observations of a system
form a form a closed curve in a three-dimensional space. We can encoder it as
a two-dimensional circle. However, we don’t have to encode it as a circle.
Ellipse or other homeomorphic curves are all allowed.

As shown in figure 2, This network consists of two parts. The left is called
encoder, and the right is called decoder. The encoder part is used to encode
high-dimensional data into low-dimensional space, while the decoder part is
used to recover the data, i.e., the input from low-dimensional data. The
latent variables are low-dimensional data, and we hope they contain all the
information needed to recover the input data. So that we require the output
$\hat{x}$ is as close as possible to input $x$. Therefore, we can train this
neuron network without supervision.

In general, the encoder and decoder can simulate arbitrary continuous function
$f$ and $f^{-1}$. That imposes restrictions on latent variables:

######  Theorem 1.

The manifold composed of latent variables must be homeomorphic to that of
input $x$.

This limitation means that the Betti numbers should keep the same. Suppose the
inputs $x$ form a circle, as shown in 1, the easiest quantity to describe
these data is the angle. According to 1, this neuron network can never find
such a quantity because the angle will form a line segment that is not
homeomorphic to a circle. On the other hand, VAE network can’t even find the
Cartesian coordinate $\\{x_{1},x_{2}\\}$ because $x_{1}$ and $x_{2}$ are not
irrelevant. To handle this case, we suggest analyzing the Betti numbers as a
priori knowledge of input $x$ and tell the neuron network.

input $x$output $\hat{x}$EncoderDecoderLatent Figure 2: There is a bottleneck
in the network which forces a compressed knowledge representation. One needn’t
to set the decoder part as the inverse of encoder part, and the networks of
encoder and decoder don’t have to be the same. In general, the dimension of
output $\hat{x}$ should be the same as the input $x$. The number of hidden
variables is usually less than the input.

The general loss function is wirtten as Higgins  _et al._ (2016):

$\mathcal{L}=-\alpha\mathbb{E}_{q_{\phi}(\mathbf{z}\mid\mathbf{x})}\left[\log
p_{\theta}(\mathbf{x}\mid\mathbf{z})\right]+\beta
D_{KL}\left(q_{\phi}(\mathbf{z}\mid\mathbf{x})\|p(\mathbf{z})\right)$ (1)

Here $\theta,\phi$ are the parameters of decoder and encoder networks,
respectively. $p(z)$ is the prior distribution about the latent variables $z$.
$q_{\phi}(z|x)$ and $p_{\theta}(x|z)$ denote the conditional probability.
$D_{KL}$ means the KL divergence of two distributions.

$D_{KL}\left(q_{\phi}(\mathbf{z}\mid\mathbf{x})\|p(\mathbf{z})\right)=\mathbb{E}_{q_{\phi}(\mathbf{z}\mid\mathbf{x})}(\log
q_{\phi}(\mathbf{z}\mid\mathbf{x})-\log p(\mathbf{z}))$ (2)

To calculate the loss function for a given output $\hat{x}$, one needs to set
up a prior $p(z)$, and parameterize the distribution $p_{\theta}$ and
$q_{\phi}$. In traditional VAE, the distributions $p(z)$, $p_{\theta}$ and
$q_{\phi}$ are often supposed to be multidimensional Gaussian distribution.
The mean of $p(z)$ is zero the and variance is $1$. The mean and variance of
$p_{\theta}$ and $q_{\phi}$ is the output of encoder network.

In our cases, we choose the prior $p(\vec{z})$ as
$p(\vec{z})=p(\vec{z_{g}})\times p(\vec{z_{t}})$. Where $\vec{z_{g}}$ denotes
the GPVs, and $\vec{z_{t}}$ denotes the TPVs. With the same as traditional
VAE, in order to minimize the number of GPVs, a convenient way is to choose
$p(\vec{z_{g}})$ as Gaussion distribution $p(\vec{z_{g}})=N(0,I)$. According
to the topological features, we choose a proper topological term
$\mathcal{T}$. We choose the $p(\vec{z_{t}})$ as $p(\vec{z_{t}})=Ae^{-T}$.
Here $A$ is a constant. The variance can be asorpted into the hyperparameters
in loss function.

For the conditional distribution we choose as s
$q_{\phi}(\mathbf{z}\mid\mathbf{x})=N(\vec{z},I)$, with constantly variance
and means determined by the encoder network.
$p_{\theta}(\mathbf{x}\mid\mathbf{z})$ has the same expression
$p_{\theta}(\mathbf{x}\mid\mathbf{z})=N(x,I)$. In this assumption, the KL
divergence can be written as

$D_{KL}\left(q_{\phi}(\mathbf{z}\mid\mathbf{x})\|p(\mathbf{z})\right)=-\mathbb{E}_{q_{\phi}(\mathbf{z}\mid\mathbf{x})}(\log
p(\mathbf{z}))+constant$ (3)

Thus the loss function can be written as a simplier expression:

$\mathcal{L}=\alpha\|x-\hat{x}\|_{2}+\beta\mathcal{T}-\gamma
log(P(\vec{z_{g}}))$ (4)

where $\alpha,\beta,\gamma$ are hyperparameters and $\mathcal{T}$ is the
topological term depending on the Betti numbers. For example, when the Betti
numbers are $\\{1,0,1\\}$, $\mathcal{T}$ can be written as
$\mathcal{T}=|z_{1}^{2}+z_{2}^{2}+z_{3}^{2}-1|$. When the Betti numbers are
$\\{1,2,0\\}$, $\mathcal{T}$ can be written as
$\mathcal{T}=|(z_{1}^{2}+z_{2}^{2})-2(z_{1}^{2}-z_{2}^{2})|$, which is known
as a Lemniscate. Here we use $\vec{z}$ to denote the latent variables.
$P(\vec{z_{g}})$ is the distribution of $\vec{z_{g}}$. We need to choose some
latent variables as TPVs, and others as general physical variables(GPVs). In
general, we know how many TPVs we need from the Betti numbers, but we usually
don’t know how many GPVs we need. One solution is to set up as many GPVs as
possible, and the redundant GPVs will be zero.

##  IV Numerical Simulation

(a) latents of osillator

(b) comparison between latent and $x_{1}$

(c) comparison between latent and $v$

(d) latent of the ball’s orbit

(e) comparison between latent and $x$

(f) comparison between latent and $y$

(g) latent of the observations of quantum states

(h) comparison between the azimuths

Figure 3:  Fig 3(a) to 3(c) belong to the first numerical simulation. 3(d) to
3(f) belong to the second numerical simulation. 3(g) and 3(h) belong to the
third simulation. $z_{1}$ and $z_{2}$ denote the TPVs and $z_{g}$ denotes the
GPV. 3(a) and 3(d) shows that the GPV is always near zero, which means that we
don’t need a GPV in these model. For the case of quantum state, we compare the
polar coordinates of a quantum state and the azimuths of effective Bloch
sphere.

We test this neuron network by numerical simulation. Three toy models whose
manifolds are different are considered here.

We use pytorch to implement neural networks. Our code can be found here. We
use the same structure in the encoding and decoding network. They have two
hidden layers, each with 20 neurons. In encoder network, we choose Tanh as the
active function, while in decoder network, we choose ReLU as the active
function. In numerical simulation, we first generate a database of $1000\times
m$, where m is the dimension of the observations. We then use the TDA tools to
analyze the Betti numbers of the manifold constituted by observations and set
up the latent neurons. We used the Adam optimizer and set the learning rate to
$0.0001$. At each training session, we randomly select 100 sets of data from
the database as a batch, then calculate the average of its loss and reverse
propagate it. All training can be done on the desktop in less than 6 hours.

###  IV.1 classical Harmonic oscillator

$k_{1}$$m$$k_{2}$ Figure 4: The conversed system consists of two spring
connected to a ball. The spring constants $k_{1}$,$k_{2}$ and the mass of the
ball keep unchanged during the experiment. The total energy comprises of the
potential energy of two springs and the kinetic of the ball.

One type of vital importance is the system with some conserved quantity. In
classical mechanics, we study these systems by their phase space, constituted
by all possible states. Generalized coordinates and generalized momentum are
usually not independent of each other, but it is not possible to uniquely
determine the other when only one is known. This is caused by the topological
nature of the phase space. In some cases, the experimenter may only be able to
make observations and cannot exert influence on the observation object. At
this time, the observations may constitute a compact manifold, and the
traditional VAE network cannot accurately reduce the dimension of the
parameter space.

The most common conversed system is the harmonic oscillator, which is a
conservative system. We first consider a ball connected with two springs, as
shown in Fig 4. We can write the energy as

$E=\frac{1}{2}mv^{2}+\frac{1}{2}k_{1}(x-\frac{1}{2})^{2}+\frac{1}{2}k_{2}(x+\frac{1}{2})^{2}$
(5)

Here we make $m=1$ and $k_{1}=k_{2}=1$. The unit is not important. In loss
function 4 we make $[\alpha,\beta,\gamma]=[1,1,100]$. In this system, the
underlying changing physical variables are $\mathcal{P}^{(k)}=(x_{1},v)$ or
$(x_{2},v)$. The observations one can choose here are
$\mathcal{E}^{k}=\\{x_{1},x_{2},v\\}$ where $x_{1}$ and $x_{2}$ denote the
distance from the ball to the bottom of two springs, respectively, and $v$
denote the speed of the ball. We specify that the direction of speed is
positive to the right and negative to the left. We generate the observations
$\mathcal{E}^{(k)}$ by randomly sampling from the evolution of the system.
According to classical mechanics, we know that the manifold should be a circle
embedded in 3-d Euclid space. Programme shows that the Betti numbers are
$(1,1,0)$. It means that we need 2 latent variables $(z_{1},z_{2})$ and the
topological term in loss function (4) is
$\mathcal{T}=|z_{1}^{2}+z_{2}^{2}-1|$. Besides, we set up one general physical
variable, whose prior $P(z_{g})$ is a Gaussian distribution.

After the train is finished, we calculate the latent variables corresponding
to the observations in $\mathcal{E}$ by encoder network. As is shown in Fig
3(a), the latent variables constitute a circle again. Fig 3(a) shows that the
GPV $z_{g}$ is always zero for different observations. This means that the
effective physical variables are $\tilde{\mathcal{P}^{(k)}}=(z_{1},z_{2})$.

We compared the new physical variables $(z_{1},z_{2})$ with observations
$(x_{1},v)$, as shown in Fig LABEL:fig:os_x and 3(c). Given effective physical
variables $\tilde{P}$, we can uniquely determine $P$ while the mapping is
continuous. They are one-to-one correspondence, so $\tilde{P}$ can be used to
build a theory.

###  IV.2 Orbit

ball-2ball-1$\omega$$2\omega$ Figure 5: Two small balls rotate around the same
immobility point. The two balls move at a constant speed circumference on the
planes of the two meetings. One ball has twice the angular speed of the other.

In classical physics and special relativity, the inertial reference system
plays an important role. The physical laws in an inertial reference frame are
usually simpler than those in a non-inertial reference frame. In fact, there
was no concept of an inertial reference frame at the beginning. Due to the
existence of gravity, there is no truly perfect inertial reference frame. When
observing some simple motions in a non-inertial frame, the observations may
constitute some complex manifolds.

Consider a scenario where two balls, labeled by $1$ and $2$, rotate around a
fixed point, as shown in Fig 5. Both balls do a constant-speed circular
motion. We assume that the distance between it and the fixed point is $1$.
Establishing the Cartesian coordinate system with a fixed point as the origin,
$ball-1$ starts at $(1,0,0)$ and $ball-2$ starts at $(0,0,1)$.They have the
same radius of rotation, but they are on different planes. $ball-1$ moves in
the $z=0$ plane and $ball-2$ moves in the $y=0$ plane. $ball-1$ has the
angular speed of $\omega_{1}=1$ and $ball-2$ has the angular speed of
$\omega_{2}=2$. Unit is not important. Observations $\mathcal{E}^{k}=(x,y,z)$
is the three-dimensional coordinates of $ball-1$ measured by $ball-2$ as the
reference system.

We generate the observations $\mathcal{E}^{(k)}$ by random sampling from the
trajectory With TDA we know the betti numbers of $\mathcal{E}$ are $(1,2,0)$
so the topological term is
$|(z_{1}^{2}+z_{2}^{2})^{2}-0.01\times(z_{1}^{2}-z_{2}^{2})|$, and again We
set up one GPV. The hyperparameters of loss function 4 is
$[\alpha,\beta,\gamma]=[1,100,100]$.

After training, the latent variables corresponding to the observations
$\mathcal{E}$ are plotted, as shown in fig 3(d). The GPV is always zero for
different observations, which means the effective physical variables is
$\tilde{P}=(z_{1},z_{2})$, like the first example. Fig 3(e) and 3(f) shows the
comparison between the effective physical variables and observations. The
results show that $\tilde{P}$ and observational measurements are still one-to-
one, so $\tilde{P}$ is an effective representation.

###  IV.3 Quantum state

Both of the previously introduced situations come from the limitations of
experimental conditions. If we can modify the conserved quantities of
conservative systems through experiments, or find some approximate inertial
reference frames, such as the earth or distant stars, then it is possible to
turn it into a situation that can be solved by traditional VAE networks.
Unlike these, the topological properties of some observations are derived from
the laws of physics. If this physical quantity can only take partial values in
a certain experimental system, then the VAE network may work at this time. But
if we want to establish a physical theory, you need to have a comprehensive
understanding of the key parameters, and then you need a VAE network based on
topological properties.

In quantum mechanics, quantum state is described by wave function, i.e. a
vector in Hilbert space. Here we consider a two-level system, which according
to quantum mechanics can be described as
$|{\psi}\rangle=a|{0}\rangle+b|{1}\rangle$, where $|a|^{2}+|b|^{2}=1$. The
machine don’t know how to describe this system, but it will learn some
efficient variables. Suppose we can get five observations in experiments,i.e.
$\mathcal{E}^{k}=\\{\mathcal{O}^{1},\mathcal{O}^{2},\dots,\mathcal{O}^{5}\\}$.
In experiments, $\mathcal{E}$ is due to the experimental setup. In our
numerical simulation, we calculate observations by
$\mathcal{O}^{i}=\langle{\psi}|O^{i}|{\psi}\rangle$, where $O^{i}$ is the
pauli matrix and the combination of pauli matrix:

$\begin{split}\mathcal{O}_{1}=\left[\begin{array}[]{cc}0&1\\\
1&0\end{array}\right],\mathcal{O}_{2}=\left[\begin{array}[]{cc}0&i\\\
-i&0\end{array}\right],\mathcal{O}_{3}=\left[\begin{array}[]{cc}1&0\\\
0&-1\end{array}\right]\\\ \mathcal{O}_{4}=\left[\begin{array}[]{cc}0&1+i\\\
1-i&0\end{array}\right],\mathcal{O}_{5}=\left[\begin{array}[]{cc}1&i\\\
-i&-1\end{array}\right]\end{split}$ (6)

We need many different states, so the coefficient $a$ and $b$ of the wave
function are the physical variables that change as the experiment progresses.
In this model, $\mathcal{P}$ consists of wave functions we use and
$\mathcal{P}^{k}=\\{a^{k},b^{k}\\}$. The first step is to characterize the
topological feature of $\mathcal{E}$. By TDA, we can find that the Betti
numbers of the data set are $[1,0,1]$, which means that the manifold of
$\mathcal{E}$ is homeomorphic to a sphere. This can be understood because we
can use the point on the Bloch sphere to represent the state of a two-level
system. So that we know the number of TVs is $3$. However, we don’t know how
many GVs are. Different from TVs, we can assume GVs are independent of each
other. In general, we do this by assuming $P(\vec{z})$ in (4) is independent
Gaussian distribution. In this case, we only introduce one GV.

As shown in Fig 3(g), after training, TVs form a sphere. At the same time GV
stabilizes near 0(not drawn in the figure). That means the five observations
can be reduced to three variables continuously. We call these three variables
as equivalent density matrix, denoted by $\tilde{\rho}$. Only two of them are
independent. We can construct two independent angles $(\theta,\phi)$ by
transforming Cartesian coordinates to polar coordinates. In figure 3(h) we
compare $(\theta_{0},\phi_{0})$ and $\phi_{1}$ corresponding to the equivalent
density matrix. We can see that these two representations have a one-to-one
correspondence. Unlike before, the mapping here is discontinuous.

##  V Conclusion

In this work, we discuss the defect of the traditional VAE network and propose
a simple solution. We can extract the minimum effective physical variables
from the experimental data with the improved method by classifying the latent
variables. We test our approach on three models. They represent three
different situations that may arise that traditional VAE cannot handle. Some
of them come from experimental restrictions, and some come from physical laws.
We think the latter situation is more essential. However, in the more complex
case, the Betti numbers are not the only useful topological features. Two
manifolds may have the same Betti numbers but not Homeomorphic. In such a
case, more topological features are needed to design reasonable restrictions
on latent variables. When the manifold dimension is higher, it may be hard for
TDA to calculate the Betti numbers. One efficient way is to firstly reduce the
dimensions by traditional autoencoder, and then calculate the Betti numbers of
latent variables. Another important question is how to ensure that the
relation between meaningful variables and observations is simple. We leave it
for the future work.

##  Appendix A persistent homology and betti numbers

Persistent homology is a useful tool for analyzing topological data. The first
step is to generate a simplicial complex from the observations. Vietoris–Rips
complex is a way of forming a topological space from distances in a set of
points. This method uses a generalisation of an $\epsilon$ neighbouring graph
and the final complex is

$\mathcal{V}_{\epsilon}=\\{\sigma|r(u,v)<\epsilon,(u,v)\in\sigma\\}$ (7)

Here $\sigma$ denotes the simplex and $u,v$ are two data points. $r$ is the
Euclidean metric. A $k$-simplex $\sigma$ is expressed as
$\sigma=[p_{1},p_{2},\dots,p_{k+1}]$, where $p$ denotes the point in the
space. Given a set of $k$-simplex $\sigma_{i}$, one $k$-chain is defined by

$c=\Sigma_{i}c_{i}\sigma_{i}$ (8)

Here the coefficients $c_{i}$ take values in some field. For the simplicial
complex $\mathcal{V}$, we denote the set of all $k$-chains by
$C_{k}(\mathcal{V})$. We can introduce the addition between two $k$-chain. For
$c=\Sigma_{i}c_{i}\sigma_{i}$ and $d=\Sigma_{i}d_{i}\sigma_{i}$, the addition
is $c+d=\Sigma_{i}(c_{i}+d_{i})\sigma_{i}$. Furthermore, the set of all
$k$-chains $C_{k}(\mathcal{V})$ form a abelian group $(C_{k}(\mathcal{V}),+)$.

For different $k$, a natural group homomorphism is the boundary operator

$\partial_{k}:C_{k}(V)\rightarrow C_{k-1}(V)$ (9)

Boundary operator is linear, and it can be defined by the simplex

$\partial_{k}(\sigma_{k})=\Sigma_{i=0}^{i=k}(-1)^{i}\langle
p_{0},\dots,p_{i-1},p_{i+1},\dots,p_{k}\rangle$ (10)

One important subgroup of $C_{k}(\mathcal{V})$ is the kernel of the boundary
operator $Z_{k}$, namely $k$-cycles. There is also an important subgroups of
$Z_{k}$, denoted by $B_{k}$, which is the imagine
$B_{k}(V)=Im(\partial_{k+1}C_{k+1}(\mathcal{V}))$.

Both $Z_{k}$ and $B_{k}$ are normal, so we can define the quotient group
$H_{k}=Z_{k}/B_{k}$, called the $k$-th homology group. The rank of $H_{k}$ is
called the $k$-th Betti number.

##  Appendix B Comparison with other work

As we point in the paper, the topological feature of latent space will
influence the construction of the output. In the field of machine learning,
this phenomenon is called manifold mismatch de Haan and Falorsi (2018);
Davidson  _et al._ (2018); Rey  _et al._ (2019); Gong and Cheng (2019), which
will lead to poor representations.

Let’s consider what happens when manifold mismatch occurs. Recall the first
example, suppose we have a latent structure as $S^{1}$, i.e., a circle. While
in normal VAE, the prior distribution $p(z)$ will limit the latent structure
to be $R^{n}$, where $n$ is the number of latent variables. This prior will
make the hidden variables independent of each other. Like before, here we set
$n=3$. One parameter can describe this system, e.g. the angle $\theta$. Thus
we get a network which maps the coordinate $(x,y)$ to a parameter $\theta$ and
then maps the $\theta$ to the coordinate $(x,y)$. In practice, our data set is
finite, and the finally network will only be effective for an arc on the
circle, instead of the total circle. This problem can be worse when the
manifold dimension is higher.

In the earlier works, some methods have been developed for capturing the
topological features of date set. However, to our understanding, these methods
are not suitable for our goal because they usually derive a set of entangled
latent vatiables.

## References

  * Liu  _et al._ (2018) J. Liu, X. Kong, F. Xia, X. Bai, L. Wang, Q. Qing, and I. Lee, IEEE Access 6, 34403 (2018).
  * Teichert  _et al._ (2019) G. H. Teichert, A. Natarajan, A. Van der Ven, and K. Garikipati, Computer Methods in Applied Mechanics and Engineering 353, 201 (2019).
  * Bourilkov (2019) D. Bourilkov, International Journal of Modern Physics A 34, 1930019 (2019).
  * Mishra  _et al._ (2016) S. Mishra, D. Mishra, and G. H. Santra, Indian Journal of Science and Technology 9, 1 (2016).
  * Vamathevan  _et al._ (2019) J. Vamathevan, D. Clark, P. Czodrowski, I. Dunham, E. Ferran, G. Lee, B. Li, A. Madabhushi, P. Shah, M. Spitzer, _et al._ , Nature Reviews Drug Discovery 18, 463 (2019).
  * Valletta  _et al._ (2017) J. J. Valletta, C. Torney, M. Kings, A. Thornton, and J. Madden, Animal Behaviour 124, 203 (2017).
  * Schmidt  _et al._ (2019) J. Schmidt, M. R. Marques, S. Botti, and M. A. Marques, npj Computational Materials 5, 1 (2019).
  * Carleo  _et al._ (2019) G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborová, Reviews of Modern Physics 91, 045002 (2019).
  * Ha and Jeong (2021) S. Ha and H. Jeong, arXiv preprint arXiv:2102.04008 (2021).
  * Zhang and Lin (2018) S. Zhang and G. Lin, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 474, 20180305 (2018).
  * Boullé  _et al._ (2021) N. Boullé, C. J. Earls, and A. Townsend, arXiv preprint arXiv:2105.00266 (2021).
  * Zobeiry and Humfeld (2019) N. Zobeiry and K. D. Humfeld, arXiv preprint arXiv:1909.13718 (2019).
  * Farina  _et al._ (2020) M. Farina, Y. Nakai, and D. Shih, Physical Review D 101, 075021 (2020).
  * Wu and Tegmark (2019) T. Wu and M. Tegmark, Physical Review E 100, 033311 (2019).
  * Zheng  _et al._ (2018) D. Zheng, V. Luo, J. Wu, and J. B. Tenenbaum, arXiv preprint arXiv:1807.09244 (2018).
  * D’Agnolo and Wulzer (2019) R. T. D’Agnolo and A. Wulzer, Physical Review D 99, 015014 (2019).
  * Udrescu and Tegmark (2020) S.-M. Udrescu and M. Tegmark, Science Advances 6, eaay2631 (2020).
  * Udrescu  _et al._ (2020) S.-M. Udrescu, A. Tan, J. Feng, O. Neto, T. Wu, and M. Tegmark, arXiv preprint arXiv:2006.10782 (2020).
  * Wasserman (2018) L. Wasserman, Annual Review of Statistics and Its Application 5, 501 (2018).
  * Murugan and Robertson (2019) J. Murugan and D. Robertson, arXiv preprint arXiv:1904.11044 (2019).
  * Kingma and Welling (2019) D. P. Kingma and M. Welling, arXiv preprint arXiv:1906.02691 (2019).
  * Kingma and Welling (2013) D. P. Kingma and M. Welling, arXiv preprint arXiv:1312.6114 (2013).
  * An and Cho (2015) J. An and S. Cho, Special Lecture on IE 2, 1 (2015).
  * Khobahi and Soltanalian (2019) S. Khobahi and M. Soltanalian, arXiv preprint arXiv:1911.12410 (2019).
  * Rezende  _et al._ (2014) D. J. Rezende, S. Mohamed, and D. Wierstra, in  _International conference on machine learning_ (PMLR, 2014) pp. 1278–1286.
  * Higgins  _et al._ (2016) I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner,  (2016).
  * Burgess  _et al._ (2018) C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner, arXiv preprint arXiv:1804.03599 (2018).
  * Luchnikov  _et al._ (2019) I. A. Luchnikov, A. Ryzhov, P.-J. Stas, S. N. Filippov, and H. Ouerdane, Entropy 21, 1091 (2019).
  * Cerri  _et al._ (2019) O. Cerri, T. Q. Nguyen, M. Pierini, M. Spiropulu, and J.-R. Vlimant, Journal of High Energy Physics 2019, 1 (2019).
  * Roscher  _et al._ (2020) R. Roscher, B. Bohn, M. F. Duarte, and J. Garcke, Ieee Access 8, 42200 (2020).
  * Cichos  _et al._ (2020) F. Cichos, K. Gustavsson, B. Mehlig, and G. Volpe, Nature Machine Intelligence 2, 94 (2020).
  * Karpatne  _et al._ (2018) A. Karpatne, I. Ebert-Uphoff, S. Ravela, H. A. Babaie, and V. Kumar, IEEE Transactions on Knowledge and Data Engineering 31, 1544 (2018).
  * Wetzel (2017) S. J. Wetzel, Physical Review E 96, 022140 (2017).
  * Rodriguez-Nieva and Scheurer (2019) J. F. Rodriguez-Nieva and M. S. Scheurer, Nature Physics 15, 790 (2019).
  * Kim  _et al._ (2020) S. Kim, P. Y. Lu, S. Mukherjee, M. Gilbert, L. Jing, V. Čeperić, and M. Soljačić, IEEE Transactions on Neural Networks and Learning Systems (2020).
  * Lample and Charton (2019) G. Lample and F. Charton, arXiv preprint arXiv:1912.01412 (2019).
  * Melnikov  _et al._ (2018) A. A. Melnikov, H. P. Nautrup, M. Krenn, V. Dunjko, M. Tiersch, A. Zeilinger, and H. J. Briegel, Proceedings of the National Academy of Sciences 115, 1221 (2018).
  * Krenn  _et al._ (2016) M. Krenn, M. Malik, R. Fickler, R. Lapkiewicz, and A. Zeilinger, Physical review letters 116, 090405 (2016).
  * Fösel  _et al._ (2018) T. Fösel, P. Tighineanu, T. Weiss, and F. Marquardt, Physical Review X 8, 031084 (2018).
  * Bukov  _et al._ (2018) M. Bukov, A. G. Day, D. Sels, P. Weinberg, A. Polkovnikov, and P. Mehta, Physical Review X 8, 031086 (2018).
  * Van Nieuwenburg  _et al._ (2017) E. P. Van Nieuwenburg, Y.-H. Liu, and S. D. Huber, Nature Physics 13, 435 (2017).
  * Uvarov  _et al._ (2020) A. Uvarov, A. Kardashin, and J. D. Biamonte, Physical Review A 102, 012415 (2020).
  * Carrasquilla and Melko (2017) J. Carrasquilla and R. G. Melko, Nature Physics 13, 431 (2017).
  * Wang (2016) L. Wang, Physical Review B 94, 195105 (2016).
  * Ch’Ng  _et al._ (2017) K. Ch’Ng, J. Carrasquilla, R. G. Melko, and E. Khatami, Physical Review X 7, 031038 (2017).
  * Yu and Deng (2021) L.-W. Yu and D.-L. Deng, Physical Review Letters 126, 240402 (2021).
  * Iten  _et al._ (2020) R. Iten, T. Metger, H. Wilming, L. Del Rio, and R. Renner, Physical review letters 124, 010508 (2020).
  * Lu  _et al._ (2020) P. Y. Lu, S. Kim, and M. Soljačić, Physical Review X 10, 031056 (2020).
  * Battaglia  _et al._ (2016) P. W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu, arXiv preprint arXiv:1612.00222 (2016).
  * de Haan and Falorsi (2018) P. de Haan and L. Falorsi, arXiv preprint arXiv:1812.10783 (2018).
  * Davidson  _et al._ (2018) T. R. Davidson, L. Falorsi, N. De Cao, T. Kipf, and J. M. Tomczak, arXiv preprint arXiv:1804.00891 (2018).
  * Rey  _et al._ (2019) L. A. P. Rey, V. Menkovski, and J. W. Portegies, arXiv preprint arXiv:1901.08991 (2019).
  * Gong and Cheng (2019) L. Gong and Q. Cheng, arXiv preprint arXiv:1901.09970 (2019).