# WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning

Guotao Wang, Chenglizhao Chen*, Aimin Hao, Hong Qin, and Deng-Ping Fan
Guotao Wang is with the State Key Laboratory of Virtual Reality Technology and
Systems, Beihang University, China. (E-mail<EMAIL_ADDRESS>Chenglizhao Chen
is with the College of Computer Science and Technology, China University of
Petroleum, China. (E-mail: cclz123@163.com). Aimin Hao is with the State Key
Laboratory of Virtual Reality Technology and Systems, Beihang University,
Research Unit of Virtual Human and Virtual Surgery, Chinese Academy of Medical
Sciences, and with Pengcheng Laboratory, China. (Email<EMAIL_ADDRESS>Hong
Qin is with the Computer Science Department, Stony Brook University, USA.
(E-mail: qin@cs.stonybrook.edu). Deng-Ping Fan is with ETH Zurich, Zurich,
Switzerland (Email<EMAIL_ADDRESS>Corresponding author: Chenglizhao
Chen.

###### Abstract

To date, the widely adopted way to perform fixation collection in panoptic
video is based on a head-mounted display (HMD), where users’ fixations are
collected while wearing an HMD to explore the given panoptic scene freely.
However, this widely-used data collection method is insufficient for training
deep models to accurately predict which regions in a given panoptic are most
important when it contains intermittent salient events. The main reason is
that there always exist “blind zooms” when using HMD to collect fixations
since the users cannot keep spinning their heads to explore the entire
panoptic scene all the time. Consequently, the collected fixations tend to be
trapped in some local views, leaving the remaining areas to be the “blind
zooms”. Therefore, fixation data collected using HMD-based methods that
accumulate local views cannot accurately represent the overall global
importance — the main purpose of fixations — of complex panoptic scenes. To
conquer, this paper introduces the auxiliary window with a dynamic blurring
(WinDB) fixation collection approach for panoptic video, which doesn’t need
HMD and is able to well reflect the regional-wise importance degree. Using our
WinDB approach, we have released a new PanopticVideo-300 dataset, containing
300 panoptic clips covering over 225 categories. Specifically, since using
WinDB to collect fixations is blind zoom free, there exists frequent and
intensive “fixation shifting” — a very special phenomenon that has long been
overlooked by the previous research — in our new set. Thus, we present an
effective fixation shifting network (FishNet) to conquer it. All these new
fixation collection tool, dataset, and network could be very potential to open
a new age for fixation-related research and applications in $360^{\rm o}$
environments.

###### Index Terms:

HMD-free, Distortion-free, Panoptic Video Fixation Learning

##  1 Introduction and Motivation

Given a panoptic scene, the primary goal of panoptic fixation prediction is to
perceive the regional-wise importance, which reflects the varying degrees of
attention that different areas of the entire scene. This objective enables the
rapid localization of the most “important regions” in the scene. Generally,
the localized important regions have large variants of applications. As shown
in Fig. 1, it can drive dynamic regional-wise compression ratio and rendering
quality [1]. When wearing a head-mounted display (HMD), we can use the
localized regions to strike a better balance between computation cost and
rendering quality and thus have better graphics and smoother performance,
ultimately increasing immersion [2].

Figure 1: Panoptic fixations that can well reflect the scene’s regional-wise
importance can be applied to multiple applications. Here we illustrate the two
most representative ones: (a) panoptic video navigation [1], which facilitates
the localizing of intermittent “salient events” in blind zoom, and (b)
panoptic video compression [2]. Figure 2:  The existing HMD-based fixation
collection method [3, 4, 5, 6] for panoptic data has a critical limitation —
blind zoom, results in the collected fixations being insufficient to train
deep models to accurately predict which regions in a given panoptic are most
important (A). The reason causing this drawback has been shown in (B), where
users wearing an HMD tend to become “retard” after the early scene exploring
stage, resulting in missing important events that occurred in blind zooms. (C)
summarizes the advantages of our novel WinDB against the existing HMD-based
fixation collection, where advantages and disadvantages are respectively
remarked as ✓ and ✗. See Sec. 1 for details.

Different from the conventional 2D fixation prediction [7, 8, 9, 10], which
has received extensive research attention, the panoptic fixation prediction
[11, 12, 13, 1, 14, 15] is currently in its infancy. The major problem causing
such slow progress is the shortage of large-scale datasets [16, 17, 18, 11],
because collecting human-eye fixations in panoptic scene is much more
challenging than that in conventional 2D scene [2, 19, 20, 21, 22, 23]. Also,
the panoptic fixation prediction is much more complex than the conventional
fixation prediction in 2D images, where 2D data only has one fixed view, yet,
panoptic data allows users to explore $\rm 360^{\rm o}$ panoptic video freely
[24, 25, 26, 27]. Thus, our panoptic fixation prediction research community is
currently facing a dilemma — using tiny small-scale training data111There are
only 208 video clips most commonly used in our research community, and most of
them are simple scenes. to beat a complex problem.

To date, the HMD-based human-eye fixation collection [3, 4, 5, 6] is the most
popular approach, where users wear an HMD to explore the given panoptic scene
freely [28, 29, 30], and, at the same time, fixations are collected. Albeit
its broad application, the HMD-based fixation collection [3, 4, 5, 6, 31] has
two problems, and one of them is extremely critical. First, there always exist
“blind zooms” when using HMD to collect fixations since the users cannot keep
spinning their heads to explore the entire panoptic scene all the time. The
blind zoom problem makes the HMD-based fixations inconsistent with the real
regional-wise importance degrees in the given panoptic scene. Thus, a salient
event occurring in “blind zoom” might receive zero fixation. To facilitate a
better understanding, we have further demonstrated a vivid example in Fig. 2-A
and B. Second, the HMD-based fixation collection is relatively
“expensive”222Our concern is not about the high cost of the equipment itself
but rather the expensiveness of the annotation process., and users usually
feel very uncomfortable (_e.g._ , cybersickness [32]) when wearing an HMD to
explore the panoptic scene. In a word, the methodology of HMD-based fixation
collection is ill-posed, and its annotation process is extremely expensive.

Besides the above-mentioned HMD-based fixation prediction approach, it is
worth mentioning that plain Equi-Rectangular Projection (ERP) [33, 34, 35, 36,
37, 38], which projects a panoptic scene, a typical spherical data, to 2D
formulation, is also generally not suitable to serve as the platform for
human-eye fixation collection. The primary reason is that the ERP-based 2D
form suffers severe visual distortions [39, 40, 41, 42], especially for those
regions around the poles (see the top row of Fig. 2-A). As a result, the
distorted regions — being irregular to their surroundings, may occasionally
draw users’ fixations even if they are not salient [43, 44, 45, 46]. In short,
compared with the HMD-based method, ERP is a double sword; its advantage
(_e.g._ , no blind zoom) cannot outweigh its disadvantage (_e.g._ , severe
visual distortions), making it unsuitable in real $360^{\rm o}$ fixation
collection.

Given the abovementioned aspects, this paper presents a novel approach, named
as auxiliary window with a dynamic blurring (WinDB), to panoptic fixation
collection. Our WinDB approach has considered all the advantages and
disadvantages of both HMD-based and ERP-based methods. The key idea is to take
full advantage of the ERP-based fixation collection (_i.e._ , blind zoom free)
and resort to a series of tailored designs to suppress the visual distortions,
and we can preview its overview in Fig. 4. Moreover, since our WinDB approach
is HMD-free, enabling users to explore a panoptic scene in front of a computer
and collect fixations via an eye tracker, which is more comfortable than the
HMD-based method (see Fig. 2-C-b). The fixations collected by our WinDB
approach can well indicate each region’s importance degree in the given
panoptic scene.

Figure 3:  Qualitative demonstration of the differences between the datasets
collected by our WinDB method and the HMD method (_i.e._ , VR-Eye Tracking
[5]). Sub-figure A illustrates the fixation shifting phenomenon in
PanopticVideo-300 dataset. Since our WinDB method does not have blind zoom,
our WinDB is able to capture salient events that are ignored by the HMD
method, _i.e._ , the sudden human and animal events within the blind zoom.
Sub-figure B shows the phenomenon of “fixation shifting”, in which the
fixation shifts from the talking person to the person pushing the door.
However, since the HMD method has a blind zoom, it has focused on the person
talking.

In addition, using our proposed WinDB approach, we have constructed a large
panoptic video fixation prediction dataset, PanopticVideo-300333The
PanopticVideo-300 is now publicly available at
https://github.com/guotaowang/PanopticVideo-300., which is the most
challenging dataset in 360o video fixation prediction due to its significant
inclusion of scenes with blind zoom. Since the blind-zoom issue has been
nicely addressed, the fixations collected by our proposed WinDB approach can
well reflect the regional-wise importance, making PanopticVideo-300 the first
comprehensive dataset in our research field.

_W.r.t_ our PanopticVideo-300, we have found an interesting phenomenon — there
exists frequent “fixation shifting”. Because of the limitation of HMD (_i.e._
, blind zoom), the fixations collected by HMD tend to be generally smooth in
the conventional panoptic fixation sets (_e.g._ , VR-EyeTracking [5] and
Wild360 [44]). As we have stated, HMD-based fixations tend to be trapped in
local view due to blind zoom, and thus, these fixations are generally smooth.
In sharp contrast, our WinDB is blind-zoom free; thus, those salient events
that were supposed to be neglected by the HMD-based method can now be fully
discovered. As a result, in our PanopticVideo-300, fixations might shift to
another long-distance position in a very short period (see Fig. 3-A), and we
call this phenomenon “fixation shifting”. This phenomenon is not bad at all;
instead, it can further verify the solidness of our WinDB in collecting
fixations. For a better reading, we have provided a vivid example in Fig. 3-B,
where the fixations are shifted from the talking person to the man who pushed
the door in. The corresponding technical details of WinDB and more in-depth
analysis will be provided in Sec. 3.

Further, we face another dilemma, _i.e._ , none of the previous fixation
prediction networks can well handle the “fixation shifting” phenomenon. The
main reasons for such deficiency are generally two-fold: 1) their networks’
designs are over emphasized on pursuing the spatiotemporal smoothness towards
their predicted fixations, and 2) their sensing scope for fixations is
basically local, making them theoretically impossible to perceive long-range
fixation shifting. So, this paper also provides a new network (_i.e._ ,
FishNet) to handle the fixation shifting, whose technical rationale is very
inspiring and has the potential to guide future works, and this novel content
will be discussed and detailed in Sec. 5. In summary, the key contributions
include the following:

  * •

We introduce a new fixation collection approach (WinDB) for panoptic data,
which is the first one that has truly conquered the HMD-induced blind zoom
limitation;

  * •

Built upon WinDB, we have released a new set, the Panoptic Video-300, which is
the first solid and most challenging dataset for 360o video fixation
prediction, whose fixations can well reflect the regional-wise importance;

  * •

As the first attempt, we have devised a novel paradigm of network design to
handle the “fixation shifting” challenge, and we coin the new network as
FishNet;

  * •

This paper has conducted a whole package work, including a new data collection
way (WinDB444The WinDB tool is publicly available at
https://github.com/guotaowang/WinDB.), a new network (FishNet555The codes and
results are also publicly available at
https://github.com/guotaowang/FishNet.), and a new dataset
(PanopticVideo-300666The PanopticVideo-300 is now publicly available at
https://github.com/guotaowang/PanopticVideo-300.), whose methodologies, new
findings, in-depth analysis, and conclusion can jointly contribute our
research community.

Figure 4:  The overall pipeline of our new HMD-free fixation collection
approach for panoptic data. Compared to the widely-used HMD-based method, our
WinDB approach is more economical, comfortable, and reasonable. The most
severe distortion in the given panoptic video scene will be solved by D and E,
while the medium distortion will be solved by A, B, and C. See Sec. 3 for
details.

##  2 Related Work

###  2.1 Panoptic Fixation Collection Approach

The fixation collection method of panoptic video fixation collection has two
main research branches, _i.e._ , the methods based on HMD [3, 4, 5, 6] and the
methods based on ERP [44, 47]. HMD-based methods [3, 4, 5, 6] require the
users to wear the HMD to watch the panoptic video, and the users’ fixation is
collected. The advantage of this method is that there is no distortion
problem. However, in this method, users can only observe the content within
the local view of HMD.

In contrast, ERP-based methods [44, 47] only require the users to watch the
video content played on a computer screen and get the users’ fixation using
eye-tracking [44] or mouse input [47]. These methods have the advantage of
ensuring users are blind-zoom-free.

###  2.2 Panoptic Fixation Learning Network

Panoptic video fixation learning aims to rapidly locate important regions, but
it’s more challenging for panoptic video compared to traditional 2D methods
[7, 8, 9, 10]. In 2D video, fixation learning extracts important regions
within a fixed view. In panoptic video [3, 4, 5, 6], all views must be
considered. However, due to bind zoom, HMD-based methods only capture the
locally important region, not across the entire panorama. This leads to
inaccuracies in predicting the most important region. Moreover, there are
other issues in panoptic fixation learning networks, _e.g._ , problems with
loading pre-trained model parameters, and feature distortion. Panoptic
fixation learning networks have three main research branches. Next, we review
recent advancements.

(1) ERP and CMP bi-stream fusion scheme. In this approach [48, 49, 24, 44], 2D
models were employed to capture both panoptic ERP information and detailed
local information from cube maps (CMP). The fusion of these ERP and CMP bi-
streams was achieved through different techniques, such as dynamic weighted
fusion [50], bi-projection fusion [48], and multiplication [49]. It’s worth
noting that this fusion process lacked explicit alignments, essentially making
it an additional step for feature embedding.

(2) Sphere convolution-based network. Spherical convolution [11, 15, 51, 52,
53, 3] involves several key steps. Firstly, it identifies the central position
of the convolution kernel on the ERP. Then, it calculates the kernel’s surface
area on the sphere using ERP-to-sphere coordinate mapping. Next, it maps the
sphere’s coordinates to their corresponding ERP pixels, which are then used as
input for the CNN layers to embed semantic features. Finally, the CNN outputs
are re-mapped back to the ERP. Spherical convolution offers the advantage of
performing operations directly on the sphere’s surface, eliminating ERP
distortion. However, it presents a challenge as it differs from the standard
CNN model, making all available feature backbones for semantic feature
embedding unavailable.

(3) Transformer-based method. This method [54] addresses ERP distortion by
incorporating local 2D projection layers into a Transformer model [55, 56].
Each local projection layer employs deformable convolution [57] to map
spherical information to local 2D patches. The spherical to 2D patch
coordinate mapping determines the projection location. While it significantly
reduces ERP distortion, it generates intermediate features from ERP with some
visual distortions, limiting the utilization of pre-trained network
parameters.

##  3 Novel 360o Fixation Collection (WinDB)

###  3.1 WinDB Overview

To leverage the advantages of ERP and HMD, particularly in eliminating blind
zoom, WinDB employs ERP as the foundational computational platform for
fixation collection. Therefore, the key problem is to eliminate the negative
impact of distortion as much as possible. We have illustrated the WinDB’s
overview in Fig. 4, which consists of five steps.

To address the distortion issue, we employ the step A — grid-like spherical-
to-2D projection (see Fig. 5-B) — to equally divide the input ERP image into
sub-regions. Then, each spherical sub-region is projected onto plain 2D image
patches. Thus, after the step A, all local sub-patches are distortion-free,
see the yellow mark ${{\normalsize{\scriptsize{2}⃝}\normalsize}}$ in Fig. 4.
However, there are noticeable “ghost effects” with massive “inter-patch
misalignment” (see the red arrows in Fig. 5-B). Clearly, these artifacts are
unacceptable when collecting fixations since they tend to attract human
fixation.

To improve, we have devised a very tricky step (the step B of Fig. 4) — we use
black narrow lines to block those intersections of neighbored patches, making
the entire image to be covered with a black “mesh screen”. This step is
inspired by a unique mechanism of our human visual system, called Persistence
of Vision (POV [58], see Fig. 6), which implies that our human brain can
automatically restore those blocked narrow regions. Thus, using such “mesh
screen tricky”, we can easily handle the “inter-patch misalignment” (see the
yellow mark ${{\normalsize{\scriptsize{3}⃝}\normalsize}}$ in Fig. 4). More
technical details will be provided in Sec. 3.3.

Next, we shall handle the “ghost effects”. Here we come up with a simple yet
effective solution (the step C of Fig. 4), _i.e._ , we perform “discriminative
vertical blur777Vertical blur: only the left and right sides of a rectangle
patch are blurred, because the adopted spherical-to-2D projection method
performs equal distance sampling over the sphere’s longitude. As a result, the
ghost effects only exist between vertically neighbored patches. For a better
understanding, please see Fig. 5-B.” on each local patch. And the blur degree
is dynamically controlled according to the patch’s position, where patches
near the poles are blurred more. By this step, the ghost effects can be
alleviated significantly; see the yellow mark
${{\normalsize{\scriptsize{4}⃝}\normalsize}}$ in Fig. 4. More technical
details will be given in Sec. 3.4.

However, noticeable ghost effects still exist in some patches, especially
those near the poles (_i.e._ , the top and bottom rows). Hence, we present a
new step (the step D of Fig. 4), where “auxiliary windows” are placed around
the poles. These windows are essentially large-size sub-patches, which are, of
course, distortion-free. By assigning an appropriate window coverage, the
“ghost effects” can be solved completely; see the yellow mark
${{\normalsize{\scriptsize{5}⃝}\normalsize}}$ in Fig. 4. More technical
details will be given in Sec. 3.5.

Further, due to the fact that the auxiliary windows are distortion-free and
more informative than local patches, users are naturally more likely to pay
attention to them. We shall avoid this tendency to ensure an objective
fixation collection. So, we adopt the step E — dynamic blurring, which
gradually blurs those auxiliary windows that have received human-eye
fixations. More technical details will be given in Sec. 3.6.

All the steps mentioned above consist of our WinDB, which is distortion-free,
has no ghost effects, no visual artifacts, no blind zoom, and is user-
friendly.

###  3.2 Grid-like Spherical-to-2D Projection

Generally, representing a panoptic image via ERP can retain the image’s
overall information. However, as shown in Fig.
4-${\normalsize{\scriptsize{1}⃝}\normalsize}$, there exists ERP visual
distortions, where the distortion level becomes more dramatic when closing to
the poles.

Inspired by the limited focal range of our HVS888The HVS has a relatively
large range field of view (FOV) [59], _i.e._ , about $\rm 110^{o}$, however,
the focal range is only about $\rm 25^{o}$ [60] (see Fig. 5-A)., we present
the “grid-like spherical-to-2D projection (Fig. 5-B)” to let the ERP stays
“local distortion-free” — each sub-patch has zero distortions. Although this
“grid-like spherical-to-2D projection” unavoidably brings noticeable inter-
patch misalignments over the vertical direction, we believe that the ERP after
the projection can also be regarded as distortion-free if we can achieve “ _a
precondition_ ” — let our HVS focus on the sub-patches’ inside regions. Here,
we shall leave the question of how to reach this “precondition” to the
following subsections and begin detailing the proposed “grid-like spherical-
to-2D projection”.

As shown in Fig. 5-B, suppose a typical ERP frame (with a degree of freedom:
horizontal 360o and vertical 180o) has been projected on a sphere, we divide
the sphere to slices, where the vertical dividing interval is fixed to $\rm
30^{\rm o}$999The dividing interval is $\rm 30^{\rm o}$ to cover HVS focus
range ($\rm 25^{\rm o}$) [60].. Yet the horizontal interval is dynamic to
ensure the obtained spherical slices have the same grid topology as the ERP
grid. As a result, each “spherical slice (light blue)” strictly correlates to
an “ERP sub-patch (yellow)”, and they have the same vertical size but have
different horizontal sizes, _i.e._ , spherical slices near the poles are
squeezed horizontally, and this is the main reason causing the visual
distortions in ERP.

Figure 5: Detailed illustration of our grid-like spherical-to-2D projection.
Sub-figure A is the HVS focal range; the human eye mainly focuses in a range
of 30 degrees. Sub-figure B is the technical detail and rationale for adopting
the spherical-to-2D projection. See section 3.2 for details.

To make ERP distortion-free, we resort to “spherical sub-windows”, which have
uniform size (_i.e._ , the red box of Fig. 5-B). Due to the limited focal
range of HVS, this size shall be the same as the largest “spherical slice” —
those slices around the sphere’s equator. To be more specific, the sizes of
“spherical slices” range between (horizontal: 0$\sim$30o, vertical: 30o), and
the sizes of “spherical sub-windows” are uniformly assigned to (horizontal:
30o, vertical: 30o). Therefore, the “spherical sub-windows” are generally
larger than “spherical slices”, especially near the poles. So, we can easily
achieve distortion-free projection using “spherical sub-windows”, which
project each “spherical sub-window” to “ERP sub-patch”. Since we already have
the mapping information between “spherical slices” and “ERP sub-patches”, our
“grid-like spherical-to-2D projection” can be realized as:

$\displaystyle\rm Sphere\leftarrow\mathcal{P}_{\rm E2S}(ERP),$ (1)
$\displaystyle\rm\\{SSlices,EPats,\mathbb{M}_{E\rightleftharpoons
S}\\}=\emph{SGrid}(Sphere,ERP),$
$\displaystyle\rm\\{SWindows,\mathbb{M}_{S\rightleftharpoons
W}\\}=\emph{SWindow}(Sphere,SSlices),$ $\displaystyle\rm
ERP^{\star}\leftarrow\emph{Fill}\big{(}EPats,\underbrace{\rm\mathcal{P}_{S2E}(SWindows)}_{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>Free}},\mathbb{M}_{S\rightleftharpoons W},\mathbb{M}_{E\rightleftharpoons
S}\big{)},$

where “ERP” is ERP image, “Sphere” is ERP’s spherical form, “SSlices” denotes
spherical slices, “SWindows” denotes spherical sub-windows, and “EPats”
represents the topology of ERP sub-patches (see Fig. 5-B);
$\rm\mathbb{M}_{E\rightleftharpoons S}$ is the mapping between “ERP sub-
patches” and “spherical sub-slices”, $\rm\mathbb{M}_{S\rightleftharpoons W}$
denotes the mapping between “spherical sub-slices” and “spherical sub-
windows”; $\mathcal{P}_{\rm E2S}$ is a typical projection from ERP to sphere,
and $\mathcal{P}_{\rm S2E}$ projects sphere back to ERP; _SGrid_ $(\cdot)$
performs grid-like dividing on sphere, _SWindow_ $(\cdot)$ divides the sphere
to uniform sub-windows; _Fill_ $(\cdot)$ uses the EPats as the indicator to
reformulate a distortion-free EPR, _i.e._ , ERP⋆ (the final result), which can
be detailed as:

$\displaystyle\rm\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>1:}$
$\displaystyle\rm\emph{f}_{\emph{i}}=\emph{PAlign}\big{(}\mathcal{P}_{S2E}(\rm
SWindows),\mathbb{M}_{S\rightleftharpoons W}\big{)},$ (2)
$\displaystyle\rm\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>2:}$
$\displaystyle\rm\emph{f}_{\emph{i}}=\emph{PAlign}\big{(}\emph{f}_{\emph{i}},\mathbb{M}_{\rm
E\rightleftharpoons S}\big{)},$
$\displaystyle\rm\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>3:}$ $\displaystyle\rm ERP^{\star}=\emph{Reform}\big{(}\underbrace{\\{\cdots,\
\emph{f}_{\emph{i}},\
\cdots\\}}_{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>All}\ \emph{f}}},\rm EPats\big{)},$

where _f_ is a temporary container, _PAlign_ $(\cdot)$ performs pixel-wise
projection according to the given mappings (_i.e._ , $\mathbb{M}$), _Reform_
$(\cdot)$ reformulate all distortion-free patches into a complete ERP map; the
step 1 projects “distortion-free spherical sub-windows” to “spherical slices”,
the step 2 projects “spherical slices” to “ERP sub-patches”, and the step 3
reformulates the obtained “ERP sub-patches” as the ERP⋆. The detailed
definitions of “spherical sub-windows”, “spherical slices”, and “ERP sub-
patches” can be reviewed from Fig. 5-B.

Qualitative demonstrations regarding ERP⋆ can be found in the
${{\normalsize{\scriptsize{2}⃝}\normalsize}}$ of Fig. 4 and Fig. 5-B. Despite
the advantage of ERP⋆ that all local patches are distortion-free, massive
visual artifacts can be easily noticed, _i.e._ , inter-patch misalignments,
and ghost effects (see ${{\normalsize{\scriptsize{2}⃝}\normalsize}}$ of Fig.
4).

###  3.3 Mesh Screen

Recall that we have mentioned “ _a precondition_ ” in Sec. 3.2 that we shall
let HVS focus on inside regions of ERP sub-patches. Here, we present a very
tricky solution to reach this precondition. Besides, this solution can well
alleviate the inter-patch misalignments in ERP⋆ (Eq. 1). That is, we propose
applying an additional “mesh screen” on ERP⋆, see the black mesh screen in the
${\normalsize{\scriptsize{3}⃝}\normalsize}$ of Fig. 4.

Our technical rationales are two-fold. First, the HVS tends to focus on visual
artifacts (see Fig. 6-A), and using the proposed “mesh screen” can shift HVS’s
focal to the inside contents of each ERP sub-patch (see Fig. 6-B). The HVS
usually pays less attention to regular patterns [61], and the proposed mesh
screen is a regular pattern, which could be automatically omitted. Also, we
designed the mesh screen to have the same grid size as the ERP sub-patches,
and thus, all inter-patch misalignments can be fully blocked. So, mesh screen
can eliminate misalignments and focus HVS on ERP’s center regions. Second,
though the mesh screen inevitably results in some information lost — the mesh
screen fully blocks the inter-patch regions, our brain can automatically
restore the overall ERP context. This phenomenon is automatically achieved by
the POV (Persistence of Vision) [58] mechanism, which says that the visual
clues echo in our brain for a while after the visual signals have been lost.
Since our task is in a video environment, the mesh screen won’t actually cause
information loss due to the POV mechanism.

Figure 6:  Schematic illustration of why we should use “Mesh Screen”. Sub-
figure A and B represent the visual information flow without and with a “mesh
screen” in visual input. In the input of sub-figure A, due to inter-patch
misalignments, HVS will continue to focus on inter-patch misalignments, and
the information the brain perceives will also focus on inter-patch
misalignments. In sub-figure B, the inter-patch misalignments are blocked by
the mesh window, which will automatically trigger the brain’s automatic
filling mechanism so that the brain will focus on the event itself. See Sec.
3.3 for details.

Our “mesh screen” consists of two parts, including mesh screen generation
(_i.e._ , generate the “GMask”) and mesh screen deployment (_i.e._ , via
$\odot$), which can be detailed as:

$\displaystyle\hskip 1.79993pt{\rm ERP^{\star\star}}\in\mathbb{R}^{w\times
h}={\rm ERP^{\star}}\odot\underset{\Uparrow}{\underline{\rm{GMask}}},$ (3)
$\displaystyle\hskip 21.49997pt\ \ \ \ \ \ \ \ \ \overbrace{\rm
GMask=\emph{Grids}(ERP,EPats)\in\\{0,1\\}^{\emph{w}\times\emph{h}}}$

where $\rm ERP^{\star}$ can be obtained via Eq. 1, ${\rm ERP^{\star\star}}$ is
the result with mesh screen solution; “EPats” has been defined in Eq. 1 — the
topology information among the ERP sub-patches, _Grids_ $(\cdot)$ extracts the
grid structure, _i.e._ , the GMask101010We use “0” to represent the grid,
whose thickness is 5 pixels. from its input, and $w$, $h$ respectively denotes
the ERP image’s width and height; $\odot$ is element-wise multiplicative
operation.

Using Eq. LABEL:eq:remapxx, the inter-patch misalignments have been addressed
nicely (see ${{\normalsize{\scriptsize{3}⃝}\normalsize}}$ of Fig. 4), and, of
course, the “precondition” has also reached. Next, we shall conquer the
remaining “ghost effects”.

###  3.4 Discriminative Vertical Blur

Although we adopted the mesh screen (Sec. 3.3) to address the inter-patch
misalignment problem, we can notice the “ghost effects” in ERP⋆⋆ (Eq.
LABEL:eq:remapxx), especially in those sub-patches near the poles. The main
reason for causing the ghost effects has been explained above, _i.e._ , the
existence of overlapped regions between two horizontally neighbored “spherical
sub-windows” (see Fig. 5-B), and the interested readers shall refer to Sec.
3.2.

To handle the “ghost effects”, our idea is to blur those overlapped regions of
“spherical sub-windows”. Although this solution looks rude, resulting in some
information loss, it effectively alleviates the ghost effects. The rationale
is quite similar to that of the proposed “mesh screen (Sec. 3.3)”, _i.e._ ,
the POV mechanism can automatically help our brain restore the primary
information of those blurred regions. Also, the HVS can still possibly be
attracted by these blurred regions since the motions, a very critical clue for
attracting fixations, remain noticeable after the blur operation111111The HVS
is extremely sensitive towards movements [61]..

Since the “ghost effects” are more frequent near the sphere’s poles, the
“spherical sub-windows” near the poles shall have larger regions to be blurred
than those near the equator, and that’s why we name it as “discriminative
vertical blur (DVB)121212In our implementation, we leave the two rows which
are closet to the equator unchanged since they tend to have zero ghost
effect.”, the details of DVB can be given as:

$\displaystyle{\rm SWindow}_{i,j}$ $\displaystyle={\rm
Olap}_{i,j}\cup\big{\\{}{\rm SWindow}_{i,j}-{\rm Olap}_{i,j}\big{\\}},$ (4)
$\displaystyle{\rm Olap}_{i,j}$ $\displaystyle=\big{\\{}{\rm
SWindow}_{i,j}\cap{\rm SWindow}_{i,j-1}\big{\\}}$ $\displaystyle\ \ \ \ \ \ \
\ \cup\big{\\{}{\rm SWindow}_{i,j}\cap{\rm SWindow}_{i,j+1}\big{\\}},$
$\displaystyle{\rm SWindow}_{i,j}^{b}$
$\displaystyle\leftarrow\mathcal{B}\Big{(}{\rm
Olap}_{i,j}\Big{)}\cup\Big{\\{}{\rm SWindow}_{i,j}-{\rm Olap}_{i,j}\Big{\\}},$

where “SWindows” denotes spherical window which has been defined in Eq. 1;
$\cap$ is the intersection operation, and $\cup$ is the union operation; ${\rm
Olap}$ denotes the overlapped regions — the regions containing ghosts effects;
$i$ and $j$ respectively denote a SWindow’s row index and column index;
$\mathcal{B}(\cdot)$ is a typical Gaussian blur131313We empirically set the
Gaussian blur as $ksize$=31 and $\sigma$=5.; ${\rm SWindow}^{b}$ is the output
of our DVB.

Then, the overall of our WinDB can now be changed from Eq. 1 to the following
equation:

$\displaystyle\hskip 112.38829pt\rm Sphere\leftarrow\mathcal{P}_{\rm
E2S}(ERP),$ (5) $\displaystyle\hskip
6.82881pt\rm\\{SSlices,EPats,\mathbb{M}_{E\rightleftharpoons
S}\\}=\emph{SGrid}(Sphere,ERP),$
$\displaystyle\rm\\{SWindows,\mathbb{M}_{S\rightleftharpoons
W}\\}=\emph{SWindow}(Sphere,SSlices),$ $\displaystyle\hskip 20.77039pt\rm
ERP^{\star\star\emph{b}}\leftarrow{\rm
Mesh}(\underset{\Uparrow}{\underline{\rm ERP^{\star\emph{b}}}}),$
$\displaystyle\overbrace{\emph{Fill}\Big{(}{\rm
EPats},{\rm\mathcal{P}_{S2E}\big{(}\underset{\Uparrow}{\underline{DVB}}(SWindows)\big{)}},\mathbb{M}_{\rm
S\rightleftharpoons W},\mathbb{M}_{\rm E\rightleftharpoons S}\Big{)}}$
$\displaystyle\hskip
4.83691pt\overbrace{\rm\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>Vertical\ Blur\ (Eq.\ \ref{eq:cccc})}}\vspace{0.2cm}$

where most of the symbols are the same as Eq. 1; Mesh$(\cdot)$ denotes the
proposed mesh screen (Eq. LABEL:eq:remapxx); ERP${}^{\star\star\emph{b}}$ is
the output of Eq. LABEL:eq:mapping2, which can be visualized in
${{\normalsize{\scriptsize{4}⃝}\normalsize}}$ of Fig. 4. As shown, the ghost
effects near the equator have been alleviated significantly, yet some ghost
effects are still noticeable near the poles even after applying the proposed
DVB. Next, we shall further improve it.

###  3.5 Auxiliary Windows

To further handle the ghost effects in ERP${}^{\star\star\emph{b}}$ (Eq.
LABEL:eq:mapping2), we propose to use “auxiliary windows”. Generally, the
concept of auxiliary windows is quite simple, _i.e._ , using multiple tailored
“spherical sub-windows” to facilitate users to watch the panoptic scene’s pole
regions, see ${{\normalsize{\scriptsize{5}⃝}\normalsize}}$ of Fig. 4. The
primary principles of designing “auxiliary windows” include three aspects.
First, for a better watching experience, the adopted auxiliary windows shall
have larger coverage and as much as possible while staying distortion-free.
Second, the auxiliary windows can only occupy a small part of the WinDB’s main
screen to maintain the “global” perceptual of ERP${}^{\star\star\emph{b}}$.
Third, the auxiliary windows shall have less overlapping to alleviate the
ghost effects significantly.

According to these principles, we design the auxiliary windows’ covering scope
as vertical-45o and horizontal-120o. This design is mainly because: 1) the
maximum horizontal range of the distortion-free projection $\mathcal{P}_{\rm
S2E}$ (Eq. 1) is about 120o [44] and the HVS’s focal range is below 120o [62]
(Eq. 5-A), and thus we set the auxiliary windows’ horizontal covering range to
120o; 2) since the severest ghost effects are near the poles, we empirically
set the horizontal covering range as 45o to strike the tradeoff between
retaining global perceptual and suppressing ghost effects. Thus, we have total
$N$ auxiliary windows (see ${{\normalsize{\scriptsize{5}⃝}\normalsize}}$ of
Fig. 4), where the auxiliary windows are distortion-free, placed at the top
and bottom rows to block the ghost effects, and about 70% regions of
ERP${}^{\star\star\emph{b}}$ are still accessible.

The whole process of deploying auxiliary windows (DAW) can be detailed as
follows:

$\displaystyle\rm\hskip
12.80365ptWinDB^{-}\rm=DAW(\underset{\Uparrow}{\underline{AWs}},{\rm
ERP^{\star\star b}}),$ (6) $\displaystyle\hskip 13.08846pt\overbrace{{\rm
AW}_{i}=\mathcal{P}_{\rm S2W}\big{(}\underset{\Uparrow}{\underline{{\rm
SWindow^{+}}_{i}}}\big{)},i\in\\{1,...,N\\}}$ $\displaystyle\overbrace{{\rm
SWindow}^{+}=\emph{SWindow}\big{(}{\rm Sphere},{\rm SSlices}^{+}\big{)},\ {\rm
Eq}.~{}\ref{eq:mapping2}}$

where most of the symbols used have been defined in Eq. LABEL:eq:mapping2, $N$
represents the total number of auxiliary windows; compared with the previous
“SWindows” and “SSlices”, ${\rm SWindows^{+}}$ and $\rm SSlices^{+}$ cover
more spherical regions, _i.e._ , about 6 times; $\rm WinDB^{-}$ represents the
early version of our WinDB (${{\normalsize{\scriptsize{5}⃝}\normalsize}}$ of
Fig. 4) which has multiple advantages, _i.e._ , much alleviated ghost effects
with global perception. Yet it still has some disadvantages, _i.e._ , few
ghost effects induced by the “inevitable overlapping141414Such overlapping is
mainly induced by the strict “1-to-many” correlation between “SWindow+” and
“SSlices.”” between the auxiliary windows, and, especially, fixations tend to
be trapped in auxiliary windows because they tend to be more informative than
the smaller ERP sub-patches. In the next step, we shall handle these
disadvantages to further improve our WinDB.

###  3.6 Dynamic Blurring

Here, we aim at two problems of $\rm WinDB^{-}$ (Eq. LABEL:eq:AW): 1) the
existence of few ghost effects, and 2) trapping HVS fixations. The collected
fixations won’t reflect the regional-wise importance degree if these two
problems are unsolved. Thus, we present the “dynamic blurring” solution, and
the general idea is to blur all auxiliary windows and dynamically clear one
that receives fixations. In this way, the ghost effects can be solved
completely, and the fixations won’t be trapped in auxiliary windows. Further,
since the movements are still noticeable in blurred auxiliary windows, the
blur operation won’t cause much information to be lost, and this phenomenon
has also been mentioned in Sec. 3.3.

Figure 7: Technical details of the proposed auxiliary window with a dynamic
blurring strategy. The auxiliary window changes (from blurred to clear) when
the fixation trajectory sweeps over the auxiliary window. The advantage is
that ghost effects can be solved completely, and the fixation won’t be trapped
in auxiliary windows. See sec. 3.6 for details.

The technical details of the proposed dynamic blurring has been shown in Fig.
7, which cyclicly repeats three sequential status, _i.e._ , B (blurred), C
(clear), and R (re-blurred). The B status: at the beginning of the fixation
collection, all auxiliary windows are blurred (Gaussian blur151515OpenCV
GaussianBlur tool, $ksize$=31 and $\sigma$=5.). The C status: a blurred
auxiliary window would become clear immediately if the fixation trajectory
sweeps over the auxiliary window during the fixation collection process. The R
status: to prevent trapping fixations into auxiliary windows, the “clear
status” of an auxiliary window won’t last long, and our method will “gradually
blur” (last about 2$\sim$3 seconds) the “clear” auxiliary window again). We
can formulate the whole process of dynamic blurring (DB) as follows:

$\displaystyle\rm\hskip 27.0ptWinDB={\rm
DB}\normalsize(\underset{\Uparrow}{\underline{WinDB^{-}}}\normalsize),$ (7)
$\displaystyle\rm\hskip
70.0001pt\overbrace{\big{\\{}\underset{\Uparrow}{\underline{{{\rm
AW}_{i}}}},i\in{\rm\\{1,...,\emph{N}\\}}}$
$\displaystyle\rm\overbrace{\leavevmode\hbox to222.29pt{\vbox
to33.83pt{\pgfpicture\makeatletter\hbox{\hskip 6.87468pt\lower-21.23138pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.54167pt}{-3.41666pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\textbf{B}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{81.74716pt}{-3.41666pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\textbf{C}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{167.036pt}{-3.41666pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{$\textbf{R}$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} { {}{}{}}{}
{{}{}}{{{{{}}{ {}{}}{}{}{{}{}}}}}{{}}{}{}{}{}{{{}{}}}{ {}{}{}}{}{{{{{{}}{
{}{}}{}{}{{}{}}}}}{}{}{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@setlinewidth{0.75pt}\pgfsys@invoke{ }{}{{
{\pgfsys@beginscope\pgfsys@setlinewidth{0.6pt}\pgfsys@setdash{}{0.0pt}\pgfsys@roundcap\pgfsys@roundjoin{}
{}{}{} {}{}{}
\pgfsys@moveto{-1.51497pt}{2.01996pt}\pgfsys@curveto{-1.38872pt}{1.26247pt}{0.0pt}{0.12624pt}{0.37874pt}{0.0pt}\pgfsys@curveto{0.0pt}{-0.12624pt}{-1.38872pt}{-1.26247pt}{-1.51497pt}{-2.01996pt}\pgfsys@stroke\pgfsys@endscope}}
}{}{}{{}}\pgfsys@moveto{170.71655pt}{-6.94966pt}\pgfsys@lineto{170.71655pt}{-19.91684pt}\pgfsys@lineto{0.0pt}{-19.91684pt}\pgfsys@lineto{0.0pt}{-7.62842pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{1.0}{-1.0}{0.0}{0.0pt}{-7.62842pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} {
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{129.35532pt}{-16.34282pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{\footnotesize{Gradual
Blurring, 2$\sim$3s}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}{}{}}{}{ {}{}{}}{
{}{}{}}{{{{}}{{}}{{}}{{}}}{{{{}}{ {}{}}{}{}{{}{}}}}
}{{{{}}{{}}{{}}{{}}{{}}}{{{{}}{ {}{}}{}{}{{}{}}}} }{{}{}}{{}}
{}{}{}{{{}}{{}}{{}}} {{{}}{{}}{{}}}
{}{{}}{}{{}}{}{{}}{}{}{}{}{}{}{}{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@setlinewidth{0.75pt}\pgfsys@invoke{
}{}{}{}{}{{}}\pgfsys@moveto{7.07468pt}{0.0pt}\pgfsys@curveto{34.81699pt}{0.0pt}{50.47185pt}{0.0pt}{77.5354pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{77.5354pt}{0.0pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} {
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{11.8721pt}{3.70801pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{\footnotesize{Receive
Fixations}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{ {}{}{}}{}{ {}{}{}}{
{}{}{}}{{{{}}{{}}{{}}{{}}}{{{{}}{ {}{}}{}{}{{}{}}}}
}{{{{}}{{}}{{}}{{}}{{}}}{{{{}}{ {}{}}{}{}{{}{}}}} }{{}{}}{{}}
{}{}{}{{{}}{{}}{{}}} {{{}}{{}}{{}}}
{}{{}}{}{{}}{}{{}}{}{}{}{}{}{}{}{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@setlinewidth{0.75pt}\pgfsys@invoke{
}{}{}{}{}{{}}\pgfsys@moveto{92.5024pt}{0.0pt}\pgfsys@curveto{120.1905pt}{0.0pt}{135.81488pt}{0.0pt}{162.82423pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{162.82423pt}{0.0pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}} {
}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{88.62479pt}{3.70801pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{\footnotesize{Last 2s
\& No Fixation}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$

where $\rm{DB}(\cdot)$ donates the proposed dynamic blur scheme which deplores
on $\rm WinDB^{-}$ (Eq. LABEL:eq:AW) with _N_ auxiliary windows; ${{\rm
AW}_{i}}$ represents the _i_ -th auxiliary window; “WinDB” presents the final
version of the proposed new panoptic fixation collection approach (see
${{\normalsize{\scriptsize{6}⃝}\normalsize}}$ of Fig. 4).

In summary, our WinDB has the following advantages: 1) blind-zoom free, 2) no
ghost-effects, 3) no inter-patch misalignment, 4) good global perceptual, 5)
almost zero information lost, and 6) user friendly. Thus, based on WinDB, we
can easily collect solid fixations that can correctly reflect the regional-
wise importance of the given panoptic scene. This novel fixation collection
tool has built a solid foundation for the panoptic saliency research field.

##  4 Proposed PanopticVideo-300 Dataset

###  4.1 Why Should We Build This New Set?

In the previous HMD-based datasets [5, 44, 3], we found almost no panoptic
video containing “sudden events” that could result in “fixation shifting” —
driving users’ fixations from one place to another with long spherical
distance. Actually, the “fixation shifting” is a very common event in our
daily life, which could be an important event deserving the HVS’s attention.
However, due to the blind-zoom issue, the fixations collected by HMD are fully
unaware of such fixation shifting phenomenon. Thanks to the advantages of our
WinDB, we can now handle this problem. Thus, based on our WinDB, we shall
construct a new set, named PanopticVideo-300, the most challenging and
comprehensive panoptic saliency detection dataset, containing various complex
scenes with frequent sudden important events.

###  4.2 Video Clip Collection

To construct the mentioned large dataset, we downloaded almost 400 video clips
from YouTube, and nearly 80% of them contain “sudden events”. Then, we removed
about 100 low-quality clips (_e.g._ , scenes with plain backgrounds, simple
movements, or low-resolutions). Thus, there is a total of 300 high-quality
ones retained. It is worth mentioning that, in the previous datasets [3, 4, 5,
6], those clips contain almost no “sudden events” since their fixation
collection is ill-posed. Yet, in facing “sudden events”, the fixations can be
collected by our WinDB correctly. Fig. 8 shows the semantic categories covered
by PanopticVideo-300, _i.e._ , 225 categories.

Figure 8: The semantic categories of PanopticVideo-300 dataset.

###  4.3 Users Fixation Collection with WinDB

Based on our newly proposed panoptic fixation collection approach (_i.e._ ,
WinDB), we have recruited 38 users, including 12 females and 26 males aged
between 18$\sim$29\. All users are completely unfamiliar with the fixation
collection process; of course, no video clips in our video clip pool have been
shown to them before. Since our WinDB approach is HMD-free, each user only
needs to watch the video with a resolution of 1,920$\times$1,080 on the PC,
and a typical eye tracker has also been set up. For each user, the entire
fixation collection process takes about 50 minutes. It could be actively
suspended at any time if the user experienced fatigue or discomfort during
fixation collection. Notice that this HMD-free approach (_i.e._ , WinDB) is
more comfortable than the HMD-based one, not to mention that fixations
collected by our approach are more consistent with the real regional-wise
importance degree in the given panoptic scene.

Figure 9:  Qualitative comparisons between fixation collected by our WinDB
approach, conventional fixation collection methods, and existing datasets
(_i.e._ , VR-Eye Tracking [5], Sports360 [3], and Wild-360 [44]). The fixation
shifting phenomenon has been highlighted via red cycles. See sec. 4.4 for
details.

###  4.4 Advantage Discussions

To demonstrate the advantage of our WinDB against the HMD/ERP-based ones, we
provide some demonstrations in Fig. 9, where we have made three in-depth
discussions.

First, Fig. 9-A illustrates the cases with “sudden events”, where important
and noticeable events suddenly occur in regions far away from the current main
fixations. By comparing these three rows, the fixations collected by our
approach (the 2nd row) are more reasonable than that collected by the HMD-
based method (the 1st row), where our approach can capture those “sudden
events” (highlighted by red cycle) yet the HMD-based method cannot. The reason
is that the sudden events in these cases occurred in the HMD’s blind zoom; the
users with HMD have missed these events. Since our WinDB has no blind zoom,
the users can fully notice all sudden events, ensuring correct fixation
collection.

Second, to verify the advantage of our WinDB against the ERP-based one, we
have shown some normal cases in Fig. 9-B. Without “sudden events”, the
fixations collected by our approach are generally consistent with those
collected by the HMD-based method, showing the correctness of our approach.
Also, compared with the ERP-based method, the fixations collected by our WinDB
are generally compact, yet the ERP-based fixations are scattered. The reason
is clear that the visual distortions in ERP (especially for those regions near
the poles) could easily influence human-eye fixations, drawing them to those
distortion-induced visual artifacts. Since our WinDB is generally distortion-
free, its fixations can be focused on the salient regions.

Third, in the above two discussions, the competitors were “re-realized” by our
team. To further demonstrate the advantage of our PanopticVideo-300 against
other real datasets (_i.e._ , [5, 3, 44]), we have provided some
representative qualitative comparisons in Fig.9-C. Our fixations are more
consistent with the real importance degree of the given panoptic scene, and
the reasons have been explained above (_i.e._ , distortion-free and no-blind
zoom).

Figure 10:  The detailed network architecture of our Fixation Shifting Network
(FishNet). Our FishNet has three major components. Component A focuses on
performing ERP-based global feature embedding to achieve panoptic perception
and avoid visual distortion. B catches fixation shifting in PanopticVideo-300
by refocusing the network to avoid the compression problem of shifted
fixations in SOTA models. C makes the network fully aware of and learns the
fixation shifting behind mechanism to ensure that the network is sensitive to
fixation shifting. See section 5.2 for details.

##  5 The Proposed FishNet

###  5.1 Why Should We Need This New Network?

Now, we have a dataset (_i.e._ , PanopticVideo-300) containing a unique
phenomenon — “fixation shifting”, making this set very challenging to the
existing SOTA fixation prediction methods [51, 11, 54]. Here, we shall brief
the main reasons causing this challenge. First, to fully use the
spatiotemporal information, a critical aspect for the fixation prediction to
suppress false alarms, the SOTA methods [49, 53, 3] designed their networks to
constrain their fixations to stay spatiotemporal smoothness. However, the
“fixation shifting” is generally intermittent, where fixations could suddenly
jump from one place to another with long spatial distance (see Fig. 9), which
is clearly different from “normal fixations” — fixations are spatiotemporally
smooth. This contradiction between these two types of fixations makes the
learning process extremely challenging. So, the “fixation shifting” could get
compressed or omitted if we directly follow the existing network design that
heavily relies on the constraint of staying spatiotemporal smoothness. Second,
almost all previous SOTA methods cannot achieve panoptic global perception,
failing to sense sudden events and eventually resulting in their incompetence
to catch fixation shifting. As the first attempt, we present a fancy network
design, coined as FishNet, to handle fixation shifting, whose key technical
innovations include 1) a fancy “panoptic perception” to perceive sudden events
globally and 2) a brand-new “deformable prober” and “fixation shifting
learning” to handle the jumpable fixations.

###  5.2 Network Overview

As shown in Fig. 10, our FishNet model mainly contains three major components
(_i.e._ , “A panoptic perception” and “B deformable prober”) with a tailored
learning scheme (_i.e._ , “C fixation shifting learning”).

Taking ERP images as input, the “sudden events” could be easily captured by
the network because the ERP-based feature embedding is a global process that
can let the network’s sensing scope cover the entire panoptic scene. So, the
key task for the “A panoptic perception” (see Fig. 10) is to avoid visual
distortions when performing ERP-based global feature embedding, which will be
detailed in Sec. 5.3.

The primary objective of “B deformable prober” is to let the network being
capable of catching “fixation shifting” that prevalent in our
PanopticVideo-300. Most of the existing video fixation prediction networks are
heavily dependent on “spatiotemporal” information, which implies a strong
constraint — the predicted fixations shall stay smooth over spatial and
temporal. So, the shifted fixations are more likely to be compressed by the
existing networks. To improve, our novel “deformable prober” provides an
inspiring way to “refocus” the network on shifted fixations without any side
effects, which will be detailed in Sec. 5.4.

Moreover, to make our FishNet “sensitive” towards the “fixation shifting,” we
shall also let the training process be fully aware of whether the given GTs
have fixation shifting. If fixation shifting exists, the training process
shall automatically learn its behind the mechanism. We realize this objective
via the newly devised “C fixation shifting learning”, which will be detailed
in Sec. 5.5.

###  5.3 Panoptic Perception

####  5.3.1 Technical Rationale

As shown in the bottom-left of Fig. 10, the proposed “panoptic perception” of
FishNet mainly consists of three parts, _i.e._ , D “grid-like spherical-
to-2D”, E “ERP local encoder”, and F “panoptic perceptual decoder”.

The technical rationale of panoptic perception is to achieve distortion-free
global feature embedding. Thus, we first use the “grid-like spherical-to-2D”
that has been explained and detailed in Sec. 3.2 to convert a typical ERP to a
distortion-free version, _i.e._ , ERP⋆ (Eq. 1). Since all sub-patches in ERP⋆
are distortion-free, we propose to treat them as individual inputs to the “ERP
local encoder”, a typical Transformer-based multi-level encoder, to build the
inter-patch relationships. So, the features generated by the “ERP local
encoder” are all distortion-free and with global perception. However, these
features are not well aligned with the original ERP containing redundant
information, which is mainly induced by the “ghost effects” that have been
mentioned and explained in Sec. 3.4. Therefore, we have devised the “panoptic
perceptual decoder”, which performs the conventional multi-level feature
collection, _i.e._ , we use the newly devised “grid-wise projection” to
project the encoder’s features back to the ERP formation. As shown in Fig. 10,
we have performed the “grid-like spherical-to-2D” projection two times to span
a multi-scale “grid” space (_i.e._ , Scale 1 and Scale 2), aiming to be in
line with the subsequent multi-level structure of “ERP local encoder”. After
this process, we can obtain two important mapping information, _i.e._ ,
$\mathbb{M}_{\rm S\rightleftharpoons W}$ and $\mathbb{M}_{\rm
E\rightleftharpoons S}$, which will be used as the indicator to guide the
feature alignments in the “grid-wise projection” part.

####  5.3.2 Technical Details

The feature computation process of the ERP local encoder can be briefed as:

$\displaystyle\rm\hskip
51.21504pt\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>Local\ Encoder}$ (8)
$\displaystyle\\{\textbf{F}_{1,2,3,4}\\}\\!=\\!\overbrace{{\rm
ERPEnC}\big{(}Split(\underset{\Uparrow}{\rm\underline{ERP^{\star}}})\big{)}},$
$\displaystyle\hskip 22.47748pt\overbrace{\underbrace{\rm
GS2E\big{(}ERP\big{)}}\\!\rightarrow\\!\\{\rm
ERP^{\star},\mathbb{M}_{S\rightleftharpoons W},\mathbb{M}_{E\rightleftharpoons
S},EPats\\}}$ $\displaystyle\rm\hskip
5.69046pt\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}{Grid\\!\\!-\\!\\!like\
Spherical\\!\\!-\\!\\!to\\!\\!-\\!\\!2D\ (Eq.~{}\ref{eq:mapping})}$

where F1,2,3,4 denotes four intermediate features obtained via “ERP local
encoder”, the operation _Split_ $(\cdot)$ splits ERP⋆ into individual patches
to be fed as the input of encoder (_i.e._ , ERPEnC); $\mathbb{M}_{\rm
S\rightleftharpoons W}$, $\mathbb{M}_{\rm E\rightleftharpoons S}$ are the
mappings, EPats is the ERP grid’s topology information, which all have been
defined in Eq. 1 and will be used later (_i.e._ , Eq. 9).

As shown in the Fig. 10, the intermediate features have varying sizes with
ranges from ${\frac{\rm W}{4}}$$\times$${\frac{\rm H}{4}}$ to ${\frac{\rm
W}{32}}$$\times$${\frac{\rm H}{32}}$ , in which the _W_ and _H_ is the ERP
size. We have abandoned the F1 because the features from the shallow level
tend to be generally noisy [63, 64]. Thus, only the three of them are actually
used (_i.e._ , F2,3,4) in our “panoptic perceptual decoder”, _i.e._ , to align
these distortion-free & global feature representations with the original ERP
topology. Then, when F2,3,4 arriving at the “panoptic perceptual decoder”,
F2,3 are fed into the “grid-wise projection” to realize the feature mentioned
above alignments. Without undergoing the “grid-wise projection”, the F4 shall
directly serve as the coarse localizer due to its low resolution. So, the
“grid-wise projection” in the “panoptic perceptual decoder (F in Fig. 10)” can
be detailed as:

$\displaystyle\vspace{0.1cm}{\rm G}$ $\displaystyle{\rm
F}={\rm\emph{Fusion}}\big{(}{\rm\emph{Concat}}\big{[}$ (9)
$\displaystyle\rm{DefConv}\big{(}{\emph{Fill}}(EPats,\mathcal{P}_{E2S}(\textbf{F}_{2}),\mathbb{M}^{1}_{S\rightleftharpoons
W},\mathbb{M}^{1}_{E\rightleftharpoons S})\big{)},$ $\displaystyle\ \ \ \ \
\rm{DefConv}\big{(}{\emph{Fill}}(EPats,\mathcal{P}_{E2S}(\textbf{F}_{3}),\mathbb{M}^{2}_{S\rightleftharpoons
W},\mathbb{M}^{2}_{E\rightleftharpoons S})\big{)},$ $\displaystyle\ \ \ \ \ \
\ \ \ \ \rm{DefConv}(\textbf{F}_{4})\big{]}\big{)},\vspace{0.1cm}$

where GF is the obtained global panoptic features,
“Concat$\left(\cdot\right)$” represents the function for channel-level feature
concatenation; EPats, $\mathbb{M}$ can be obtained from Eq. LABEL:eq:feature,
and the superscripts of $\mathbb{M}$ denote different scales; function _Fill_
$(\cdot)$ has been defined in Eq. 1 and detailed in Eq. LABEL:eq:fill;
_Fusion_ $(\cdot)$ is a typical collector which includes convolution, batch-
normalization, and ReLU; notice that “DefConv” denotes deformable convolution
[57], which can further mitigate the tiny misalignments161616There exist some
tiny misalignments when projecting “spherical sub-windows” to “spherical
slices” due to the slight coverage mismatch on the spherical surface. after
the _Fill_ $(\cdot)$ operation.

####  5.3.3 Panoptic Perception v.s. SOTA Solutions

There are three types of SOTA panoptic networks in our research community; we
shall briefly explain our advantages.

1) Bi-stream Networks [48, 49, 24, 44]. This type of network consists of two
research branches (Fig. 11-A), _i.e._ , one branch for the ERP global
information with heavy visual distortions, and another branch for the local
distortion-free views. These two research branches are combined to make the
network’s features global and distortion-free at the same time. The critical
problem of this type of network is that the fusion between these research
branches is extremely challenging — they don’t have explicit alignments,
making the fusion to be an additional feature embedding. Thus, the features
obtained by this approach are inferior to ours.

2) Spherical Convolution-based Networks [11, 15, 51, 52, 53, 3]. Actually,
this type of network has a perfect design, which can simultaneously achieve
global panoptic perception and stay distortion-free (Fig. 11-B). However, the
key problem is that the adopted “spherical convolution” differs from the
existing CNN, making all 2D feature backbones that could provide strong
semantic feature embedding unavailable. Consequently, these types of networks
generally perform poorly without the support of semantic information.

3) Transformer-based Networks [54]. This type of network resorts to additional
CNN layers after each transformer layer, _i.e._ , the CNN-based local
projection, to handle side-effects of ERP’s distortions (Fig. 11-C). The major
problem is that the intermediate features are generated using ERP’s patches
with several visual distortions to original ERP, which cannot take full
advantage of the pre-trained feature backbones.

4) Our Panoptic Perception. As shown in Fig. 11-D, our approach is generally
“independent” to the existing transformers, which can be regarded as a generic
plug-in to handle the visual distortions when generating ERP-based global
features. Notice that our solution enables end-to-end training and testing.
Thus, this plug-in nature enables the network to make full use of the pre-
trained feature backbones [56]. So, the features obtained by our approach are
global, distortion-free, and with strong semantics; all these attributes bring
superior performance against the above-mentioned SOTAs. And these attributes
are also very necessary for handling the “fixation shifting” phenomenon, which
will be detailed next. In a word, our approach is very inspiring, and the
proposed panoptic perception provides a brand-new foundation for feature
computation, which is also the first attempt to bridge the gap between the
conventional 2D research field and the panoptic research field.

Figure 11:  The comparison of technical details between our FishNet and the
SOTA panoptic video saliency learning methods. Sub-figure A, B, and C
represent the SOTA panoptic video saliency learning method, and sub-figure D
is our method. “Ours” stands for our proposed panoptic perception component
(_i.e._ , Sec. 5.3). The “collector” is used to fuse the extracted panoptic
ERP features. See section 5.3.3 for details.

###  5.4 Deformable Prober

####  5.4.1 Technical Rationale

The proposed “deformable prober” has been visualized in the bottom-right of
Fig. 10, consists of two parts, _i.e._ , “G selective feature filter” and “H
shifting aware feature enhancing”.

As we have mentioned in Sec. 5.1, learning to predict the “fixation shifting”
is very challenging because these unique fixations generally conflict with the
“normal fixations”. That is, the normal fixations are spatiotemporally smooth,
yet the shifted fixations are not, which tend to jump from one place to
another with a long spherical distance. Since the “fixation shifting”
phenomenon is quite less than the normal ones171717We designed our
PanopticVideo-300 to include frequent “fixation shifting” on purpose,
primarily because the other datasets almost have zero “fixation shifting”. in
real works, it tends to be overwhelmed by the normal fixations, then be
treated as noises during network training, and get compressed eventually. So,
we shall make the proposed FishNet “fixation shifting aware” — exactly know if
there exists fixation shifting and where they are. Therefore, the key
technical rationale of the proposed “deformable prober” is to adopt the
“selective feature filter” to catch the shifted fixations, and then use the
“shifting aware feature enhancing” to focus on those shifted fixations.

Figure 12:  Detailed calculation of the spherical distance of “Spot P” and
“Spot Q”. Spot P and Q are from the “selective feature filter” component of
FishNet. The main purpose is to measure the spherical distance between two
“spotlights” that belong to two different frames within a short time span.
Latt and Lont+m denote the latitude and longitude of the t-th and {t+m}-th
frame. See section 5.4.2 for details.

####  5.4.2 Technical Details of Selective Feature Filter

To catch the shifted fixations, we shall know which attributes differentiate
them from the “normal fixations”. Generally, there are three attributes that
make the fixation shifting phenomenon special. First, the shifted fixations
shall have strong feature responses. Second, the regions correlating to
“fixation shifting” shall receive the majority of fixations of the scene,
_i.e._ , to be the “spotlight” of the scenes. The reason is clear that the
“fixation shifting” always comes with “sudden events”; without blind zoom, the
users are very likely to be attracted and focused. Third, there shall be a
large spherical distance between temporally neighbored frames’ spotlight
region. _W.r.t._ all these attributes, we design the “G selective feature
filter” of deformable prober as:

$\displaystyle\hskip
91.04872pt\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>Dynamic\ Thresholding}$ (10)
$\displaystyle\Big{\\{}\rm\underset{\Downarrow}{\underline{{Fo}_{1},...,{Fo}_{\emph{u}}}}\Big{\\}}\\!\\!=\\!\mathcal{C}\bigg{(}\bigg{|}\overbrace{\mathcal{M}({\rm
GF})\\!-\\!\mathcal{T}_{d}\\!\times\\!{\rm\emph{max}}\Big{\\{}{\mathcal{M}}({\rm
GF})\Big{\\}}}\bigg{|}_{+}\bigg{)},$ $\displaystyle
Ms\big{\\{}\rm\cdots\big{\\}}\rightarrow{\rm Spot}$

where “$\mathcal{M}(\cdot)$” returns a matrix that represents the channel-wise
mean of its input; GF denotes the features generated via Eq. 9; _max_
$(\cdot)$ is the typical maximum function, and $\mathcal{T}_{d}$ is a pre-
defined hard threshold; using the “dynamic thresholding”, following the 1st
attribute mentioned above, we can easily obtain those regions with high
feature responses, and these regions are very potential to be the “spotlight”
containing shifted fixations; $|\cdot|_{\rm+}$ stands for keeping only the
positive values of the matrix; $\mathcal{C}(\cdot)$ is the connected component
analysis [65], which returns $u$ isolated regions (_i.e._ , Fo); Then, the
“spot” can be localized via the function _Ms_ $(\cdot)$, whose rationale
correlates to the 2nd attribute mentioned above, which returns an isolated
region that has the largest feature response (region’s average).

So, the above process has let us meet the first two attributes of “fixation
shifting”, then we resort to “spherical distance” to satisfy the 3rd
attribute, which measures the spherical distance between two “spotlights” that
respectively belong to two different frames within a short time span. This
process can be formulated as:

$\displaystyle\omega_{t}$ $\displaystyle=\Big{|}\Big{|}\mathcal{P}_{\rm
E2S}\big{(}\digamma(\underset{\Uparrow}{\underline{{\rm
Spot}_{t}}})\big{)},\mathcal{P}_{\rm E2S}\big{(}\digamma({\rm
Spot}_{t+m})\big{)}\Big{|}\Big{|}_{\rm S},$ (11) $\displaystyle\hskip
11.09654pt\overbrace{\rm Spot_{\emph{t}}=\emph{SFF}(GF_{\emph{t}}),\ Eq.\
\ref{eq:Att}}$

where the subscripts t and t+m denote the t-th frame and {t+m} frame, and m
random between {1,2,…,15}181818Since the HVS’s response limit is 500ms, _i.e._
, about 15 frames.; GF can be obtained by Eq. 9, _SFF_ $(\cdot)$ is the
“selective feature filter (Eq. 10)”, and Spott is the obtained spotlight in
the _t_ -th frame; $\digamma(\cdot)$ returns the center coordinates of its
input, _i.e._ , $\rm\\{Lat_{\emph{t}},Lon_{\emph{t}}\\}=\digamma(\rm
Spot_{\emph{t}})$, Lat and Lon respectively denotes the latitude and
longitude; $\mathcal{P}_{\rm E2S}$ is a typical projection from ERP to sphere;
$||\cdot||_{\rm S}$ measures the spherical distance between its inputs, which
can be detailed as:

$\displaystyle||\rm\\{Lat_{\emph{t}},Lon_{\emph{t}}\\},\\{Lat_{\emph{t}+\emph{m}},Lon_{\emph{t}+\emph{m}}\\}||_{\rm
S}=$ (12) $\displaystyle\hskip 8.5359pt\arccos\\!\Big{(}\sin({\rm
Lat}_{t})\\!\times\\!\sin({\rm Lat}_{t\\!+\\!m})+$ $\displaystyle\hskip
28.45274pt\\!\cos({\rm Lat}_{t})\\!\times\\!\cos({\rm
Lat}_{t\\!+\\!m})\\!\times\\!\cos({\rm Lon}_{t}\\!-\\!{\rm
Lon}_{t\\!+\\!m})\Big{)},$

where we have provided a vivid demonstration regarding the computation of
spherical distance, see Fig. 12.

The output of Eq. 11, $\omega$, can reflect the degree of how the 3rd
attribute is satisfied — a larger $\omega$ indicates the higher probability of
Spott to be a region containing shifted fixations.

Figure 13:  Visualizing the “shifting-aware feature enhancing” consists of two
sequential parts: PART 1, enhancing features associated with shifted fixations
by increasing their feature values (_i.e._ , _Lightup_), and PART 2, ensuring
these modified features facilitate the network’s understanding of the actual
“process” of fixation shifting. See section 5.4.3 for details.

####  5.4.3 Shifting-aware Feature Enhancing

In Fig. 10, using the proposed “selective feature filter”, we have already
known which regions in the panoptic scene contain “fixation shifting”, _i.e._
, the Spot (Eq. 10). To let the network focus on these regions, we present the
“H shifting-aware feature enhancing”, whose key idea is to highlight those
features correlating to the shifted fixations. Our idea is somewhat tricky,
which consists of two sequential parts: 1) “light up” all features that
possibly contain shifted fixations, then 2) make these modified features
trainable.

In PART 1, we simply use a very crude scheme to realize the “light up” process
— simply increase the feature values according to $\omega$ (Eq. 11). That is,
given a larger $\omega$, the “Spot’s” feature values shall be increased more.
The rationale of this process is very straightforward, the features that are
crudely increased are, of course, more distinct from others, and thus, the
network could pay more attention to them since they take a large portion of
the overall training loss. However, simply lighting up features as above has a
critical problem, _i.e._ , though the network already knows where the shifted
fixations are, the network could still be “fixation shifting-unaware”. That
is, from the network’s perspective, the modified features are just the same to
the normal ones except having some regions with large values, saying the
networks can learn from these features, yet what they learned is completely
independent of the “fixation shifting process”. So, what we shall let the
network learn is the “process” of fixation shifting, _i.e._ , spotlight focus
shifts from one spherical position to another. Thus, the primary objective of
PART 2 is to achieve this goal. Next, we shall respectively detail parts 1)
and 2), which have also been visualized in Fig. 13.

The GF feature light up process (_i.e._ , the PART 1) can be formulated as:

(13) $\displaystyle\rm GF^{*}\leftarrow\emph{Lightup}(GF,Spot,\omega),$

where the function _Lightup_ $(\cdot)$ positively increases the feature values
according to the spherical distance $\omega$ (Eq. 11) by $\times(1+\omega)$,
and the increased features are indicated by Spot (Eq. 10). This process has
been visually demonstrated in PART 1 of Fig. 13. Then, the PART 2 can be
represented as:

$\displaystyle{\rm\emph{dFlat}}(\underset{\Uparrow}{\underline{\mathbb{A}^{*}}})\rightarrow\Big{\\{}{\rm
EF}_{t},{\rm EF}_{t\\!+\\!m}\Big{\\}},$ (14) $\displaystyle\hskip
16.78702pt\mathbb{A}^{*}=\underset{\Uparrow}{\mathbb{A}}\odot\sigma\big{(}{\rm\emph{Softmax}}(\mathbb{A}\times\mathbb{A}^{\top})\times\mathbb{A}\big{)}$
$\displaystyle\overbrace{\rm\mathbb{A}=\emph{Concat}\big{(}\emph{Flat}(GF^{*}_{\emph{t}}),\emph{Flat}(GF^{*}_{\emph{t}+\emph{m}})\big{)}}$

where _Flat_ $(\cdot)$ flattens its input matrix to a single column, and
_dFlat_ $(\cdot)$ divides its input back to two matrixes; $\odot$ denotes
element-wise matrix multiplicative operation; _Concat_ $(\cdot)$ represents
the concatenation; ${\sigma}(\cdot)$ denotes the sigmoid function;
$\mathbb{A}^{\top}$ denotes the transpose of matrix $\mathbb{A}$; EFt
denotes the enhanced features of the _t_ -th frame. It is worth mentioning
that the key of Eq. 14 is the computation of $\mathbb{A}^{*}$, which follows
the typical co-attention [66] to convert the individual Spots of two different
frames into “a unified shifting process”. The whole Eq. 14 has been shown in
the PART 2 of Fig. 13.

###  5.5 Fixation Shifting Learning

In the above subsections, we have presented a brand-new network tailored for
the “fixation shifting” problem, and the fixation shifting-aware features
(_i.e._ , EF) can be obtained via Eq. 14. However, from the perspective of
network training, using these fixation shifting-aware features alone cannot
ensure good learning towards the fixation shifting phenomenon. Here, we shall
further explain this problem.

Generally, the overall training loss in video-related tasks (_i.e._ , video
saliency [67]) usually covers multiple consecutive 15 frames; the “fixation
shifting” is a sudden process that could be done in just two frames. In this
case, the frames with fixation shifting (_i.e._ , the two frames) only take a
small part of the entire loss, which overlooks the “fixation shifting” process
during training.

Therefore, we present the “fixation shifting learning”, which targets focusing
the loss backpropagation process on those frames with fixation shifting. The
key idea is in line with the “shifting-aware feature enhancing”. Instead of
performing on features, our fixation shifting learning directly modifies the
back propagated training loss, _i.e._ , amplifies the losses of those frames
with shifted fixations. The whole process of “fixation shifting learning” has
been shown at the top of Fig. 10, and the loss function can be expressed as:

$\displaystyle\mathcal{L}oss=$ $\displaystyle\sum_{t}{\mathcal{L}_{\rm
KL}({\rm EF}_{t},\underset{\Uparrow}{\underline{{\rm
GT}_{t}^{*}}})}\\!+\\!\lambda\\!\times\\!\sum{\mathcal{L}_{\text{MSE}}\left({\omega}_{t},{\omega}^{\star}_{t}\right)},$
(15) $\displaystyle\overbrace{\rm
GT^{*}\leftarrow\emph{Lightup}(GT,\underset{\Uparrow}{\underline{Spot^{*}}},\omega^{\star}),\
Eq.~{}\ref{eq:ligu}}$ $\displaystyle\hskip 49.79231pt\overbrace{{\rm
Spot^{*}}=MS\\{\underset{\Uparrow}{\cdots}\\}}$ $\displaystyle\hskip
8.2511pt{\emph{Clustering}({\rm GT})}\rightarrow\Big{\\{}{\overbrace{{\rm
Fo}_{1},...,{\rm Fo}_{\emph{u}}}}\Big{\\}}\vspace{-2pt}$

where GT is the real fixations, $\mathcal{L}_{\rm KL}$ is the KL divergence
loss [68], $\mathcal{L}_{\rm MSE}$ is the MSE loss [53], $\rm EF_{\emph{t}}$
can be obtained via Eq. 14, ${\omega}$ and ${\omega}^{\star}$ denote the
spherical distances respectively on the features and the GT; {Fo} denotes the
fixation clusters generated via _Clustering_ $\cdot$ (_i.e._ , the classic
DBSCAN [69]), and the function _MS_ select one fixation cluster from {Fo}, and
the selected cluster shall have the largest number of fixations.

##  6 Experiments

This experimental section covers two aspects. Firstly, we provide relevant
details about the WinDB approach and the PanopticVideo-300 dataset. We also
conduct comparative experiments using fixation data collected with HMD to
validate our method. Moreover, we performed a user study to demonstrate the
superiority of our WinDB. We also provided the generic dataset analysis to
demonstrate the field boost of our WinDB approach and PanopticVideo-300
dataset. Secondly, we present the benchmark results of our fixation shifting
network, FishNet, on the PanopticVideo-300 dataset. We evaluated its
performance through quantitative and qualitative comparisons with SOTA
methods. Additionally, we conducted an ablation study on different components
to verify the effectiveness. We summarized key parameters mentioned in the
previous technical details, shown in Table I.

TABLE I: Details of key parameters used in our WinDB and FishNet.
---

###  6.1 Experiments of WinDB and PanopticVideo-300

####  6.1.1 Platform and Hardware

Our WinDB uses Tobii Eye Tracker (v2) to collect panoptic fixations. To verify
the effectiveness of our fixation collection method, we also use HMD to
collect fixations as the references, where we use the HTC VIVE PRO EYE with
7invensun a-Glass eye tracker. We use a PC with an NVIDIA RTX 3090 GPU for the
representative models’ training and testing.

Figure 14:  Technical details of dividing our PanopticVideo-300 into “blind
group” and “ordinary group”. See Sec. 6.1.2 for details.

####  6.1.2 Dataset Split

Since our data set contains “blind zoom” and “ordinary” scenes, for the
convenience of subsequent experimental “user study” and “generic analysis”, we
divide the video clips in our PanopticVideo-300 into two groups: 1) clips with
fixation shifting, also called the “blind group”, and 2) clips without
fixation shifting, named as “ordinary group”. We measure the maximum fixation
shifting distance for every 15 frames191919Since the HVS’s response limit is
500 ms, _i.e._ , about 15 frames. to determine if a given clip contains
fixation shifting; for every 15 frames, we compute the spherical distance for
any two frames. We can obtain a 15$\times$15 diagonal distance matrix for each
clip. If the maximum fixation shifting distance is below a pre-defined
threshold (we set this threshold to $\rm 110^{o}$ according to the maximum FOV
of our HVS [1, 70]), we will regard this clip as a blind one 202020We compute
the spherical distance for any two frames in every 15 frames. We can obtain a
15$\times$15 diagonal spherical distance matrix for each clip, and the maximum
element is considered. Each one in the blind zoom group MUST contain at least
one shifting clip.; otherwise, we label it as an ordinary one. This detailed
process has been shown in Fig. 14. Thus, all 300 clips in our set can be
divided into 195 “blind zoom” clips and 105 “ordinary” clips.

TABLE II: Components quantitative evaluations in Fig. 4 to verify if all parts
adopted in WinDB are effective. See Sec. 6.1.3 for details.
---

####  6.1.3 Correctness of Our WinDB Approach

The main advantage of our WinDB collection approach against the classic HMD-
based one is to handle the blind zoom limitation. Thus, for those ordinary
panoptic scenes (_i.e._ , _without_ blind zooms), the fixations collected by
our approach should stay consistent with those fixations collected by the HMD-
based method, and this aspect can verify the soundness of our WinDB.

To demonstrate the effectiveness of each technical step used in our WinDB
(_i.e._ , A - E in Fig. 4), we conducted a component evaluation, where each
component recruited no overlap of 10 users to provide fixations and was
quantitatively tested by measuring four widely-used metrics (AUC-J, SIM, CC,
NSS [12, 49, 71, 72]). The ERP-based and HMD-based fixations are newly
collected here to serve as the references212121Since the fixation shifting
phenomenon is very rare in the existing HMD-based datasets, we have to perform
HMD-based fixation collection in our new PanopticVideo-300 for the
quantitative verification.. Notice that we have randomly selected ten clips
from the “ordinary” group (_i.e._ , all the panoptic scenes have no blind
zoom) as a small validation set.

The quantitative results have been shown in Table II. As can be seen, compared
with the ERP-based fixations collection approach (marked by O), our WinDB
method (marked by E) can achieve significant performance improvement. We can
also notice that the overall performance can get promoted once a critical
component has been applied, showing the necessity of each technical component.
_B.t.w._ , we may notice that some quantitative numerics are not very high.
The main reason is that the HMD-based fixations are not the perfect “ground
truths”, _i.e._ , the “blind zoom group” and “ordinary group” split is not
perfect since it is just based on an empirical threshold (110∘), resulting
limited numeric scores.

####  6.1.4 User Study

We have conducted a user study to further verify the overall quality of
fixation maps collected by our WinDB method against those collected by the
HMD-based methods. The overall experiment setting is shown in Fig. 15. In this
experiment, we have randomly selected 16 video clips from the “blind group”
(Fig. 14). To conduct this user study, we recruited 30 users equally divided
into three groups (see the groups A, B, and C in Fig. 15). Subject groups A
and B are the experimental groups that provide HMD-based fixations and our
WinDB-based fixations. Then, we show the local salient views of each clip, in
which salient views are automatically selected using the same scheme mentioned
above, to the subject group C, and each subject in group C will provide an
overall score ranging between 0$\sim$9 to each clip. Notice that each clip
will be shown to users in group C three times. The 1st time is the plain ERP
version to let the users become familiar with the overall contents in advance.
The 2nd and 3rd are randomly shown clips with salient views selected by either
the HMD-based fixations or our WinDB fixations. Meanwhile, when showing clips
(with all regions blurred except the salient view) to users in group C, we
collect each subject’s fixations since a higher quality of salient views
should receive more fixations when users watch it. Thus, we can obtain two
indicators after this user study, _i.e._ , 1) the subjective quality scores,
and 2) the fixation point numbers in salient views. Notice that a good view
can attract more subject’s fixations and receive a higher quality score. These
two indicators’ results are shown in Fig. 16, where the right part is the
subjective quality scores, and the left is the fixation point numbers. As
shown in the figure, our method can significantly outperform the HMD-based
method in both indicators, where salient views determined by our method can
receive more fixations and higher quality scores, verifying the superiority of
our WinDB approach.

Figure 15:  Details of the subjective user study. The user study is divided
into three groups (_i.e._ , A, B, and C), with ten users in each group. Group
A collects gaze data and locates salient views using HMD. Group B collects
gaze data and locates salient views using our WinDB. Group C scores the
salient view generated by Groups A and B (Task 1). During the scoring process
of Task 1, Group C’s gaze data was collected without their informed consent
(Task 2). See Sec. 6.1.4 for details.  Figure 16:  User study results. The
left part shows the difference in fixation numbers in salient views selected
by our WinDB and the HMD-based approach. The right part illustrates the sum of
scores assigned by users after respectively experiencing our WinDB approach
and the HMD-based approach. These two results suggest that our approach is
more favorable than the HMD-based one. See Sec. 6.1.4 for details.

####  6.1.5 Generic Analysis

This experiment targets to verify if fixations collected by our WinDB approach
can promote the existing panoptic fixation prediction models, where we choose
the three most representative models (SpCNN [73], SalGAN [74], and SalEMA
[75]). Our rationales are two-fold. (1) In “ordinary” panoptic scenes, if the
fixations collected by our WinDB approach are adaptive to the fixations
collected by the HMD-based method, a panoptic fixation prediction model
trained on the HMD-based fixations only could get significant performance
improvement once including our fixations into the training set. (2) The
fixations collected by our WinDB approach shall be able to let a panoptic
fixation prediction model handle “blind” panoptic scenes well. To verify these
two aspects, we have conducted the experiments in Table III.

We choose the existing HMD-based VR-EyeTracking dataset [5] as the baseline.
Then, we randomly select 50 clips with “ordinary” scenes as A1 and randomly
select 50 clips with “blind” scenes as A2. Similarly, from our
PanopticVideo-300, we randomly select 50 clips with “ordinary” scenes as B1
and 50 clips with “blind” scenes as B2. Meanwhile, from our set, we randomly
select 30 clips with “ordinary” scenes as C1 and 30 clips with “blind” scenes
as C2. The formulated A1, A2, B1, and B2 will serve as the training set to
train the adopted three SOTA models; then these models will be tested on C1
and C2 respectively. Notice that there exists no intersection between these
splits.

TABLE III: Quantitative evaluation to verify if PanopticVideo-300 is adaptive
to the existing HMD-based fixations [5], and please details in corresponding
paper context (Generic Analysis). The B1+B2 split denotes the combination of
the ordinary group and the blind group; the corresponding tendency can be seen
in Table IV. See Sec. 6.1.5 for details.
---

By comparing mark ➊ with mark ➍ in the table, we can easily notice that models
trained on “ordinary” scenes could generally perform worse on “blind” scenes.
This is because these models have never learned the fixation shifting
phenomenon when taking A1 as the training set only. Then, we have tested the
three models by using A1+A2 as the training set, and the testing results on C1
and C2 are still the same tendency (see mark ➊ _vs._ ➋, and ➍ _vs._ ➎), _i.e._
, numerics in C2 are generally worse than that in C1. Also, as expected, we
have noticed that the SOTA models’ performances on both C1 and C2 have been
promoted after using the additional training data. The reason is that the
initial 50 clips are far from enough for models to achieve good performances,
and frames in C2 do not completely belong to the fixation shifting case.
Finally, we have tested the models’ performances by respectively adding B1 and
B2 to the training set A1. By comparing ➌ and ➏, we can easily notice a
significant performance improvement, implying that the added B2 is very
effective in enabling models to tackle the “blind” panoptic scenes.

In summary, the experiment conducted above can ensure: 1) our
PanopticVideo-300 is adaptive to the existing HMD-based sets, and 2) models
trained on set without containing any “blind” scenes cannot perform well in
scenes with fixation shifting — a very common phenomenon in real works. Thus,
our PanopticVideo-300 is a necessary complementary part to the existing ones.

###  6.2 Experiments of FishNet

####  6.2.1 Implementation details of FishNet

Our FishNet model uses Transformer [56] as “ERP local encoder” (Sec. 5.3),
implemented with SGD optimizers in the PyTorch framework. The learning rate
was set to 1e-4, and a batch size of three was used for 11 epochs. Each video
frame is resized to a resolution of W$\times$H (768$\times$384) and divided
into patches of size lon$\times$lat (15o$\times$15o). During training, we
construct each training instance consisting of two frames, _e.g._ , the _t_
-th frame and the {_t_ +_m_}-th frame in Fig. 10, where _m_ random between
{1,2,…,15}.

Figure 17:  Qualitative comparisons between our model (_i.e._ , FishNet and
FishNet-) and SOTA models (_i.e._ , SalEMA [75], SalGAN [74], and SpCNN [73])
on the PanopticVideo-300. FishNet- denotes a version of FishNet that includes
only the “panorama perception” component, excluding the “deformable prober”
and “fixation shifting learning” components. All models were retrained using
the training set of PanopticVideo-300and tested on the corresponding testing
set. See Sec. 6.2.3 for details.  TABLE IV: Quantitative comparisons between
FishNet and SOTA models on our set. +p/–p: with/without equatorial prior;
ATsalAtt-I/V, ATsalImg/ATsalVideo: ATSal attention/expert model trained on
image/video set [76, 5]; TE, RT, TR: tested/retrained/trained the model on our
set. ‘*’ indicates that the method does not load a pre-trained model, while
‘#’ indicates that the method is a salient object detection method. See Sec.
6.2.3 for details.
---

####  6.2.2 Quantitative Comparisons

Table IV presents our FishNet (see Fig. 10) and some SOTA panoptic fixation
prediction models’ performances on our PanopticVideo-300. We have divided our
set into training and testing sets, where the training set contains 240 clips
and the rest are testing set222222The detailed training and testing split can
be found on our GitHub https://github.com/guotaowang/PanopticVideo-300..

The compared methods here include GBVS360, BMS360 [77],
GBVS232323https://github.com/Pinoshino/gbvs., ATsal [49], SalEMA [75], SalGAN
[74], SpCNN [73], SAVT [78], and TMFI [79]. Different from conventional 2D
fixation prediction community or 2D salient object detection community, which
tend to release their codes publicly, only two panoptic fixation prediction
models’ codes are made publicly trainable, _i.e._ , SalGAN [74], SpCNN [73].
Also, we have reached to the authors of SAVT [78], and they helped us to
retrain the model. Others’ codes are neither available nor retrainable. To
ensure a fair comparison, those models with available codes are all retrained
on our training set, marked by “RT or TR” in the “Train” column. As shown in
Table IV, our FishNet model has achieved the best performance, indicating that
the adopted “deformable prober” and “fixation shifting learning” is useful in
handling the fixation shifting problem. Because our set can be adaptive to the
existing HMD-based sets, which have been verified above (Sec. 6.1.5), it is
still possible for other 360 models to achieve good performance gain,
demonstrating the generic advantage of our PanopticVideo-300 again.

####  6.2.3 Qualitative Comparisons

Fig.17 illustrates the visual results of FishNet, FishNet- (_i.e._ , a version
of FishNet that includes only the “panorama perception” component, excluding
the “deformable prober” and “fixation shifting learning” components), and
three representative SOTA models (_i.e._ , SalEMA [75], SalGAN [74], SpCNN
[73]) on PanopticVideo-300. As shown in the figure, compared to the FishNet-
and other SOTA models, our proposed FishNet effectively focuses on objects
that fixation shifted, providing evidence for the effectiveness of our
“deformable prober” (Sec. 5.4) and “fixation shifting learning” (Sec. 5.5)
component of FishNet. For example, in line 1 and line 3 of Fig.17, our FishNet
can accurately focus on the sudden appearance of “cat” and “wingsuit flying”,
while other models without the capability of perceiving fixation shifting fail
to change the fixations to the sudden events. In addition, our FishNet-, with
its powerful global perception and local distortion-free ability, can catch
potential salient events in panoptic videos, _i.e._ , the “black leopard” in
line 5 and the “aircraft pilots” in line 2 of Fig.17.

###  6.3 Effectiveness Evaluation on Different Components

####  6.3.1 Effectiveness of the Panoptic Perception of FishNet

To verify the effectiveness of the proposed “panoptic perception” component of
FishNet, we designed three different implementations, as detailed in Table V.

First, O “No pre-trained parameters” corresponds to the spherical convolution
network [51, 52, 53, 3], where the convolution kernel is defined on the sphere
to achieve distortion-free processing. However, due to the absence of pre-
trained spherical convolution model parameters, loading pre-trained parameters
is impossible. Consequently, for the sake of fairness in comparison, our
proposed “panoptic perception” component does not utilize pre-trained models
and is labeled as O.

Second, A “Transformer with local projection” [54] incorporates CNN-based
local projection after each transformer layer to handle the ERP’s distortions.
Consequently, we made similar modifications and retrained our proposed
“panoptic perception” component accordingly.

Third, B “Panoptic Perception” represents a completely transformer-independent
approach that enables global panoptic perception and local distortion-free.
This approach can effectively leverage the pre-trained transformer model [56]
in the 2D domain. To maintain consistency with the previous two
implementations (_i.e._ , O and A), we retained only the “panoptic perception”
component in FishNet (_i.e._ , B). It is important to note that we re-trained
on our training set and re-tested on our test set for all three
implementations.

1) When comparing line 1 and line 3 in Table V, we can observe that loading
pre-trained parameters (_i.e._ , B panoptic perception) leads to an
approximate 3%$\sim$10% performance improvement compared to O. This is because
loading pre-trained weights can essentially be regarded as introducing
additional training data, resulting in a more powerful semantic feature
representation, _e.g._ , the pre-trained Transformer [56] was trained on the
ImageNet dataset, which consists of 1.28 million training images from 1,000
categories with strong distinctive semantics.

2) By comparing line 1 with line 2 in Table V, we can find that, by leveraging
the pre-trained parameters and “local projection ($\mathcal{P}_{\rm S2E}$)”, A
transformer with local projection achieves a significant performance
improvement (+4%) compared to O. However, when comparing line 2 and line 3,
there is an 8% gap in CC between A transformer-based local projection and B
our proposed panoptic perception. This is mainly because the adopted plain
$\mathcal{P}_{\rm S2E}$ of A can alleviate the side effects induced by the
ERP’s visual distortions. However, there is another clear side-effect, _i.e._
, the pre-trained Transformer network parameters are not fully used.

3) In line 3 in the Table V, we observe a significant performance boost in B
panoptic perception. This improvement is attributed to its independence from
the transformer, making it a universal plug-in. Thus, it can achieve global
perception without any network modifications. Thus, the pre-trained
Transformer network parameters [56] can be fully used.

####  6.3.2 Effectiveness of Deformable Prober

In Sec. 5.4, we proposed a “deformable prober” to enhance the FishNet’s
learning ability for fixation shifting. It can capture and enhance the shifted
fixation feature and prevent the fixation shifting feature from being
compressed as noise during training. As shown in Fig. 10, it contains two
parts: 1) “selective feature filter” and 2) “fixation shifting aware feature
enhancement”. As shown in Fig. 13, the “fixation shifting aware feature
enhancement” also contains two sequential parts: PART 1 — “light up features
that possibly contain shifted fixations by increasing their feature values”,
and PART 2 — “ensuring these modified features are trainable”.

TABLE V: Quantitative evidence of component studies for FishNet. (See Sec.
6.3)
---

As shown in Table V, by comparing line 3 to line 4, we observe that C
shifting-aware feature enhancing without _Lightup_ leads to an approximate 1%
improvement compared to B panoptic perception. In other words, the C shifting-
aware feature enhancing without _Lightup_ can also be used for fixation
learning. Although there is no specific component designed for perceptual
fixation shifting, fixation learning can still be achieved with the assistance
of the inter-frame similarity relation matrix, albeit with less ideal
performance.

Furthermore, when comparing line 3 to line 5 in Table V, we notice that our
designed D selective feature filter with _Lightup_ yields an approximate 2%
performance improvement compared to B panoptic perception. The primary reason
for this improvement is that the introduction of _Lightup_ can enhance the
feature response associated with fixation shifting, thereby enhancing the
network’s capability to perceive fixation shifting. However, the enhanced
feature remains isolated without establishing inter-frame correlation.

Finally, when comparing line 3 and line 6 in Table V, we can find that our
proposed E deformable prober (comprising selective feature filter and fixation
shifting aware feature enhancement) can bring about a performance improvement
of about 3%$\sim$4% compared to B panoptic perception. The key reason behind
this enhancement is that the deformable prober enables our network to perceive
fixation shifting, enhances the fixation-shifted features, and establishes
learnable inter-frame feature enhancement at the network level.

TABLE VI: Ablation study on the FishNet’s
input size, _i.e._ , the input
ERP’s size (Sec. 5.3).

---

TABLE VII:  Ablation study on the FisNet’s
sub-patch size of
ERP⋆ (Sec. 5.3).

---

TABLE VIII: Ablation study on the balancing factor used in fixation shifting
learning (Sec. 5.5).

---

TABLE IX: Ablation study on $\mathcal{T}_{d}$ used in Technical Details of
Selective Feature Filter (Sec. 5.4.2).

---

####  6.3.3 Effectiveness of Fixation Shifting Learning

In Sec. 5.5, we presented the “fixation shifting learning”, which aims to
focus the network training process towards fixation shifting by emphasizing
the loss associated with frames containing shifted fixations.

As demonstrated in the lines 6 and 7 of Table V, we found that our proposed F
“fixation shifting learning” leads to an approximate 2% improvement compared
to E deformable prober. Furthermore, when comparing line 3 and line 7 of Table
V, we observe that our designed fixation shifting related component (_i.e._ ,
deformable prober and fixation shifting learning) leads to an approximate 4%
performance improvement compared to the proposed panoptic perception
component. Since most video clips in the PanopticVideo-300 contain fixation
shifting scenes, achieving a substantial improvement depends on effectively
addressing the primary challenge — fixation shifting. Therefore, the
substantial performance improvement reported in the table demonstrates the
effectiveness of our fixation shifting learning. This approach enables the
FishNet network to focus better on fixation shifting learning during the
training process.

###  6.4 Ablation Study

####  6.4.1 The Size of Panoptic Perception Input

In Sec. 5.3, we introduced the “grid-like spherical-to-2D (Eq. 1)” to address
ERP distortion (see D in Fig. 10) and convert ERP to ERP⋆, where the visual
distortions can be solved and the global information has been well retained.

During this process, there are two “sizes” that could influence the overall
network performance of FishNet, _i.e._ , the FishNet’s input ERP’s size, and
the FishNet’s sub-patch size in ERP⋆. So, here we perform two ablation studies
to determine the optimal choices of these two sizes, and the quantitative
results have been reported in Table 6.3.2 and Table 6.3.2.

As shown in Table 6.3.2, the performance improves (+1%) when the input ERP
width increases from 512 to 768. This is mainly because the larger ERP input
can capture richer details and contextual information, which helps to
understand the panoptic content. However, further increasing the input width
from 768 to 1024 results in model performance degradation (-2%). This is
because larger network input implies additional information to be considered
during the learning, complicating the learning process and degenerating the
overall performance.

We also tested different patch sizes, and the corresponding results are
reported in Table 6.3.2. The table shows that the best choice is
15o$\times$15o. Other choices could lead to some performance degradation. The
reason is that all other parameters are selected based on our default sub-
patch size — 15o$\times$15o. Though using other choices could hurt the overall
performance, the decreasing degree is marginal, further showing our approach’s
robustness.

####  6.4.2 Balancing Factor of Fixation Shifting Learning

In Eq. 15, we introduced a factor $\lambda$ to balance the impact of fixation
shifting loss $\mathcal{L}_{\text{MSE}}$ and fixation prediction loss
$\mathcal{L}_{\text{KLD}}$. Gradually increasing the value of $\lambda$ makes
the FishNet pay more attention to the fixation shifting. According to Table
6.3.2, we observe an improvement (+2%) in model performance when gradually
increasing $\lambda$ from 1 to 5. However, as $\lambda$ increases
(5$\rightarrow$7), the performance degrades (-3%). This phenomenon is quite
reasonable because a very large $\lambda$ could make the network over-
emphasize learning the fixation shifting, and a small $\lambda$ could do the
opposite. And the assign $\lambda=5$ exactly strikes the optimal trade-off.

####  6.4.3 $\mathcal{T}_{d}$ Used in Selective Fixation Filter

In the Sec. 5.4.2 of “selective feature filter”, we utilize a dynamic
threshold ($\mathcal{T}_{d}$) to extract “Spot” (_i.e._ , Eq. 10). The dynamic
threshold is crucial for the quality of the Spot, because it directly
determines whether the resulting Spot can represent the area where fixation
shifting is most likely to occur in the current frame. We have tested multiple
choices of $\mathcal{T}_{d}$. In Table 6.3.2, we observe that increasing
$\mathcal{T}_{d}$ (_i.e._ , 0.2$\rightarrow$0.4) improves the performance
(+2%). This is because the higher threshold accurately identifies the regions
where fixation shifting is most likely to occur. However, the performance
degrades (-4%) as $\mathcal{T}_{d}$ increases (_i.e._ , 0.4$\rightarrow$0.8)
since some regions containing shifted fixations could get filtered.

##  7 Limitations

Our WinDB approach effectively addresses the “blind zoom” issue in HMD
fixation data collection. However, the human-designed “dynamic auxiliary
window” (Sec. 3.6) in WinDB may introduce minor biases to user attention. For
example, as shown in Fig. 18-(a), the salient event consistently remains
within the yellow auxiliary window, often in a static state. As a result, the
user tends to continuously fixate on the salient event, leading to the dynamic
blur of the auxiliary window. Although the auxiliary window quickly becomes
clear when it detects user interest, it may slightly affect fixation data
collection. Additionally, our FishNet ensures the perception of fixation
shifting but focuses on a single salient event of each frame. In extreme
cases, as shown in Fig. 18-(b), limited discrimination between multiple
potential spotlights of each frame can lead to fixation shifting misjudgment.
This is because various regions exhibit high feature responses; these regions
can potentially be the “spotlight”, and minor changes between them can
influence the generated “spot” (Eq. 10). In this case, FishNet may
inaccurately perceive fixation shifts, limiting its performance in fixation
shifting learning.

Figure 18:  The limitations of the proposed WinDB and FishNet, see Sec. 7 for
more details.

##  8 Conclusion

In this paper, we have made three distinguished innovations. First, we have
pointed out one critical limitation of the widely-used HMD-based panoptic
fixation collection method, _i.e._ [5, 3, 6, 4], it suffers from “blind
zooms”, making the collected fixations not suitable in real works. Thus, we
have presented an HMD-free panoptic fixation collection approach named WinDB,
which is more economical, comfortable, and, more importantly, technically
correct. Then, using our WinDB approach, we constructed a large set
PanopticVideo-300, containing 300 clips with over 225 semantic categories.
Compared to the existing set [5, 44, 3], our PanopticVideo-300 contains 195
clips with fixation shifting. The proposed WinDB approach and the
PanopticVideo-300 have a large potential to promote the panoptic fixation
prediction tool to a new era. Finally, we have devised a simple yet effective
FishNet model to handle the fixation shifting issue. We verified the
effectiveness and necessity of all technical components adopted in the paper
through extensive objective and subjective experiments.

## References

  * [1] H.-N. Hu, Y.-C. Lin, M.-Y. Liu, H.-T. Cheng, Y.-J. Chang, and M. Sun, “Deep 360 pilot: Learning a deep agent for piloting through 360 sports videos,” in _CVPR_ , 2017.
  * [2] V. Sitzmann, A. Serrano, A. Pavel, M. Agrawala, D. Gutierrez, B. Masia, and G. Wetzstein, “Saliency in vr: How do people explore virtual environments?” _TVCG_ , vol. 24, no. 4, pp. 1633–1642, 2018.
  * [3] Z. Zhang, Y. Xu, J. Yu, and S. Gao, “Saliency detection in 360 videos,” in _ECCV_ , 2018, pp. 488–503.
  * [4] Y. Zhang, F.-Y. Chao, W. Hamidouche, and O. Deforges, “Pav-sod: A new task towards panoramic audiovisual saliency detection,” _TOMM_ , 2022.
  * [5] Y. Xu, Y. Dong, J. Wu, Z. Sun, Z. Shi, J. Yu, and S. Gao, “Gaze prediction in dynamic 360 immersive videos,” in _CVPR_ , 2018.
  * [6] M. Xu, Y. Song, J. Wang, M. Qiao, L. Huo, and Z. Wang, “Predicting head movement in panoramic video: A deep reinforcement learning approach,” _TPAMI_ , 2018.
  * [7] K. Aberman, J. He, Y. Gandelsman, I. Mosseri, D. Jacobs, K. Kohlhoff, Y. Pritch, and M. Rubinstein, “Deep saliency prior for reducing visual distraction,” in _CVPR_ , 2022.
  * [8] L. Jiang, Y. Li, S. Li, M. Xu, S. Lei, Y. Guo, and B. Huang, “Does text attract attention on e-commerce images: A novel saliency prediction dataset and method,” in _CVPR_ , 2022.
  * [9] G. Wang, C. Chen, D.-P. Fan, A. Hao, and H. Qin, “From semantic categories to fixations: A novel weakly-supervised visual-auditory saliency detection approach,” in _CVPR_ , 2021.
  * [10] A. Tsiami, P. Koutras, and P. Maragos, “Stavis: Spatio-temporal audiovisual saliency network,” in _CVPR_ , 2020.
  * [11] Y. Djilali, T. Krishna, K. McGuinness, and N. OConnor, “Rethinking 360deg image visual attention modelling with unsupervised learning.” in _ICCV_ , 2021.
  * [12] Y. Zhu, G. Zhai, Y. Yang, H. Duan, X. Min, and X. Yang, “Viewing behavior supported visual saliency predictor for 360 degree videos,” _TCSVT_ , vol. 32, no. 7, pp. 4188–4201, 2021.
  * [13] H. Yun, S. Lee, and G. Kim, “Panoramic vision transformer for saliency detection in 360 videos,” in _ECCV_ , 2018.
  * [14] A. Nguyen, Z. Yan, and K. Nahrstedt, “Your attention is unique: Detecting 360-degree video saliency in head-mounted display for head movement prediction,” in _ACM MM_ , 2018.
  * [15] Y. Zhu, G. Zhai, X. Min, and J. Zhou, “The prediction of saliency map for head and eye movements in 360 degree images,” _TMM_ , vol. 22, no. 9, pp. 2331–2344, 2019.
  * [16] T. Cohen, M. Geiger, J. Kohler, and M. Welling, “Spherical cnns,” _ICLR_ , 2018.
  * [17] C. Jiang, J. Huang, K. Kashinath, P. Marcus, and M. Niessner, “Spherical cnns on unstructured grids,” _ICLR_ , 2019.
  * [18] Y.-C. Su and K. Grauman, “Kernel transformer networks for compact spherical convolution,” in _CVPR_ , 2019.
  * [19] D. Chen, C. Qing, X. Lin, M. Ye, X. Xu, and P. Dickinson, “Intra- and inter-reasoning graph convolutional network for saliency prediction on 360 images,” _TCSVT_ , 2022.
  * [20] J. Li, L. Han, C. Zhang, Q. Li, and Z. Liu, “Spherical convolution empowered viewport prediction in 360 video multicast with limited fov feedback,” _TOMM_ , 2022.
  * [21] Y. Lee, J. Jeong, J. Yun, W. Cho, and K.-J. Yoon, “Spherephd: Applying cnns on 360∘ images with non-euclidean spherical polyhedron representation,” _TPAMI_ , 2020.
  * [22] C. Esteves, C. Allen-Blanchette, A. Makadia, and K. Daniilidis, “Learning so (3) equivariant representations with spherical cnns,” in _ECCV_ , 2018.
  * [23] M. Weiler, F. A. Hamprecht, and M. Storath, “Learning steerable filters for rotation equivariant cnns,” in _CVPR_ , 2018.
  * [24] M. Qiao, M. Xu, Z. Wang, and A. Borji, “Viewport-dependent saliency prediction in 360 video,” _TMM_ , vol. 23, pp. 748–760, 2020.
  * [25] Y.-C. Su and K. Grauman, “Learning spherical convolution for fast features from 360 imagery,” _NeurIPS_ , 2017.
  * [26] S. Lee, J. Sung, Y. Yu, and G. Kim, “A memory network approach for story-based temporal summarization of 360 videos,” in _CVPR_ , 2018, pp. 1410–1419.
  * [27] Y.-C. Su and K. Grauman, “Making 360 video watchable in 2d: Learning videography for click free viewing,” in _CVPR_ , 2017.
  * [28] M. Xu, L. Jiang, C. Li, Z. Wang, and X. Tao, “Viewport-based cnn: A multi-task approach for assessing 360∘ video quality,” _TPAMI_ , vol. 44, no. 4, pp. 2198–2215, 2020.
  * [29] D. Tome, P. Peluse, L. Agapito, and H. Badino, “xr-egopose: Egocentric 3d human pose from an hmd camera,” in _ICCV_ , 2019.
  * [30] C. Li, M. Xu, X. Du, and Z. Wang, “Bridge the gap between vqa and human behavior on omnidirectional video: A large-scale dataset and a deep learning model,” in _ACM MM_ , 2018, pp. 932–940.
  * [31] M. F. R. Rondon, L. Sassatelli, R. Aparicio-Pardo, and F. Precioso, “Track: A new method from a re-examination of deep architectures for head motion prediction in 360∘ videos,” _TPAMI_ , vol. 44, no. 9, pp. 5681–5699, 2021.
  * [32] H. Kim, H.-T. Lim, S. Lee, and Y. M. Ro, “Vrsa net: Vr sickness assessment considering exceptional motion for 360 vr video,” _TIP_ , vol. 28, no. 4, pp. 1646–1660, 2018.
  * [33] C. Sun, M. Sun, and H.-T. Chen, “Hohonet: 360 indoor holistic understanding with latent horizontal features,” in _CVPR_ , 2021.
  * [34] J. Xu, J. Zheng, Y. Xu, R. Tang, and S. Gao, “Layout-guided novel view synthesis from a single indoor panorama,” in _CVPR_ , 2021.
  * [35] Y. Yang, S. Jin, R. Liu, S. B. Kang, and J. Yu, “Automatic 3d indoor scene modeling fr om single panorama,” in _CVPR_ , 2018.
  * [36] C. Zhang, S. Liwicki, W. Smith, and R. Cipolla, “Orientation-aware semantic segmentation on icosahedron spheres,” in _ICCV_ , 2019.
  * [37] Y.-C. Su, D. Jayaraman, and K. Grauman, “Pano2vid: Automatic cinematography for watching 360∘ videos,” in _ACCV_ , 2016.
  * [38] Y. Yu, S. Lee, J. Na, J. Kang, and G. Kim, “A deep ranking model for spatio-temporal highlight detection from a 360∘ video,” in _AAAI_ , 2018.
  * [39] C. Zhuang, Z. Lu, Y. Wang, J. Xiao, and Y. Wang, “Spdet: Edge-aware self-supervised panoramic depth estimation transformer with spherical geometry,” _TPAMI_ , 2023.
  * [40] A. Taneja, L. Ballan, and M. Pollefeys, “Geometric change detection in urban environments using images,” _TPAMI_ , vol. 37, no. 11, pp. 2193–2206, 2015\.
  * [41] L. Jin, Y. Xu, J. Zheng, J. Zhang, R. Tang, S. Xu, J. Yu, and S. Gao, “Geometric structure based and regularized depth estimation from 360 indoor imagery,” in _CVPR_ , 2020.
  * [42] G. Pintore, M. Agus, E. Almansa, J. Schneider, and E. Gobbetti, “Slicenet: Deep dense depth estimation from a single indoor panorama using a slice-based representation,” in _CVPR_ , 2021.
  * [43] R. Song, W. Zhang, Y. Zhao, Y. Liu, and P. L. Rosin, “3d visual saliency: an independent perceptual measure or a derivative of 2d image saliency?” _TPAMI_ , 2023.
  * [44] H.-T. Cheng, C.-H. Chao, J.-D. Dong, H.-K. Wen, T.-L. Liu, and M. Sun, “Cube padding for weakly-supervised saliency prediction in 360 videos,” in _CVPR_ , 2018.
  * [45] B. Xiong and K. Grauman, “Snap angle prediction for 360 panoramas,” in _ECCV_ , 2018.
  * [46] G. Ma, S. Li, C. Chen, A. Hao, and H. Qin, “Stage-wise salient object detection in 360 omnidirectional image via object-level semantical saliency ranking,” _TVCG_ , vol. 26, no. 12, pp. 3535–3545, 2020.
  * [47] A. Rana, C. Ozcinar, and A. Smolic, “Towards generating ambisonics using audio-visual cue for virtual reality,” in _ICASSP_ , 2019.
  * [48] F.-E. Wang, Y.-H. Yeh, Y.-H. Tsai, W.-C. Chiu, and M. Sun, “Bifuse++: Self-supervised and efficient bi-projection fusion for 360 depth estimation,” _TPAMI_ , vol. 45, no. 5, pp. 5448–5460, 2022.
  * [49] Y. Dahou, M. Tliba, K. McGuinness, and N. OConnor, “Atsal: An attention based architecture for saliency prediction in 360 videos,” in _ICPR_ , 2021.
  * [50] R. Cong, K. Huang, J. Lei, Y. Zhao, Q. Huang, and S. Kwong, “Multi-projection fusion and refinement network for salient object detection in 360∘ omnidirectional image,” _TNNLS_ , 2023.
  * [51] J. Li, L. Han, C. Zhang, Q. Li, and Z. Liu, “Spherical convolution empowered viewport prediction in 360 video multicast with limited fov feedback,” _ACM TMCCA_ , vol. 19, no. 1, pp. 1–23, 2023.
  * [52] Y.-C. Su and K. Grauman, “Learning spherical convolution for 360∘ recognition,” _TPAMI_ , vol. 44, no. 11, pp. 8371–8386, 2021.
  * [53] Y. Xu, Z. Zhang, and S. Gao, “Spherical dnns and their applications in 360 images and videos,” _TPAMI_ , vol. 44, no. 10, pp. 7235–7252, 2021.
  * [54] H. Yun, S. Lee, and G. Kim, “Panoramic vision transformer for saliency detection in 360∘ videos,” in _ECCV_ , 2022, pp. 422–439.
  * [55] N. Liu, N. Zhang, K. Wan, L. Shao, and J. Han, “Visual saliency transformer,” in _ICCV_ , 2021.
  * [56] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in _ICCV_ , 2021, pp. 568–578.
  * [57] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in _ICCV_ , 2017, pp. 764–773.
  * [58] J. Anderson and B. Fisher, “The myth of persistence of vision,” _JUFA_ , vol. 30, no. 4, pp. 3–8, 1978.
  * [59] Y. Yoon, I. Chung, L. Wang, and K.-J. Yoon, “Spheresr: 360deg image super-resolution with arbitrary projection via continuous spherical image representation,” in _CVPR_ , 2022.
  * [60] O. Younis, W. Al-Nuaimy, F. Rowe _et al._ , “A hazard detection and tracking system for people with peripheral vision loss using smart glasses and augmented reality,” _IJACSA_ , vol. 10, no. 2, 2019.
  * [61] A. Borst and M. Helmstaedter, “Common circuit design in fly and mammalian motion vision,” _Nature neuroscience_ , vol. 18, no. 8, pp. 1067–1076, 2015\.
  * [62] W. Yang, Y. Qian, J.-K. Kamarainen, F. Cricri, and L. Fan, “Object detection in equirectangular panorama,” in _ICPR_ , 2018, pp. 2190–2195.
  * [63] W. Gao, S. Fan, G. Li, and W. Lin, “A thorough benchmark and a new model for light field saliency detection,” _TPAMI_ , 2023.
  * [64] J.-J. Liu, Q. Hou, Z.-A. Liu, and M.-M. Cheng, “Poolnet+: Exploring the potential of pooling for salient object detection,” _TPAMI_ , vol. 45, no. 1, pp. 887–904, 2022.
  * [65] W. Xia, Q. Gao, Q. Wang, X. Gao, C. Ding, and D. Tao, “Tensorized bipartite graph learning for multi-view clustering,” _TPAMI_ , vol. 45, no. 4, pp. 5187–5202, 2022.
  * [66] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _NeurIPS_ , 2017.
  * [67] M. Cornia, L. Baraldi, G. Serra, and R. Cucchiara, “Predicting human eye fixations via an lstm-based saliency attentive model,” _TIP_ , vol. 27, no. 10, pp. 5142–5154, 2018.
  * [68] W. Wang, J. Shen, F. Guo, M.-M. Cheng, and A. Borji, “Revisiting video saliency: A large-scale benchmark and a new model,” in _CVPR_ , 2018.
  * [69] M. Ester, H.-P. Kriegel, J. Sander, X. Xu _et al._ , “A density-based algorithm for discovering clusters in large spatial databases with noise,” in _AAAI_ , vol. 96, no. 34, 1996, pp. 226–231.
  * [70] S.-H. Chou, Y.-C. Chen, K.-H. Zeng, H.-N. Hu, J. Fu, and M. Sun, “Self-view grounding given a narrated 360 video,” in _AAAI_ , 2018.
  * [71] A. Borji, “Saliency prediction in the deep learning era: Successes and limitations,” _TPAMI_ , 2019.
  * [72] Z. Bylinskii, T. Judd, A. Oliva, A. Torralba, and F. Durand, “What do different evaluation metrics tell us about saliency models?” _TPAMI_ , vol. 41, no. 3, pp. 740–757, 2018.
  * [73] D. Martin, A. Serrano, and B. Masia, “Panoramic convolutions for 360 single-image saliency prediction,” in _CVPRW_ , 2020.
  * [74] F.-Y. Chao, L. Zhang, W. Hamidouche, and O. Deforges, “Salgan360: Visual saliency prediction on 360 degree images with generative adversarial networks,” in _ICMEW_ , 2018.
  * [75] P. Linardos, E. Mohedano, J. Nieto, K. McGuinness, X. GiroiNieto, and N. OConnor, “Simple vs complex temporal recurrences for video saliency prediction,” in _BMVC_ , 2019.
  * [76] Y. Rai, J. Gutierrez, and P. LeCallet, “A dataset of head and eye movements for 360 degree images,” in _MMSys_ , 2017.
  * [77] P. Lebreton and A. Raake, “Gbvs360, bms360, prosal: Extending existing saliency prediction models from 2d to omnidirectional images,” _SP:IP_ , vol. 69, pp. 69–78, 2018.
  * [78] J. Wu, C. Xia, T. Yu, and J. Li, “View-aware salient object detection for 360∘ omnidirectional image,” _TMM_ , pp. 1–15, 2022.
  * [79] X. Zhou, S. Wu, R. Shi, B. Zheng, S. Wang, H. Yin, J. Zhang, and C. Yan, “Transformer-based multi-scale feature integration network for video saliency prediction,” _TCSVT_ , 2023.