###### Abstract

Predicting saliency in videos is a challenging problem due to complex modeling
of interactions between spatial and temporal information, especially when
ever-changing, dynamic nature of videos is considered. Recently, researchers
have proposed large-scale datasets and models that take advantage of deep
learning as a way to understand what’s important for video saliency. These
approaches, however, learn to combine spatial and temporal features in a
static manner and do not adapt themselves much to the changes in the video
content. In this paper, we introduce Gated Fusion Network for dynamic saliency
(GFSalNet), the first deep saliency model capable of making predictions in a
dynamic way via gated fusion mechanism. Moreover, our model also exploits
spatial and channel-wise attention within a multi-scale architecture that
further allows for highly accurate predictions. We evaluate the proposed
approach on a number of datasets, and our experimental analysis demonstrates
that it outperforms or is highly competitive with the state of the art.
Importantly, we show that it has a good generalization ability, and moreover,
exploits temporal information more effectively via its adaptive fusion scheme.

###### Index Terms:

dynamic saliency estimation, gated fusion, deep saliency networks

A Gated Fusion Network for Dynamic Saliency Prediction Aysun Kocak1, Erkut
Erdem1 and Aykut Erdem2
1Department of Computer Engineering, Hacettepe University, Ankara, Turkey
2Department of Computer Engineering, Koç University, Istanbul, Turkey

##  I Introduction

Human visual system employs visual attention mechanisms to effectively deal
with huge amount of information by focusing only on salient or attention
grabbing parts of a scene, and thus filtering out irrelevant stimuli. Saliency
estimation methods offer different computational models of attention to mimic
this key component of our visual system. These methods generate a so-called
saliency map within which a pixel value indicates the likelihood of that pixel
being fixated by a human. Since the pioneering work of [1], this research area
has gained a lot of interest in the last few decades (please refer to [2, 3]
for an overview), and it has found to have practical use in a variety of
computer vision tasks such as visual quality assessment [4, 5], image and
video resizing [6, 7], video summarization [8], to name a few. Early saliency
prediction approaches use low-level (color, orientation, intensity) and/or
high-level (pedestrians, faces, text, etc.) image features to estimate salient
regions. While low-level cues are used to detect regions that are different
from their surroundings, top-down cues are used to infer high-level semantics
to guide the model. For example, humans tend to focus some object classes more
than others. Recently, deep learning based models have started to dominate
over the traditional approaches as they can directly learn both low and high-
level features relevant for saliency prediction [9, 10].

Most of the literature on saliency estimation focuses on static images.
Lately, predicting saliency in videos has also gained some attraction, but it
still remains a largely unexplored field of research. Video saliency models
(also called dynamic saliency models) aim to predict attention grabbing
regions in dynamically changing scenes. While static saliency estimation
considers only low-level and high-level spatial cues, dynamic saliency needs
to take into account temporal information too as there is evidence that moving
objects or object parts can also guide our attention. Motion and appearance
play complementary roles in human attention and their significance can change
over time. As we illustrate in Fig. 1, in dynamic scenes, humans tend to focus
more on moving parts of the scene and the eye fixations change over time,
showing the importance of motion cues (bottom row). On the other hand, when
there is practically no motion in the scene, low-level appearance cues
dominantly guide our attention and we focus more on the regions showing
different visual characteristics than their surroundings (top row). Motivated
by these observations, in this work, we develop a deep dynamic saliency model
which handles spatial and temporal changes in the visual stimuli in an
adaptive manner.

|
---|---
|
A single input frame and its corresponding fixation map |  Four consecutive overlaid frames and their overlaid fixation maps
Figure 1: Predicting video saliency requires finding a harmonious interaction
between appearance and temporal information. For example, while the first row
shows a case in which attention is guided more by visual appearance, in the
second row, motion is the most determining factor for attention. Hence, we
speculate that an adaptive scheme would be better suited for this task.

The first generation of dynamic saliency methods were simply extensions of the
static saliency approaches, e.g. [11, 12, 13, 14, 15]. In other words, these
methods adapted the strategies proposed for static scenes and mostly modified
them to work on either 3D feature maps that are formed by stacking 2D spatial
features over time or 2D feature maps encoding motion information like optical
flow images. Several follow-up works, however, have approached the problem
from a fresh perspective and developed specialized methods for dynamic
saliency detection, e.g. [16, 17, 18, 19, 20, 21, 22, 23, 24]. These models
either utilize novel spatio-temporal features or employ data-driven techniques
to learn relevant features from data. As with the case of state-of-the-art
static saliency models, approaches based on deep learning have also shown
promise for dynamic saliency. These studies basically explore different neural
architectures used for processing temporal and spatial information in a joint
manner, and they either use 3D convolutions [25], LSTMs [25, 26] or multi-
stream architectures that encode temporal information separately [27, 28, 29].

In this work, we introduce Gated Fusion Network for video saliency (GFSalNet).
Our proposed network model is radically different from the previously proposed
deep models in that it includes a novel content-driven fusion scheme to
combine spatial and temporal streams in a more dynamic manner. In particular,
our model is based on two-stream CNNs [30, 31], which have been successfully
applied to various video analysis tasks. To our interest, these architectures
are inspired by the ventral and dorsal pathways in the human visual cortex
[32]. Although the use of two-stream CNNs in video saliency prediction has
been investigated before [28], the main novelty of our work lies in the
ability to fuse appearance and motion information in a spatio-temporally
coordinated manner by estimating the importance of each cue with respect based
on the current video content.

The rest of the paper is organized as follows: In Section 2, we give a brief
overview of the existing dynamic saliency approaches. In Section 3, we present
the details of our proposed deep architecture for video saliency. In Section
4, we give the details of our experimental setup, including evaluation
metrics, datasets and the competing dynamic saliency models, and discuss the
results of our experiments. Finally, in the last section, we offer some
concluding remarks.

Our codes and predefined models, along with the saliency maps extracted with
our approach, will be publicly available at the project
website111https://hucvl.github.io/GFSalNet/.

##  II Related Work

Early visual saliency models can be dated back to 1980s with the Feature
Integration Theory by [33]. The first models of saliency, such as [34, 1],
provide computational solutions to [33], and since then a notable number of
saliency models are developed, most of which deal with static scenes. For a
detailed list of pre-deep learning saliency estimation approaches, please
refer to [2]. After the availability of large-scale datasets, researchers
proposed various deep learning based models for static saliency that
outperformed previous approaches by a large margin [35, 36, 37, 38, 39, 40,
41, 42].

Early models for dynamic saliency generally depend on previously proposed
static saliency models. Adaptation of these models to dynamic scenes is
achieved by considering features related to motion such as the optical flow
information. For example, [11] proposed a saliency prediction method called
PQFT that predicts the salient regions via the phase spectrum of Fourier
Transform of the given image. In particular, PQFT generates a quaternion image
representation by using color, intensity, orientation and motion features and
estimates the salient regions in the frequency domain by using this combined
representation. [12] extracted salient parts of video frames by similarly
performing a spectral analysis of the frames considering both spatial and
temporal domains. [13] employed local regression kernels as features to
calculate self similarities between pixels or voxels for figure-ground
segregation. [14] extended the previously proposed static saliency model by
[43]’s model by including motion cues to the graph-theoretic formulation. [44]
employ a two stream approach that generates spatial saliency map (using color
and texture features) and temporal saliency map (using optical flow feature)
separately and combines these maps with an entropy based adaptive method. [15]
proposed a dynamic saliency model for activity recognition that works in an
unsupervised manner. Their method is based on an encoding scheme that
considers color along with motion cues.

Following these early approaches, the researchers started to develop novel
video saliency models specifically designed for dynamic stimuli. For instance,
[16] proposed a sparsity based framework that generates spatial saliency maps
and temporal saliency maps separatelty based on entropy gain and temporal
consistency, respectively, and then combines them. [17] integrated several
visual cues such as static and dynamic image features based on color, texture,
edge distribution, motion boundary histograms, through learning-based fusion
strategies and later employed this dynamic saliency model for action
recognition. [18] suggested a learning-based model that generates a candidate
set regions with the use of existing methods and then predicts gaze
transitions over subsequent video frames conditionally on these regions. [19]
proposed a simple dynamic saliency model that combines spatial saliency maps
with temporal saliency using pixel-wise maximum operation. In their work,
while the spatial saliency maps are extracted using multi-scale analysis of
low-level features, temporal saliency maps are obtained by examining dynamic
consistency of motion through an optical flow model. [20] suggested an
approach that independently estimates superpixel-level and pixel-level
temporal and spatial saliency maps and subsequently combines them using an
adaptive fusion strategy. [21] proposed an approach that oversegments video
frames by using both spatial and temporal information and estimates the
saliency score for each region by computing the regional contrast values via
low-level features extracted from these regions. [22] suggested to learn a
filter bank from low-level features for fixations. This filterbank encodes the
association between local feature patterns and probabilities of human
fixations, and is used to re-weight fixation candidates. [23] formulated
another dynamic saliency model by exploiting the compressibility principle.
More recently, [24] proposed a saliency model (called AWS-D) for dynamic
scenes by considering the observation that high-order statistical structures
carry most of the perceptually relevant information. AWS-D [24] removes the
second-order information from input sequence via a whitening process. Then, it
computes bottom-up spatial saliency maps using a filter bank at multiple
scales, and temporal saliency maps with the use of a 3D filter bank. Finally,
it combines all these maps by considering their relative significance.

Deep learning based dynamic saliency models have received attention only
recently. [25] proposed a recurrent mixture density network (RMDN) for spatio-
temporal visual attention. The method uses a C3D architecture [45] as a
backbone to integrate spatial and temporal information. This representation
module is fed to a Long Short-Term Memory (LSTM) network, which is connected
to Mixture Density Network (MDN) whose outputs are the parameters of a
Gaussian mixture model expressing the saliency map of each frame. [28]
suggested a two stream CNN model [30, 31] which considers the motion and
appearance clues in videos. While, optical flow images are used to feed the
temporal stream, raw RGB frames are used as input for the spatial stream. [27]
presented an attention network to predict where driver is focused. In this
work, the authors also proposed a dataset that consists of ego-centric and
car-centric driving videos and eye tracking data belongs to the videos. Their
network consists of three independent paths, namely spatial, temporal and
semantic paths. While the spatial path uses raw RGB data as input, the
temporal one uses optical flow data to integrate motion information and the
last one processes the segmentation prediction on the scene given by the model
by [46]. In the final layer of the network, the three independent maps are
summed and then normalized to obtain the final saliency map. [29] proposed a
deep model called OM-CNN which consists of two subnetworks, namely objectness
subnet to highlight the regions that contain an object, motion subnet to
encode temporal information, whose outputs are then combined to generate some
spatio-temporal features. [26] proposed a model called ACLNet which employs a
CNN-LSTM architecture to predict human gaze in dynamic scenes. The proposed
approach focuses static information with an attention module and allows an
LSTM to focus on learning dynamic information. Recently, [47] proposed an
encoder-decoder based deep neural network called SalEMA, which employs a
convolutional recurrent neural network method to include temporal information.
In particular, it processes a sequence of RGB video frames as input to employ
spatial and temporal information with the temporal information being inferred
by the weighted average of the convolution state of the current frame and all
the previous frames. [48] suggested a different model called TASED-Net, which
utilizes a 3D fully-convolutional encoder-decoder network architecture where
the encoded features are spatially upsampled while aggregating the temporal
information. [49] recently developed another two-stream spatiotemporal salieny
model called STRA-Net that considers dense residual cross connections and a
composite attention module.

Figure 2: Our two-stream dynamic saliency model uses RGB frames for spatial
stream and optical flow images for temporal stream. These streams are
integrated with a dynamic fusion strategy that we referred to as gated fusion.
Our architecture also employs multi-level information block to fuse multi-
scale features and attention blocks for feature selection.

The aforementioned dynamic saliency models suffer from different drawbacks.
The early methods employ (hand-crafted) low-level features that do not provide
a high-level understanding of the video frames. Deep models eliminate this
pitfall by utilizing an end-to-end learning strategy and, hence, provide
better saliency predictions. They differ from each other by how they include
motion information within their respective architectures. As we reviewed, the
two main alternative approaches include using recurrent connections or
processing data in multiple streams. Although RNN-based models help to encode
temporal information with less amount of parameters, the encoding procedure
compresses all the relevant information into a single vector representation,
which affects the robustness especially for longer sequences. In that respect,
the accuracy of the two-stream models do not, in general, degrade as the
length of a sequence increases. Moreover, they are more interpretable as they
need to perform fusion of spatial and temporal features in an explicit manner.
On the other hand, their performance depends on accurate estimation of the
optical flow maps used as input to the temporal stream. Hence, most of these
two-stream models employ recent deep-learning based optical flow estimation
models and even some of them uses some additional post-processing steps such
as confining the absolute values of the magnitudes within a certain interval
to avoid noise, as in STRA-Net [49]. Our proposed model also uses a two-stream
approach, but as we will show, it exploits a novel and more dynamic fusion
strategy, which boosts the performance and further improves the
interpretability.

##  III Our Model

A general overview of our proposed spatio-temporal network architecture is
given in Fig. 2. We use a two-stream architecture that processes temporal and
spatial information in separate streams, similar to the one in [28]. That is,
we respectively feed the spatial stream and temporal stream with RGB video
frames and the corresponding optical flow images as inputs. Different than
[28], however, our network combines information coming from several levels
(Section III-A) and fuses both streams via a novel dynamic fusion strategy
(Section III-C). We additionally utilize attention blocks (Section III-B) to
select more relevant features to further boost the performance of our model.
Here, we use a pre-trained ResNet-50 model [50] as the backbone of our
saliency network as commonly explored by the previous saliency studies. In
particular, we remove the average pooling and fully connected layers after the
last residual block (ResBlock4) and then adapt it for saliency prediction by
adding extra blocks. Using ResNet-50 model allows us to encode both low-, mid-
and high-level cues in the visual stimuli in an efficient manner. Moreover,
the number of network parameters is much smaller as compared to other
alternative backbone networks.

###  III-A Multi-level Information Block

As its name implies, the purpose of multi-level information block is to let
the information extracted at different levels guide the saliency prediction
process. It has proven to be useful that employing a multi-level/multiscale
structure almost always improves the performance for many different vision
tasks such as object detection [51], segmentation [52, 53, 54], and static
saliency detection [55, 56]. In our work, we also employ a multi-level
information block to enhance feature learning capability of our model.
Specifically, it allows low-, mid-, and high-level information to be fused
together and to be taken into account simultaneously while making predictions.

Figure 3: Multi-level information block. It is used to integrate multiscale
features extracted at different levels of the deep network for predicting
salient parts of the given input video frame.

Fig. 3 shows the proposed multi-level information block that we employ in our
model. This block considers low-level and high-level representations of frames
by processing features maps which are extracted at each residual block. The
aim is to combine primitive image features (e.g. edges, shared common
patterns) obtained at lower levels with rich semantic information (e.g. object
parts, faces, text) extracted at higher levels of the network. Here, we prefer
to utilize $1\times 1$ convolution and bilinear interpolation layers to
combine cues from higher and lower levels. That is, after each residual block,
we expand the feature map with bilinear interpolation to make equal size of
the feature map with the size of the output of the previous residual block.
Then, we concatenate the expanded feature map with the previous residual
block’s output and fuse them via $1\times 1$ convolution layers.

###  III-B Attention Blocks

Neural attention mechanisms allow for learning to pay attention to features
more useful for a given task, and hence, it has been demonstrated many times
that they can boost the performance of a neural network architecture proposed
for any computer vision problem, such as object detection [57]), visual
question answering [58], pose estimation [59], image captioning [60] and
salient object detection [55]. Motivated with these observations, in our work,
we integrate several attention blocks to our proposed deep architecture to let
the model choose the most relevant features for the dynamic saliency
estimation problem. Resembling the structures in [60, 55], we exploit two
separate attention mechanisms: _spatial_ and _channel-wise_ attention, as
explained below.

Fig. 4(a) shows our spatial attention block, which we introduce at the lower
levels of our network model (see Fig. 2) that helps to filter out the
irrelevant information. The block takes the output of ResBlock4, shaped
$[B\times C\times H\times W]$ with $C=2048$, as input and it determines the
important locations by calculating a weight tensor, which is shaped $[B\times
1\times H\times W]$. To estimate this tensor, input channels are fused via
$1\times 1$ convolution layer following by a sigmoid layer. The output (shaped
$[B\times C\times H\times W]$) of this block is a result of Hadamard product
between input and spatial weight tensor.

(a) Spatial attention

(b) Channel-wise attention

Figure 4: Attention blocks: (a) spatial attention block, (b) channel-wise
attention block. While the spatial attention block defines spatial importance
weights for individual feature maps, the channel-wise attention block
introduces feature-level weighting which allows for a better use of context
information.

The second type of our attention block, the channel-wise attention block, is
shown in Fig. 4(b), whose main purpose is to utilize the context information
in a more efficient way. The block consists of average pooling, full connected
and ReLU layers. In particular, it takes the concatenation of the feature maps
from the main stream and multi-level information block as input which is
shaped $[B\times 96\times H\times W]$, then downsamples it with average
pooling (output shape is $[B\times 96]$). The weight of each channel is
determined after two fully connected layers followed by ReLUs. The shape of
the matrices are $[B\times 24]$ and $[B\times 96]$ respectively. The output of
last ReLU which is shaped $[B\times 96\times 1\times 1]$, contains a scalar
value to weight each channel. At the end of the block, the input feature map
is weighted via Hadamard product.

###  III-C Gated Fusion Block

One of the main contributions of our framework is to employ a dynamic fusion
strategy to combine temporal and spatial information. Gated fusion has been
exploited before for different problems such as image dehazing [61], image
deblurring [62], semantic segmentation [63]. The main purpose to use a gated
fusion block is to combine different kind of information with a dynamic
structure which considers the current inputs’ characteristics. For example, in
[63] feature maps that are generated via RGB information and depth information
is combined for solving semantic segmentation. In our case, our aim is to come
up with a fusion module that considers the content of the video at inference
time. To our knowledge, we are the first to provide a truly dynamic approach
for dynamic saliency. As opposed to the classical learning based approaches
that learn the contributions of temporal and spatial streams in a static
manner from the training data, our gated fusion block performs the fusion
process in an adaptive way. That is, it decides the contribution of each
stream on a location- and time-aware manner according to the content of the
video.

Figure 5: Gated fusion block. It integrates the spatial and temporal streams
to learn a weighted gating scheme to determine their contributions in
predicting dynamic saliency of the current input video frame.

Appearance |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Motion |  |  |  |  |  |  |  |
$\quad\;\;\;S_{A}$ |  |  |  |  |  |  |  |
$\quad\;\;\;G_{A}$ |  |  |  |  |  |  |  |
$\quad\;\;\;S_{T}$ |  |  |  |  |  |  |  |
$\quad\;\;\;G_{T}$ |  |  |  |  |  |  |  |
Prediction |  |  |  |  |  |  |  |
GT |  |  |  |  |  |  |  |

Figure 6: Gated fusion block estimates the final saliency map by combining the
appearance and the temporal maps $S_{A}$ and $S_{T}$ with the spatially
varying weights $G_{A}$ and $G_{T}$.

The structure of the proposed gated fusion block is shown in Fig. 5. It takes
the feature maps of the spatial and temporal streams as inputs and produces a
probability map which is used to designate contribution of each stream with
regard to their current characteristics. Let $S_{A}$, $S_{T}$ denote the
feature maps from spatial and temporal streams, respectively. Gated fusion
module first concatenates these features and then learns their correlations by
applying a $1\times 1$ convolution layer. After that, it uses a sigmoid layer
to regularize the feature map which is used to estimate weights of the gate.
Let $G_{A}$ and $G_{T}$ denote how confidently we can rely on appearance and
motion, respectively, as follows:

$G_{A}=P\;,\quad G_{T}=1-P\;,$ (1)

where $P$ is the output of the sigmoid layer. Then, gated fusion module
estimates the weights denoting the contributions of the spatial and temporal
streams, as given below:

$S^{\prime}_{A}=S_{A}\odot G_{A}\;,\quad S^{\prime}_{T}=S_{T}\odot G_{T}\;,$
(2)

where $\odot$ represents the Hadamard product operation. Finally, it generates
the final saliency map, $S_{final}$, via weighting the appearance and temporal
streams’ feature maps with the estimated probability map:

$S_{final}=S^{\prime}_{A}+S^{\prime}_{T}\;.$ (3)

Fig. 6 visualizes how gated fusion block works. While the appearance stream
computes a saliency map from the RGB frame, the temporal stream extracts a
saliency map from the optical image obtained from successive frames. As can be
seen, these intermediate maps encode different characteristic of the input
dynamic stimuli. The appearance based saliency map mostly focuses on the
regions that have distinct visual properties than theirs surroundings, whereas
the motion based saliency map mainly pay attention to motion. Gated fusion
scheme estimates spatially varying probability maps and employs them to
integrate the appearance and temporal streams, which results in more confident
predictions. The spatial stream generally gives more accurate predictions than
the temporal stream, as will be presented in the Experiments section. On the
other hand, as can be seen from the estimated weight maps, the gated fusion
scheme in the proposed model has a tendency to pay more attention to the
temporal stream. We suspect that this is because the model considers that it
may carry auxiliary information.

##  IV Experiments

In the following, we first provide a brief review of the benchmark datasets we
used in our experimental analysis. Then, we give the details of our training
procedure including the loss functions and settings we use to train our
proposed model. Next, we summarize the evaluation metrics and the dynamic
saliency models used in our experiments. We then discuss our findings and
present some qualitative and quantitative results. Finally, we present an
ablation study to evaluate the effectiveness of the blocks of the proposed
dynamic saliency model.

###  IV-A Datasets

In our experiments, we employ six different datasets to evaluate the
effectiveness of the proposed saliency model. The first four, namely UCF-
Sports [64], Holywood-2 [65], DHF1K [26], and DIEM [66], are the most commonly
used benchmarks. Among them, we specifically utilize DIEM to test the
generalization ability of our model. The last two datasets considered in our
analysis, DIEM-Meta [67] and LEDOV-Meta [67], are two recently proposed
datasets, which are particularly designed to explore the performance of a
dynamic saliency model under situations where understanding temporal effects
is critical to give results more compatible with humans.

UCF-Sports dataset [64] is the smallest dataset in terms of its size,
consisting of 150 videos obtained from 13 different action classes. It is
originally collected for action recognition, but then enriched by [65] to
include eye fixation data. The videos are annotated by 4 subjects under free-
viewing condition. In the experiments, we used the same train/test splits
given in [68].

Holywood-2 dataset [65] contains 1,707 videos from Hollywood-2 action
recognition dataset [69], among which 823 are used for training and the
remaining 884 are left for testing. Since the videos are collected from 69
Hollywood movies with 12 action categories, its content is limited to human
actions. In [65], the authors collected human fixation data for each sequence
from 3 subjects under free-viewing condition. In our experiments, we use all
train and test frames.

DHF1K [26] is the most recent and the largest video saliency dataset, which
contains a total of 1000 videos with eye tracking data collected from 17
different human subjects. The authors split the dataset into 600 training, 100
validation videos and 300 test videos. The ground truth fixation data for the
test split is intentionally kept hidden and the evalution of a model on the
test data is carried out by the authors themselves.

DIEM [66] includes 84 natural videos. Each video sequence has eye fixation
data collected from approximately 50 different human subjects. Following the
common experimental setup first considered in [18], we used all frames from 64
videos for training and the first 300 frames from the remaining 20 videos as
test set.

DIEM-Meta [67] and LEDOV-Meta [67] are two recently proposed datasets
collected from the existing video saliency datasets DIEM [66] and LEDOV [29],
respectively. The main difference between these and the aforementioned
datasets lies in the characteristics of the video frames they consider. [67]
constructed these so-called meta-datasets by eliminating the video frames from
their original counterparts where spatial patterns are generally enough to
predict where people look. To detect them, they employ a deep static saliency
model that they developed. DIEM-Meta and DIEM-Meta are thus better testbeds
for evaluating whether or not a dynamic saliency model learns to use the
temporal domain effectively. DIEM-Meta contains only 35% of the video frames
from DIEM, LEDOV-Meta includes just 20% of the original LEDOV frames.

###  IV-B Training Procedure

As we mentioned previously, our network takes RGB video frames and optical
flow images as inputs. We extract the frames from the videos by considering
their original frame rate. We employ these RGB frames to feed our appearance
stream. For the temporal stream, we generate the optical flow images between
two consecutive frames by using PWC-Net [70]. We resize all the input images
to $640\times 480$ pixels and map the ground truth fixation points
accordingly.

Instead of training our dynamic saliency network from scratch, we first train
the subnet for the appearance stream on SALICON dataset [71]. Then, we
initialize the weights of both of our subnets for spatial and temporal streams
with this pre-trained static saliency model and finetune our whole two-stream
network model using the dynamic saliency datasets described above. Pre-
training on static data allows our dynamic saliency model to converge in fewer
epochs when trained on dynamic stimuli. We use Kullback-Leibler (KL)
divergence and Normalized Scanpath Saliency (NSS) loss functions (which we
will explain in detail later) with Adam optimizer during the training process.
We set the initial learning rate to 10e-5 and reduce it to one tenth in every
3000th iteration. The batch size is set to 8 for UCF-Sports and 16 for the
other video datasets. We train our model on NVIDIA V100 GPUs (3$\times$GPUs)
and while one epoch takes approximately 2 days for the larger datasets of
DHF1K, DIEM and Hollywood-2, it takes approximately 2 hours for UCF-Sports. We
train our models for 2-3 epochs. Our (unoptimized) Pytorch implementation
achieves a near real-time performance of 8.2 fps for frames of size $640\times
480$ on a NVidia Tesla K40c GPU.

For our experiments on standard benchmark datasets, we consider two different
training settings for dynamic stimuli. In our first setting, we use the
training split of the dataset under consideration to train our proposed model.
On the other hand, in our second setting, we utilize a combined training set
containing training sequences from both UCF-Sports, Hollywood-2 and DHF1K
datasets. The second setting further allows us to test the generalization
ability of our model on DIEM, DIEM-Meta and LEDOV-Meta datasets.

Loss functions. In our work, we employ the combination of KL-divergence and
NSS loss functions to train our proposed dynamic saliency model. As explored
in previous studies, [72, 26], considering more than one loss function during
training, in general, improves the model performance. Moreover, empirical
experiments on the analysis of the existing automatic evaluation metrics in
[73] have shown that KL-divergence and NSS are good choices for evaluating
saliency models.

Let $P$ denote the predicted saliency map, $F$ represent ground truth (binary)
fixation map collected from human subjects and $S$ be the ground truth
(continuous) fixation density map which is generated by blurring fixation maps
with a small Gaussian kernel.

KL-divergence is a widely used metric to compare two probability
distributions. It has been proven to be effective for evaluating and trainig
the performance of saliency models where the ground truth fixation map $S$ and
the predicted saliency map $P$ are interpreted as probability distributions.
Formally, KL-divergence loss function is defined as:

$\mathcal{L}_{KL}(P,S)=\sum_{i}S(i)log\left(\frac{S(i)}{P(i)}\right)\;.$ (4)

NSS is a location based metric which is computed as the average of the
normalized predicted saliency values at fixated locations that is provided
with the ground truth. By using this metric as a loss function, we force the
saliency model to better detect the fixation locations and assign high
likelihood scores to those pixel locations. This loss function is defined as
below:

$\mathcal{L}_{NSS}(P,F)=-\frac{1}{N}\sum_{i}\bar{P}(i)\times F(i)\;,$ (5)

where $N$ is the total number of fixated pixels $\sum_{i}F(i)$ and $\bar{P}$
is the normalized saliency map $\frac{P-\mu(P)}{\sigma(P)}$.

Our final loss function is then defined as:

$\mathcal{L}(P,F,S)=\alpha\mathcal{L}_{KL}(P,S)\;+\;\beta\mathcal{L}_{NSS}(P,F)\;,$
(6)

where $\mathcal{L}_{KL}$ is the KL loss function, $\mathcal{L}_{NSS}$ is the
NSS loss function, and $\alpha$ and $\beta$ are the weights for these loss
functions. We first perform a set of experiments on SALICON dataset to
empirically determine the optimal values of $\alpha$ and $\beta$, and then set
$\alpha=1$ and $\beta=0.1$ for all the experiments.

###  IV-C Evaluation Metrics and Compared Saliency Models

In our evaluation, we employ the following five commonly reported saliency
metrics: Area Under Curve (AUC-Judd), Pearson’s Correlation Coefficient (CC),
Normalized Scanpath Saliency (NSS), Similarity Metric (SIM) and KL-divergence
(KLDiv). For a detailed analysis of these metrics and their definitions,
please refer to [73]. Each metric measures a different aspect of visual
saliency and none of them is superior to the others. AUC metric considers the
saliency map as classification map. A ROC curve is constituted by measuring
the true and false positive rates under different binary classifier
thresholds. While a score of 1 indicates a perfect match, a score close to 0.5
indicates the performance of chance. NSS is another commonly used metric,
which we formally defined before while describing our loss functions. CC
metric is a distribution based metric which is used to measure the linear
relationship between saliency and fixation maps using the following formula:

${CC}(P,S)=\frac{\sigma(P,S)}{\sigma(P)\times\sigma(S)}\;$ (7)

where $\sigma$ corresponds to covariance. A CC value close to +1/-1
demonstrates a perfect linear relationship. SIM is another popular metric that
measures the similarity between the predicted and human saliency maps, as
defined below:

$\displaystyle{\text{SIM}}(P,S)=\sum_{i}\min(P_{i},S_{i})\;$
$\displaystyle\text{where}\;\sum_{i}P_{i}=1\;\text{and}\;\sum_{i}S_{i}=1$ (8)

KLDiv metric evaluates the dissimilatrity between two distributions. Since
KLDiv represents the difference between the saliency map and the density map,
a small value indicates a good result. However, we note that, according to the
aforementioned study, NSS and CC seem to provide more fair results. In our
experiments, we report the scores obtained with the implementations provided
by MIT benchmark
website222https://github.com/cvzoya/saliency/tree/master/code_forMetrics.

We compare our method with ten different models: SalGAN [74], PQFT [11], [44],
AWS-D [24], [28], OM-CNN [29], ACLNet [26], SalEMA [47], STRA-Net [49], and
TASED-Net [48]. Among these, SalGAN [74] is the only static saliency model
that gives the state-of-the-art results in the image datasets. We evaluate
this method on video datasets considering each frame as a static image. PQFT
[11], [44], and AWS-D [24] are non-deep learning models whereas all the other
models employs deep learning techniques to predict where people look in
videos. We note that in [28], the authors tested different fusion strategies
with static weighting schemes and here we only report the results obtained
with convolutional fusion strategy, which was shown to perform better than the
others. In our experiments, we use the implementations and the trained models
provided by the authors and test our approach against them with the settings
explained in Sec. IV-A for fair comparison. In particular, after a careful
analysis, we notice that some methods do not report results on whole test set
of Hollywood-2 and/or they mistakenly consider task-specific gaze data
collected for UCF-Sports while generating the groundtruth fixation density
maps. Hence, some of the results are different than those reported in the
papers but they give a better picture of their performances. Moreover, in our
experiments, we also provide the results of single-stream versions of our
model that respectively consider either spatial or temporal information.

###  IV-D Qualitative and Quantitative Results

Performance on UCF-Sports. Table I reports the comparative results on UCF-
Sports test set, which contains 43 sequences. As can be seen, the single-
stream versions of our proposed model gives worse scores than our full model.
Moreover, spatial stream generally predicts saliency much better than the
temporal stream, which is a trend that we observe on the other standard
benchmark datasets too. Our model trained only on UCF-Sports outperforms all
the competing models in most of the metrics. It results in a performance very
close to those of SalEMA and STRA-Net in terms of SIM. We believe that
weighting the predictions by the spatial and temporal streams using a gating
mechanism allows the model to better handle the variations throughout video
sequence, thus resulting in more accurate saliency maps on this action-
specific relatively small dataset.

TABLE I: Performance comparison on UCF-Sports dataset. The best and the second
best performing models are shown in bold typeface and underlined,
respectively.

Method Metric | AUC-J$\uparrow$ | CC$\uparrow$ | NSS$\uparrow$ | SIM$\uparrow$ | KLDiv$\downarrow$
---|---|---|---|---|---
Static | SalGAN | 0.869 | 0.389 | 2.074 | 0.258 | 2.169
| PQFT* | 0.776 | 0.211 | 1.189 | 0.157 | 2.458
| Fang et al.* | 0.879 | 0.387 | 2.319 | 0.247 | 2.012
| AWS-D* | 0.845 | 0.313 | 1.870 | 0.195 | 2.202
| Bak et al. | 0.864 | 0.387 | 2.231 | 0.130 | 2.575
Dynamic | OM-CNN | 0.880 | 0.398 | 2.443 | 0.294 | 1.902
| ACLNet | 0.876 | 0.367 | 2.045 | 0.292 | 2.135
| SalEMA | 0.895 | 0.470 | 2.979 | 0.384 | 1.728
| STRA-Net | 0.902 | 0.479 | 2.916 | 0.384 | 2.483
| TASED-Net | 0.887 | 0.453 | 2.680 | 0.369 | 1.876
Ours | Spatial | 0.870 | 0.461 | 3.029 | 0.377 | 2.504
(Single) | Temporal | 0.851 | 0.418 | 2.535 | 0.345 | 2.721
Ours | Setting 1 | 0.914 | 0.526 | 3.333 | 0.382 | 1.516
(Gated) | Setting 2 | 0.911 | 0.499 | 2.980 | 0.353 | 1.568

  * •

* Non-deep learning model

Performance on Hollywood-2. In our experiments on Hollywood-2 dataset, we use
all the frames from the test set that contains 884 video sequences. In that
regard, it is the largest test set that we considered in our experimental
evaluation. In Table II, we provide comparison against the competing saliency
models. Our results show that our model gives better saliency predictions than
all the other methods in terms of the AUC-J and KLDiv metrics. The performance
of the model trained considering our second training setting that includes a
larger and more diverse training set provides much better results than the one
trained with the first setting. In terms of the remaining evaluation metrics,
our results are highly competitive as compared to the recent state-of-the-art
models, namely STRA-Net and TASED-Net, as well.

TABLE II: Performance comparison on Hollywood-2 dataset. The best and the
second best performing models are shown in bold typeface and underlined,
respectively.

Method Metric | AUC-J$\uparrow$ | CC$\uparrow$ | NSS$\uparrow$ | SIM$\uparrow$ | KLDiv$\downarrow$
---|---|---|---|---|---
Static | SalGAN | 0.892 | 0.428 | 2.383 | 0.298 | 1.760
| PQFT* | 0.689 | 0.150 | 0.610 | 0.139 | 2.387
| Fang et al.* | 0.862 | 0.312 | 1.614 | 0.221 | 1.781
| AWS-D* | 0.747 | 0.227 | 0.994 | 0.193 | 2.256
| Bak et al. | 0.840 | 0.310 | 1.439 | 0.158 | 2.339
Dynamic | OM-CNN | 0.893 | 0.430 | 2.625 | 0.330 | 1.896
| ACLNet | 0.899 | 0.459 | 2.463 | 0.342 | 1.701
| SalEMA | 0.873 | 0.383 | 2.226 | 0.330 | 3.157
| STRA-Net | 0.913 | 0.558 | 3.226 | 0.459 | 2.251
| TASED-Net | 0.916 | 0.570 | 3.324 | 0.471 | 2.740
Ours | Spatial | 0.904 | 0.501 | 3.051 | 0.378 | 1.473
(Single) | Temporal | 0.898 | 0.489 | 2.581 | 0.362 | 1.468
Ours | Setting 1 | 0.914 | 0.549 | 3.114 | 0.413 | 1.277
(Gated) | Setting 2 | 0.919 | 0.563 | 3.201 | 0.424 | 1.242

  * •

* Non-deep learning model

Performance on DHF1K. We test the performance of our model on the recently
proposed DHF1K video saliency dataset, which includes 300 test videos. As
mentioned before, the annotations for the test split are not publicly
available and all the evaluations are carried out externally by the authors of
the dataset. As Table III shows, our proposed model achieves performance on
par with the state-of-the-art models. In terms of AUC-J, along with the recent
STRA-Net and TASED-Net models, it outperforms all the other saliency models.
In terms of CC, our model gives roughly the second best result.

Performance on DIEM. We also evaluate our model on DIEM test set consisting of
20 videos. Table IV summarizes these quantitative results. As can be seen, our
model achieves the highest scores in NSS and KLDiv metrics and very
competitive in others. The second setting demonstrates the generalization
capability of our proposed approach as compared to the recent models like
SalEMA, STRA-Net and TASED-Net.

TABLE III: Performance comparison on DHF1K dataset. The best and the second
best performing models are shown in bold typeface and underlined,
respectively.

Method Metric | AUC-J$\uparrow$ | CC$\uparrow$ | NSS$\uparrow$ | SIM$\uparrow$
---|---|---|---|---
Static | SalGAN | 0.866 | 0.370 | 2.043 | 0.262
| PQFT* | 0.699 | 0.137 | 0.749 | 0.139
| Fang et al.* | 0.819 | 0.273 | 1.539 | 0.198
| AWS-D* | 0.703 | 0.174 | 0.940 | 0.157
| Bak et al. | 0.834 | 0.325 | 1.632 | 0.197
Dynamic | OM-CNN | 0.856 | 0.344 | 1.911 | 0.256
| ACLNet | 0.890 | 0.434 | 2.354 | 0.315
| SalEMA | 0.890 | 0.449 | 2.574 | 0.466
| STRA-Net | 0.895 | 0.458 | 2.558 | 0.355
| TASED-Net | 0.895 | 0.470 | 2.667 | 0.361
Ours | Setting 1 | 0.891 | 0.448 | 2.505 | 0.326
(Gated) | Setting 2 | 0.895 | 0.457 | 2.528 | 0.321

  * •

* Non-deep learning model

TABLE IV: Performance comparison on DIEM dataset. The best and the second best
performing models are shown in bold typeface and underlined, respectively.

Method Metric | AUC-J$\uparrow$ | CC$\uparrow$ | NSS$\uparrow$ | SIM$\uparrow$ | KLDiv$\downarrow$
---|---|---|---|---|---
Static | SalGAN | 0.860 | 0.492 | 2.068 | 0.392 | 1.431
| PQFT* | 0.680 | 0.190 | 0.656 | 0.220 | 2.140
| Fang et al.* | 0.825 | 0.360 | 1.407 | 0.313 | 1.688
| AWS-D* | 0.768 | 0.313 | 1.228 | 0.272 | 1.825
| Bak et al. | 0.810 | 0.313 | 1.212 | 0.206 | 2.050
Dynamic | OM-CNN | 0.847 | 0.464 | 2.037 | 0.381 | 1.599
| ACLNet | 0.878 | 0.554 | 2.283 | 0.444 | 1.331
| SalEMA | 0.863 | 0.513 | 2.249 | 0.452 | 2.393
| STRA-Net | 0.864 | 0.527 | 2.277 | 0.456 | 2.461
| TASED-Net | 0.872 | 0.535 | 2.259 | 0.470 | 2.635
Ours | Spatial | 0.868 | 0.512 | 2.202 | 0.439 | 1.387
(Single) | Temporal | 0.846 | 0.446 | 1.785 | 0.391 | 1.513
Ours | Setting 1 | 0.870 | 0.543 | 2.313 | 0.454 | 1.401
(Gated) | Setting 2 | 0.874 | 0.525 | 2.228 | 0.421 | 1.176

  * •

* Non-deep learning model

GT |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Ours |  |  |  |  |  |  |  |
SalEMA |  |  |  |  |  |  |  |
ACLNet |  |  |  |  |  |  |  |
TASED-Net |  |  |  |  |  |  |  |
STRA-Net |  |  |  |  |  |  |  |
| (a) UCF-Sports | (b) Hollywood-2
GT |  |  |  |  |  |  |  |
Ours |  |  |  |  |  |  |  |
SalEMA |  |  |  |  |  |  |  |
ACLNet |  |  |  |  |  |  |  |
TASED-Net |  |  |  |  |  |  |  |
STRA-Net |  |  |  |  |  |  |  |
| (c) DHF1K | (d) DIEM

Figure 7: Qualitative results of our proposed framework and the deep learning
based SalEMA, ACLNet and SalGAN models. Our approach, in general, produces
more accurate saliency predictions than these state-of-the-art models.

In Fig. 7, we show some sample saliency maps predicted by our proposed model
and three other deep saliency networks: ACLNet, SalEMA, STRA-Net, and TASED-
Net models. As one can observe, our model makes generally better predictions
than the competing approaches. For instance, for the sequence from UCF-Sports
(Fig. 7(a)) most the models fail to identify the salient region on the
swimmer, or for the sequence from the Hollywood-2 dataset (Fig. 7(b)) our
model is the only model that correctly predicts the soldier at the center of
the background as salient. Similar kind of observations are also valid for the
sample sequences from DHF1K (Fig. 7(c)) and DIEM (Fig. 7(d)) datasets.

Performance on DIEM-Meta and LEDOV-Meta. As mentioned before, [67] have
recently showed that most of the current benchmarks for video saliency include
many sequences in which spatial attention is more dominant than temporal
effects in describing saliency. DIEM-Meta and LEDOV-Meta datasets are curated
in a special way to contain video frames in which temporal signals are found
to be more influential than appearance cues. Hence, they both offer a better
way to test how well a dynamic saliency model utilizes temporal information.
In our experimental evaluation, we compare our proposed model with the state-
of-the-art deep trackers, which are all trained on the combined training set
that includes frames from DIEM or LEDOV datasets. As can be seen from Table V
and Table VI, our model outperforms all the other models in DIEM-Meta, and is
the second best model in LEDOV-Meta, achieving highly competitive
performances. These results demonstrate the effectiveness of the proposed
gated mechanism and its ability to use temporal information to the full
extent, as compared to the state-of-the-art approaches.

Overall, the results reported on all the six datasets considered in our
experimental analysis suggest that our model has better capacity to mimic
human attention mechanism by combining the temporal and static clues in an
effective way. It has a better generalization ability that it can predict
where people look at the videos from unseen domains much better. Moreover, it
utilizes the temporal information more successfully with its gated fusion
mechanism, which adaptively integrates spatial and temporal cues depending on
video content.

TABLE V: Performance comparison on DIEM-Meta dataset. The best and the second
best performing models are shown in bold typeface and underlined,
respectively.

Method Metric | AUC-J$\uparrow$ | CC$\uparrow$ | NSS$\uparrow$ | SIM$\uparrow$ | KLDiv$\downarrow$
---|---|---|---|---|---
ACLNet | 0.845 | 0.437 | 1.627 | 0.391 | 1.473
SalEMA | 0.832 | 0.392 | 1.576 | 0.374 | 1.664
STRA-Net | 0.840 | 0.419 | 1.637 | 0.385 | 1.634
TASED-Net | 0.857 | 0.455 | 1.810 | 0.416 | 1.479
Ours | 0.857 | 0.460 | 1.814 | 0.395 | 1.305

Figure 8: Our model dynamically decides the contribution of motion and
appearance streams via gated fusion. Here, we plot the average motion
probabilities (the contribution of motion stream) for two regions having
different characteristic, one containing a moving object (the gummy bear) and
the other with relatively no motion, shown with red and blue, respectively. As
can be seen, our model assigns higher weights to the motion stream when motion
becomes the dominant visual cue, and the weights adaptively change throughout
the sequence. TABLE VI: Performance comparison on LEDOV-Meta dataset. The best
and the second best performing models are shown in bold typeface and
underlined, respectively.

Method Metric | AUC-J$\uparrow$ | CC$\uparrow$ | NSS$\uparrow$ | SIM$\uparrow$ | KLDiv$\downarrow$
---|---|---|---|---|---
ACLNet | 0.879 | 0.384 | 1.750 | 0.342 | 1.837
SalEMA | 0.863 | 0.380 | 1.815 | 0.353 | 1.850
STRA-Net | 0.893 | 0.423 | 2.041 | 0.370 | 2.304
TASED-Net | 0.882 | 0.489 | 2.450 | 0.403 | 1.697
Ours | 0.892 | 0.457 | 2.190 | 0.370 | 1.485

###  IV-E Ablation study.

In this section, we aim to analyze the influence of each component of our
proposed deep dynamic saliency model. We perform the ablation study on UCF-
Sports dataset by disabling or removing some blocks of our model and by
examining how these changes affect the model performance. As we did with
training our proposed model, for each version of our model under evaluation,
we first train a single stream model on SALICON dataset and then use this
model to finetune the actual two-stream version. Accordingly, Table VII
reports the performance of different versions of our saliency model.

Effect of gated fusion. As we emphasized before, the role of gated fusion
block is to adaptively integrate spatial and temporal streams is a key
component of our model. In our analysis, we replace the gated fusion block
with a standard $1\times 1$ convolution layer. As can be seen from Table VII,
the performance of the model decreases considerably without the gated fusion
mechanism. That is, using a dynamic weighting strategy, instead of a fixed
weighting scheme (learned via $1\times 1$ convolution), generates much better
predictions. Fig. 8 shows a visualization of how our proposed gated fusion
operates, demonstrating the behavior of the weighting scheme for both dynamic
and static parts of a given scene, In particular, we plot the motion
probabilities averaged within the corresponding image regions over time, which
clearly shows that the motion probability (the contribution of motion stream)
for the region that contains a moving object is, in general, much higher than
that of the static region. Moreover, depending on the characteristics of the
regions, it shows the changes in the motion probabilities throughout the whole
sequence. For example, when no motion is taking place in the region initially
containing the moving object, the weight of the temporal stream starts to
fall. These results supports our main claim that considering the content of
the video while combining temporal and spatial cues is a more appropriate way
to model saliency estimation on dynamic scenes.

Effect of multi-level information. Previous studies demonstrate that low and
high-level cues are equally important for saliency prediction [9, 10].
Motivated with these, we included a multi-level information block to fuse
features extracted from different levels of our deep model. For this analysis,
we disable this multi-level information block and train a single-scale model
instead. Compared to our full model, disabling this block reduces the
performance as can be seen in Table VII. Employing a representation that
contains information from low and high levels helps to improve the performance
of our model. We speculate that our multi-level information block allows the
network to better identify the regions semantically important for saliency.

TABLE VII: Ablation study on UCF-Sports dataset.

Method Metric | AUC-J$\uparrow$ | CC$\uparrow$ | NSS$\uparrow$ | SIM$\uparrow$ | KLDiv$\downarrow$
---|---|---|---|---|---
w/o spatial attention | 0.872 | 0.474 | 2.884 | 0.374 | 2.223
w/o channel-wise attention | 0.892 | 0.489 | 2.923 | 0.319 | 1.707
w/o spatial & ch.-wise attention | 0.875 | 0.447 | 2.885 | 0.364 | 2.646
w/o multi-level information | 0.890 | 0.484 | 2.755 | 0.303 | 1.711
w/o gated fusion | 0.900 | 0.480 | 2.913 | 0.353 | 1.676
full model | 0.914 | 0.526 | 3.333 | 0.382 | 1.516

Effect of attention blocks. As discussed before, the reasons we introduce the
attention blocks are to eliminate the irrelevant features via the spatial
attention and to choose the most informative feature channels via the channel-
wise attention when processing a video frame. In this experiment, we remove
the spatial and the channel-wise attention blocks from our full model and
train two different models, respectively. The results given in Table VII
support our assertion that both of these attention blocks improve the model
performance. Disabling them results in a much lower performance as compared to
that of the full model.

##  V Summary and Conclusion

In this study, we proposed a new spatio-temporal saliency network for video
saliency. It follows a two-stream network architecture that processes spatial
and temporal information in separate streams, but it extends the standard
structure in many ways. First, it includes a gated fusion block that performs
integration of spatial and temporal streams in a more dynamic manner by
deciding the contribution of each channel one frame at a time. Second, it
utilizes a multi-level information block that allows for performing multi-
scale processing of appearance and motion features. Finally, it employs
spatial and channel-wise attention blocks to further increase the selectivity.
Our extensive set of experiments on six different benchmark datasets shows the
effectiveness of the proposed model in extracting the most salient parts of
the video frames both qualitatively and quantitatively. Moreover, our ablation
study demonstrates the gains achieved by each component of our model. Our
analysis reveals that the proposed model deals with the videos from unseen
domains much better that the existing dynamic saliency models. Additionally,
it uses temporal cues more effectively via the proposed gated fusion mechanism
which allows for adaptive integration of spatial and temporal streams.

We believe that our work highlights several important directions to pursue for
better modeling of saliency in videos. As future work, we plan to explore more
efficient ways to include the temporal information. For instance, instead of
using optical flow images, one can use features extracted from early and mid
layers of an optical flow network model to encode motion information. This can
reduce the memory footprint of the model and decreases the running times.

## Acknowledgments

This work was supported in part by TUBA GEBIP fellowship awarded to E. Erdem.

## References

  * [1] L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual attention for rapid scene analysis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 20, no. 11, pp. 1254–1259, 1998.
  * [2] A. Borji and L. Itti, “State-of-the-art in visual attention modeling,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 1, pp. 185–207, 2013.
  * [3] S. Filipe and L. A. Alexandre, “Retracted article: From the human visual system to the computational models of visual attention: a survey,” _Artificial Intelligence Review_ , vol. 43, no. 4, pp. 601–601, 2015.
  * [4] H. Kim and S. Lee, “Transition of visual attention assessment in stereoscopic images with evaluation of subjective visual quality and discomfort,” _IEEE Transactions on Multimedia_ , vol. 17, no. 12, pp. 2198–2209, 2015\.
  * [5] K. Gu, S. Wang, H. Yang, W. Lin, G. Zhai, X. Yang, and W. Zhang, “Saliency-guided quality assessment of screen content images,” _IEEE Transactions on Multimedia_ , vol. 18, no. 6, pp. 1098–1110, 2016.
  * [6] Y. Fang, Z. Chen, W. Lin, and C. Lin, “Saliency detection in the compressed domain for adaptive image retargeting,” _IEEE Transactions on Image Processing_ , vol. 21, no. 9, pp. 3888–3901, 2012.
  * [7] D. Chen and Y. Luo, “Preserving motion-tolerant contextual visual saliency for video resizing,” _IEEE Transactions on Multimedia_ , vol. 15, no. 7, pp. 1616–1627, 2013.
  * [8] G. Evangelopoulos, A. Zlatintsi, A. Potamianos, P. Maragos, K. Rapantzikos, G. Skoumas, and Y. Avrithis, “Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention,” _IEEE Transactions on Multimedia_ , vol. 15, no. 7, pp. 1553–1568, 2013.
  * [9] N. D. B. Bruce, C. Catton, and S. Janjic, “A deeper look at saliency: Feature contrast, semantics, and beyond,” in _CVPR_ , 2016, pp. 516–524.
  * [10] Z. Bylinskii, A. Recasens, A. Borji, A. Oliva, A. Torralba, and F. Durand, “Where should saliency models look next?” in _Proc. ECCV_ , 2016, pp. 809–824.
  * [11] C. Guo, Q. Ma, and L. Zhang, “Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform,” in _Proc. CVPR_ , 2008, pp. 1–8.
  * [12] X. Cui, Q. Liu, and D. Metaxas, “Temporal spectral residual: fast motion saliency detection,” in _ACM MM_ , 2009, pp. 617–620.
  * [13] H. J. Seo and P. Milanfar, “Static and space-time visual saliency detection by self-resemblance,” _Journal of Vision_ , vol. 9, no. 12, pp. 15–15, 2009\.
  * [14] W. Sultani and I. Saleemi, “Human action recognition across datasets by foreground-weighted histogram decomposition,” in _Proc. CVPR_ , 2014, pp. 764–771.
  * [15] T. Mauthner, H. Possegger, G. Waltner, and H. Bischof, “Encoding based saliency detection for videos and images,” in _CVPR_ , 2015, pp. 2494–2502.
  * [16] Y. Luo and Q. Tian, “Spatio-temporal enhanced sparse feature selection for video saliency estimation,” in _2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops_ , June 2012, pp. 33–38.
  * [17] S. Mathe and C. Sminchisescu, “Dynamic eye movement datasets and learnt saliency models for visual action recognition,” in _Proc. ECCV_ , 2012, pp. 842–856.
  * [18] D. Rudoy, D. B. Goldman, E. Shechtman, and L. Zelnik-Manor, “Learning video saliency from human gaze using candidate selection,” in _Proc. CVPR_ , 2013, pp. 1147–1154.
  * [19] S. Zhong, Y. Liu, F. Ren, J. Zhang, and T. Ren, “Video saliency detection via dynamic consistent spatio-temporal attention modelling,” in _Proc., AAAI_ , 2013.
  * [20] Z. Liu, X. Zhang, S. Luo, and O. Le Meur, “Superpixel-based spatiotemporal saliency detection,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 24, no. 9, pp. 1522–1540, 2014.
  * [21] F. Zhou, S. B. Kang, and M. F. Cohen, “Time-mapping using space-time saliency,” in _Proc. CVPR_ , 2014, pp. 3358–3365.
  * [22] J. Zhao, C. Siagian, and L. Itti, “Fixation bank: Learning to reweight fixation candidates,” in _CVPR_ , 2015, pp. 3174–3182.
  * [23] S. H. Khatoonabadi, N. Vasconcelos, I. V. Bajić, and Yufeng Shan, “How many bits does it take for a stimulus to be salient?” in _Proc. CVPR_ , 2015, pp. 5501–5510.
  * [24] V. Leborán, A. García-Díaz, X. R. Fdez-Vidal, and X. M. Pardo, “Dynamic whitening saliency,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 5, pp. 893–907, 2017.
  * [25] L. Bazzani, H. Larochelle, and L. Torresani, “Recurrent mixture density network for spatiotemporal visual attention,” in _Proc. ICLR_ , 2017.
  * [26] W. Wang, J. Shen, J. Xie, M. Cheng, H. Ling, and A. Borji, “Revisiting video saliency prediction in the deep learning era,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019.
  * [27] A. Palazzi, D. Abati, S. Calderara, F. Solera, and R. Cucchiara, “Predicting the driver’s focus of attention: the dr(eye)ve project,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2018.
  * [28] C. Bak, A. Kocak, E. Erdem, and A. Erdem, “Spatio-temporal saliency networks for dynamic saliency prediction,” _IEEE Transactions on Multimedia_ , vol. 20, no. 7, pp. 1688–1698, 2018.
  * [29] L. Jiang, M. Xu, T. Liu, M. Qiao, and Z. Wang, “Deepvs: A deep learning based video saliency prediction approach,” in _Proc. ECCV_ , 2018, pp. 625–642.
  * [30] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in _CVPR_ , 2014, pp. 1725–1732.
  * [31] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in _Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS)_ , 2014, p. 568–576.
  * [32] M. A. Goodale and A. D. Milner, “Separate visual pathways for perception and action,” _Trends in Neurosciences_ , vol. 15, no. 1, pp. 20–?25, 1992.
  * [33] A. M. Treisman and G. Gelade, “A feature-integration theory of attention,” _Cognitive Psychology_ , vol. 12, no. 1, pp. 97–136, 1980.
  * [34] C. Koch and S. Ullman, “Shifts in selective visual attention: Towards the underlying neural circuitry,” _Human neurobiology_ , vol. 4, pp. 219–27, 1985.
  * [35] X. Huang, C. Shen, X. Boix, and Q. Zhao, “SALICON: Reducing the semantic gap in saliency prediction by adapting deep neural networks,” in _Proc. ICCV_ , 2015, pp. 262–270.
  * [36] S. Jetley, N. Murray, and E. Vig, “End-to-end saliency mapping via probability distribution prediction,” in _Proc. CVPR_ , 2016, pp. 5753–5761.
  * [37] S. S. S. Kruthiventi, K. Ayush, and R. V. Babu, “Deepfix: A fully convolutional neural network for predicting human eye fixations,” _IEEE Transactions on Image Processing_ , vol. 26, no. 9, pp. 4446–4456, 2017.
  * [38] N. Liu, J. Han, T. Liu, and X. Li, “Learning to predict eye fixations via multiresolution convolutional neural networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 29, no. 2, pp. 392–404, 2018.
  * [39] J. Pan, E. Sayrol, X. Giró-i Nieto, K. McGuinness, and N. E. OConnor, “Shallow and deep convolutional networks for saliency prediction,” in _Proc. CVPR_ , 2016, pp. 598–606.
  * [40] W. Wang and J. Shen, “Deep visual attention prediction,” _IEEE Transactions on Image Processing_ , vol. 27, no. 5, pp. 2368–2378, 2018.
  * [41] E. Vig, M. Dorr, and D. Cox, “Large-scale optimization of hierarchical features for saliency prediction in natural images,” in _Proc. CVPR_ , 2014, pp. 2798–2805.
  * [42] M. Cornia, L. Baraldi, G. Serra, and R. Cucchiara, “Predicting human eye fixations via an lstm-based saliency attentive model,” _IEEE Transactions on Image Processing_ , vol. 27, no. 10, pp. 5142–5154, 2018.
  * [43] J. Harel, C. Koch, and P. Perona, “Graph-based visual saliency,” in _Proceedings of the 19th International Conference on Neural Information Processing Systems (NIPS)_ , 2006, pp. 545–552.
  * [44] Y. Fang, Z. Wang, W. Lin, and Z. Fang, “Video saliency incorporating spatiotemporal cues and uncertainty weighting,” _IEEE Transactions on Image Processing_ , vol. 23, no. 9, pp. 3910–3921, 2014.
  * [45] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in _Proc. ICCV_ , 2015, pp. 4489–4497.
  * [46] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” in _Proc. ICLR_ , 2016.
  * [47] P. Linardos, E. Mohedano, J. J. Nieto, K. McGuinness, X. Giro-i Nieto, and N. E. O’Connor, “Simple vs complex temporal recurrences for video saliency prediction,” in _Proc. BMVC_ , 2019.
  * [48] K. Min and J. J. Corso, “Tased-net: Temporally-aggregating spatial encoder-decoder network for video saliency detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 2394–2403.
  * [49] Q. Lai, W. Wang, H. Sun, and J. Shen, “Video saliency prediction using spatiotemporal residual attentive networks,” _IEEE Trans. on Image Processing_ , 2019.
  * [50] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. CVPR_ , 2016, pp. 770–778.
  * [51] T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proc. CVPR_ , 2017, pp. 936–944.
  * [52] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Proc. MICCAI_ , 2015, pp. 234–241.
  * [53] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proc. CVPR_ , 2015, pp. 3431–3440.
  * [54] P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollár, “Learning to refine object segments,” in _Proc. ECCV_ , 2016, pp. 75–91.
  * [55] T. Zhao and X. Wu, “Pyramid feature attention network for saliency detection,” in _Proc. CVPR_ , 2019, pp. 3080–3089.
  * [56] S. Dong, Z. Gao, S. Sun, X. Wang, M. M. Li, H. Zhang, G. Yang, H. Liu, and S. Li, “Holistic and deep feature pyramids for saliency detection,” in _Proc. BMVC_ , 2018.
  * [57] V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu, “Recurrent models of visual attention,” in _Proc. NIPS_ , 2014.
  * [58] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in _Proc. ICML_ , 2015.
  * [59] X. Chu, W. Yang, W. Ouyang, C. Ma, A. L. Yuille, and X. Wang, “Multi-context attention for human pose estimation,” in _Proc. CVPR_ , 2017, pp. 5669–5678.
  * [60] L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, W. Liu, and T.-S. Chua, “SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning,” in _Proc. CVPR_ , 2017.
  * [61] W. Ren, L. Ma, J. Zhang, J. Pan, X. Cao, W. Liu, and M.-H. Yang, “Gated fusion network for single image dehazing,” in _Proc. CVPR_ , 2018.
  * [62] X. Zhang, H. Dong, Z. Hu, W.-S. Lai, F. Wang, and M.-H. Yang, “Gated fusion network for joint image deblurring and super-resolution,” in _Proc. BMVC_ , 2018.
  * [63] Y. Cheng, R. Cai, Z. Li, X. Zhao, and K. Huang, “Locality-sensitive deconvolution networks with gated fusion for RGB-D indoor semantic segmentation,” in _Proc. CVPR_ , 2017, pp. 1475–1483.
  * [64] M. D. Rodriguez, J. Ahmed, and M. Shah, “Action MACH a spatio-temporal maximum average correlation height filter for action recognition,” in _Proc. CVPR_ , 2008.
  * [65] S. Mathe and C. Sminchisescu, “Actions in the eye: Dynamic gaze datasets and learnt saliency models for visual recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 7, pp. 1408–1424, 2015.
  * [66] P. K. Mital, T. J. Smith, R. L. Hill, and J. M. Henderson, “Clustering of gaze during dynamic scene viewing is predicted by motion,” _Cognitive Computation_ , vol. 3, no. 1, pp. 5–24, 2011.
  * [67] M. Tangemann, M. Kümmerer, T. S. Wallis, and M. Bethge, “Measuring the importance of temporal features in video saliency,” 2020.
  * [68] T. Lan, Y. Wang, and G. Mori, “Discriminative figure-centric models for joint action localization and recognition,” in _Proc. ICCV_ , 2011, pp. 2003–2010.
  * [69] M. Marszalek, I. Laptev, and C. Schmid, “Actions in context,” in _Proc. CVPR_ , 2009, pp. 2929–2936.
  * [70] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” in _Proc. CVPR_ , 2018.
  * [71] M. Jiang, S. Huang, J. Duan, and Q. Zhao, “SALICON: Saliency in context,” in _Proc. CVPR_ , 2015, pp. 1072–1080.
  * [72] X. Huang, C. Shen, X. Boix, and Q. Zhao, “SALICON: Reducing the semantic gap in saliency prediction by adapting deep neural networks,” in _Proc. ICCV_ , 2015, pp. 262–270.
  * [73] Z. Bylinskii, T. Judd, A. Oliva, A. Torralba, and F. Durand, “What do different evaluation metrics tell us about saliency models?” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 3, pp. 740–757, 2019.
  * [74] J. Pan, C. Canton, K. McGuinness, N. E. O’Connor, J. Torres, E. Sayrol, and X. a. Giro-i Nieto, “SalGAN: Visual saliency prediction with generative adversarial networks,” in _arXiv_ , 2017.


# Brown-York Energy in Spacetimes with Horizon

Jarmo Mäkelä<EMAIL_ADDRESS>Vaasa University of Applied Sciences,
Wolffintie 30, 65200 Vaasa, Finland

###### Abstract

We obtain a simple relationship between the change in the Brown-York energy
inside of a closed two-surface just outside of a horizon of spacetime, and the
change in the area of that two-surface.

Ever since the publication of Einstein’s general relativity more than hundred
years ago physicists have introduced several concepts of energy in curved
spacetime. One of them is the Brown-York energy, which is defined in static,
asymptotically flat spacetimes as: yy

$E_{BY}:=-\frac{1}{8\pi}\oint_{S}(k-k_{0})\,dS.$ (1)

In this definition $S$ is a closed, two-dimensional surface embedded into a
spacelike hypersurface of spacetime, where the time coordinate $t=constant$.
$dS$ denotes the area element on that two-surface, and we have integrated over
the two-surface. $k$ is the trace of the exterior curvature tensor on the two-
surface, and $k_{0}$ is the trace of the exterior curvature tensor in the
limit, where the two-surface has been carried to the spatial infinity.

The Brown-York energy may be understood as the total energy inside of the
closed two-surface $S$. In this paper we shall show that when the two-surface
$S$ is just outside of a closed, compact horizon of spacetime, the Brown-York
energy has a very interesting property: If the proper acceleration vector
field $a^{\mu}$ of an observer at rest on the two-surface $S$ has a constant
norm $a$, is perpendicular to $S$ at every point of $S$, and we change both
the geometry of spacetime and the two-surface $S$ such that the proper
acceleration $a$ is kept unchanged, then between the change $dE_{BY}$ in the
Brown-York energy $E_{BY}$ and the change $dA$ in the area $A$ of the two-
surface $S$ there is the relationship:

$dE_{BY}=\frac{a}{8\pi}dA.$ (2)

In Ref. kaa  the validity of Eq. (2) was proved for the Schwarzschild
spacetime, in Ref. koo  for the general, sphere symmetric spacetime with
horizon, and in Ref. nee  for the Kerr-Newman spacetime. Since the proper
acceleration is a constant, Eq. (2) suggests that we may consider the quantity

$E=\frac{a}{8\pi}A$ (3)

as the total energy inside of the surface $S$ from the point of view of an
observer with constant proper acceleration $a$ just outside of a horizon of
spacetime. Indeed, the energy of Eq. (3) was used as the energy of the
gravitational field in Refs. kaa ; koo ; nee ; vii ; kuu ; seite1 . A closely
related energy concept has been introduced in Ref. seite .

As the starting point of the general derivation of Eq. (2), which is still
lacking, we use the properties of the Rindler spacetime. As it is well known,
Rindler spacetime is just two-dimensional, flat spacetime equipped with the
so-called Rindler coordinates $t$ and $x$ such that the line element takes the
form: kasi

$ds^{2}=-x^{2}\,dt^{2}+dx^{2}.$ (4)

In this equation the spatial coordinate $x$ is related to the proper
acceleration $a$ of the observer such that

$x=\frac{1}{a}.$ (5)

In other words, an observer with constant $x$ has constant proper acceleration
$a$. The Rindler coordinates $t$ and $x$ are related to the flat Minkowski
coordinates $T$ and $X$ such that:

$\displaystyle t$ $\displaystyle=$
$\displaystyle\tanh^{-1}\left(\frac{T}{X}\right),$ (6a) $\displaystyle x$
$\displaystyle=$ $\displaystyle\sqrt{X^{2}-T^{2}}.$ (6b)

The line element (4) is singular, when $x=0$, and Eq. (6b) implies that

$T=\pm X,$ (7)

when $x=0$. The lines $T=\pm X$ constitute the so-called Rindler horizon from
the point of view of the accelerating observer. According to Eq. (5) the
proper acceleration $a$ of the observer tends to infinity at the Rindler
horizon.

We now proceed from the two-dimensional flat Rindler spacetime to a four-
dimensional curved spacetime equipped with a horizon. As the first step we
define a new spatial coordinate

$y:=x^{2},$ (8)

and Eq. (4) takes the form:

$ds^{2}=-y\,dt^{2}+\frac{1}{4y}\,dy^{2}.$ (9)

We consider a spacetime metric

$ds^{2}=-y\,dt^{2}+\frac{1}{4y}\,dy^{2}+q_{jk}\,d\chi^{j}\,d\chi^{k},$ (10)

where $j,k=2,3$. In this equation $q_{jk}$ is the metric tensor induced on a
spacelike, closed two-surface $S$ embedded into the spacelike hypersurface of
spacetime, where the time coordinate $t=constant$. $\chi^{2}$ and $\chi^{3}$
are the coordinates on that two-surface. In what follows, we shall assume that
$y$ is an injective, increasing function of coordinate $\lambda$, and an
arbitrary function of $n$ parameters $\alpha^{1},\alpha^{2},\dots,\alpha^{n}$.
In other words,

$y=y(\lambda;\alpha^{1},\alpha^{2},\dots,\alpha^{n}).$ (11)

In contrast, the metric tensor $q_{jk}$ is assumed to be a function of
$\lambda$ and the coordinates $\chi^{2}$ and $\chi^{3}$ i. e.

$q_{jk}=q_{jk}(\lambda,\chi^{2},\chi^{3}).$ (12)

Hence we find that if we denote:

$y^{\prime}:=\frac{\partial y}{\partial\lambda},$ (13)

Eq. (10) takes the form:

$ds^{2}=-y\,dt^{2}+\frac{1}{4y}(y^{\prime})^{2}\,d\lambda^{2}+q_{jk}\,d\chi^{j}\,d\chi^{k}.$
(14)

The coordinate $\lambda$ determines the location of the two-surface $S$ in
spacetime, whereas the parameters $\alpha^{s}$ $(s=1,2,\dots,n)$ determine the
overall geometrical properties of spacetime. For instance, if spacetime
contains a black hole, the parameters may be, for example, the mass and the
electric charge of the hole. Since the metric tensor $q_{jk}$ is a function of
$\lambda$, the proper acceleration $a$ is everywhere a constant on the two-
surface $S$, and it is perpendicular to $S$. Spacetime equipped with the
metric (14) has the horizon, when $y=0$. We choose the coordinate $\lambda$
such that

$y(\lambda_{0})=0,$ (15)

and hence $\lambda=\lambda_{0}$ at the horizon. Since $y$ is assumed to be an
increasing, injective function of $\lambda$, we must have

$y^{\prime}(\lambda_{0})>0.$ (16)

In general, the proper acceleration vector field of an observer is

$a^{\mu}=u^{\sigma}u^{\mu}_{;\sigma},$ (17)

where $u^{\mu}$ is the future pointing unit tangent vector field of the world
line of the observer. For an observer at rest with respect to the spatial
coordinates $\lambda$, $\chi^{2}$ and $\chi^{3}$ the only non-zero component
of $u^{\mu}$ is

$u^{t}=\frac{1}{\sqrt{y}},$ (18)

and therefore the only non-zero component of $a^{\mu}$ is

$a^{\lambda}=u^{t}u^{\lambda}_{;t}=\Gamma^{\lambda}_{tt}(u^{t})^{2}=\frac{2}{y^{\prime}}.$
(19)

As one may observe, $a^{\mu}$ is indeed perpendicular to our two-surface $S$.
The norm of $a^{\mu}$ is:

$a=\sqrt{a_{\lambda}a^{\lambda}}=\frac{1}{\sqrt{y}},$ (20)

which is exactly what one might have expected on grounds of Eqs. (5) and (8).
The exterior curvature tensor on the two-surface $S$, in turn, is:

$k_{jk}=-\frac{y^{\prime}}{2\sqrt{y}}\Gamma^{\lambda}_{jk}=\frac{\sqrt{y}}{y^{\prime}}{q^{\prime}}_{jk},$
(21)

and its trace is:

$k=q^{jk}k_{jk}=\frac{\sqrt{y}}{y^{\prime}}q^{jk}{q^{\prime}}_{jk}.$ (22)

In Eqs. (21) and (22) the prime means the derivative with respect to
$\lambda$. Hence the Brown-York energy of Eq. (1) takes the form:

$E_{BY}=-\frac{1}{8\pi}\oint_{S}\left(\frac{\sqrt{y}}{y^{\prime}}q^{jk}{q^{\prime}}_{jk}-k_{0}\right)\sqrt{q}\,d\chi^{2}\,d\chi^{3},$
(23)

where we have integrated the coordinates $\chi^{2}$ and $\chi^{3}$ over the
two-surface $S$.

Consider now what happens to the Brown-York energy, if we change the
geometrical parameters $\alpha^{s}$ of spacetime and, at the same time, the
location of the two-surface $S$ in such a way that the proper acceleration is
kept as a constant. Since the proper acceleration $a$ depends on $y$ as in Eq.
(20), $y$ must be kept as a constant. This means that if the parameters
$\alpha^{s}$ take on small changes $d\alpha^{s}$, the coordinate $\lambda$
takes on the change $d\lambda$ such that:

$dy=\frac{\partial y}{\partial\lambda}\,d\lambda+\frac{\partial
y}{\partial\alpha^{s}}\,d\alpha^{s}=0,$ (24)

which implies:

$\frac{\partial y}{\partial\alpha^{s}}\,d\alpha^{s}=-y^{\prime}\,d\lambda.$
(25)

Because $y$ depends on the parameters $\alpha^{s}$, but the metric tensor
$q_{jk}$ does not, the change taken by the Brown-York energy is:

$dE_{BY}=\frac{\partial
E_{BY}}{\partial\alpha^{s}}\,d\alpha^{s}=-\frac{1}{8\pi}\left\\{\oint_{S}\left[\frac{1}{2\sqrt{y}y^{\prime}}\frac{\partial
y}{\partial\alpha^{s}}-\frac{\sqrt{y}}{(y^{\prime})^{2}}\frac{\partial
y^{\prime}}{\partial\alpha^{s}}\right]q^{jk}{q^{\prime}}_{jk}\sqrt{q}\,d\chi^{2}\,d\chi^{3}\right\\}\,d\alpha^{s}.$
(26)

At the horizon $y=0$, whereas $y^{\prime}>0$. Hence we may ignore the second
term inside of the brackets, when we are just outside of the horizon.
Employing Eqs. (20) and (25) we may write $dE_{BY}$ in terms of the proper
acceleration $a$ and the change $d\lambda$ in the coordinate $\lambda$ as:

$dE_{BY}=\frac{a}{8\pi}\left(\oint_{S}\frac{1}{2}q^{jk}{q^{\prime}}_{jk}\sqrt{q}\,d\chi^{2}\,d\chi^{2}\right)\,d\lambda,$
(27)

or:

$dE_{BY}=\frac{a}{8\pi}\,dA,$ (28)

where

$dA=\frac{dA}{d\lambda}\,d\lambda=\left[\oint_{S}\left(\frac{d}{d\lambda}\sqrt{q}\right)\,d\chi^{2}\,d\chi^{3}\right]\,d\lambda=\left(\oint_{S}\frac{1}{2}q^{jk}{q^{\prime}}_{jk}\sqrt{q}\,d\chi^{2}\,d\chi^{2}\right)\,d\lambda,$
(29)

is the change in the area

$A=\oint_{S}\sqrt{q}\,d\chi^{2}\,d\chi^{3}$ (30)

of the two-surface $S$. Hence we have managed to derive Eq. (2) in a very
general case.

As an example, consider the Schwarzschild spacetime equipped with the line
element

$ds^{2}=-\left(1-\frac{2M}{r}\right)\,dt^{2}+\frac{dr^{2}}{1-\frac{2M}{r}}+r^{2}\,d\theta^{2}+r^{2}\sin^{2}(\theta)\,d\phi^{2},$
(31)

where $M$ is the Schwarzshild mass. The only non-zero components of the
exterior curvature tensor on the two-sphere, where $r=constant$ are:

$\displaystyle k_{\theta\theta}$ $\displaystyle=$
$\displaystyle-\frac{1}{\sqrt{1-\frac{2M}{r}}}\Gamma^{r}_{\theta\theta}=\sqrt{1-\frac{2M}{r}}r,$
(32a) $\displaystyle k_{\phi\phi}$ $\displaystyle=$
$\displaystyle-\frac{1}{\sqrt{1-\frac{2M}{r}}}\Gamma^{r}_{\phi\phi}=\sqrt{1-\frac{2M}{r}}r\sin^{2}(\theta),$
(32b)

and therefore the Brown-York energy of Eq. (1) inside of the two-sphere takes
the form:

$E_{BY}=-\frac{1}{8\pi}\left(\sqrt{1-\frac{2M}{r}}\frac{2}{r}-\frac{2}{r}\right)4\pi
r^{2}=-r\sqrt{1-\frac{2M}{r}}+r.$ (33)

The only non-zero component of the proper acceleration vector field $a^{\mu}$
of an observer at rest on the two-sphere is:

$a^{r}=u^{t}u^{r}_{;t}=\Gamma_{tt}^{r}(u^{t})^{2}=\frac{M}{r^{2}},$ (34)

and its norm is:

$a=\sqrt{a_{r}a^{r}}=\left(1-\frac{2M}{r}\right)^{-1/2}\frac{M}{r^{2}}.$ (35)

The geometrical parameter of spacetime is now the Schwarzschild mass $M$,
whereas the coordinate $\lambda$ is represented by $r$. If $M$ and $r$ take on
small changes $dM$ and $dr$ such that $a$ is kept as a constant, we must have:

$da=\frac{\partial a}{\partial r}\,dr+\frac{\partial a}{\partial M}\,dM=0,$
(36)

or:

$dM=\frac{M^{2}+\left(1-\frac{2M}{r}\right)2Mr}{Mr+\left(1-\frac{2M}{r}\right)r^{2}}\,dr.$
(37)

The resulting change in the Brown-York energy is:

$dE_{BY}=\frac{\partial E_{BY}}{\partial
M}\,dM=\left(1-\frac{2M}{r}\right)^{-1/2}\frac{M^{2}+\left(1-\frac{2M}{r}\right)2Mr}{Mr+\left(1-\frac{2M}{r}\right)r^{2}}\,dr,$
(38)

where we have used Eq. (37). Just outside of the horizon, where $r=2M$ we may
write, in the leading approximation:

$dE_{BY}=\left(1-\frac{2M}{r}\right)^{-1/2}\frac{M}{r^{2}}r\,dr,$ (39)

and hence Eq. (35) implies:

$dE_{BY}=\frac{a}{8\pi}\,dA,$ (40)

where $dA=8\pi r\,dr$ is the change in the area of the two-sphere. The result
is consistent with Eq. (2).

In Ref. seite1  it was shown that in the Reissner-Nordström spacetime the
change $dE_{BY}$ in the Brown-York energy inside of a closed two-surface just
outside of the event horizon of the Reissner-Nordström black hole equals with
the amount of energy carried by the matter fields through that surface from
the point of view of an observer at rest with respect to that surface. There
are good grounds to believe the result to be of general validity. It would be
interesting to study the possible relationship of this result with an
interesting hypothesis published in Refs. ysi  and kymppi  that at the horizon
of spacetime the sum of the gravitational and the non-gravitational energies
equals zero. This topic, however, is very wide, and its proper investigation
must therefore be left to another publication.

In this paper we have managed to obtain a very simple relationship between the
change in the Brown-York energy inside of a closed two-surface just outside of
a horizon of spacetime, and the change in the area of that two-surface. The
essential point in our derivation was an assumption that the proper
acceleration $a$ of an observer at rest with respect to the surface was kept
as a constant during the process. If we keep the proper acceleration as a
constant, the changes in the Brown-York energy reflect the changes in the
true, physical properties of spacetime, rather than the changes in the state
of motion of the observer. Our result suggests that from the point of view of
our observer the energy operator in curved spacetime takes an extremely simple
form:

$\hat{H}=\frac{a}{8\pi}\hat{A}.$ (41)

Since our two-surface is just outside of the horizon, we may identify the
operator $\hat{A}$ as the area operator of the horizon, and it depends on the
model of quantum gravity. Loop quantum gravity-related consequences of Eq.
(41) in black hole thermodynamics and cosmology have been considered, for
instance, in Ref. nee .

## References

  * (1) D. Brown and J. W. York, Phys. Rev. D47 (1993), 1407.
  * (2) J. Mäkelä, Entropy 13 (2011) 1324.
  * (3) J. Mäkelä, Phys. Rev. D91 (2015) 124050.
  * (4) J. Mäkelä, Int. J. Mod. Phys. D28 (2019) 1950129.
  * (5) J. Mäkelä, Phys. Rev. D87 (2013) 104040.
  * (6) J. Mäkelä, Phys. Rev. D93 (2016) 084002.
  * (7) J. Mäkelä, Int. J. Mod. Phys. D23 (2104) 1450001.
  * (8) E. Frodden, A. Gosh and A. Perez, Phys. Rev. D87 (2013) 121503.
  * (9) See, for example, R. M. Wald, General Relativity (The University of Chicago Press, Chicago 1984).
  * (10) N. Dadhich, Curr. Sci. 76 (1999) 831.
  * (11) S. Chakraborty, N. Dadhich, JHEP 12 (2015) 003.