# Viewport Prediction for Volumetric Video Streaming by Exploring Video
Saliency and Trajectory Information

Jie Li, Zhixin Li, Zhi Liu, , Pengyuan Zhou, , Richang Hong, , Qiyue Li, , Han
Hu Jie Li, Zhixin Li and Richang Hong are with the School of Computer Science
and Information Engineering, Hefei University of Technology, Hefei, China
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>hongrc.hfut@gmail.com).Qiyue Li is with the School of Electrical Engineering
and Automation, Hefei University of Technology, Hefei, China, and also with
the Engineering Technology Research Center of Industrial Automation of Anhui
Province, Hefei, China (e-mail: liqiyue@mail.ustc.edu.cn)Zhi Liu is with The
University of Electro-Communications, Japan (e-mail:
liuzhi@uec.ac.jp).Pengyuan Zhou is with University of Science and Technology
of China, Hefei, China. (e-mail: pyzhou@ustc.edu.cn).Han Hu is with the School
of Information and Electronics, Beijing Institute of Technology, Beijing,
China (e-mail: huhan627@gmail.com).Corresponding author: Zhi Liu.

###### Abstract

Volumetric video, also known as hologram video, is a novel medium that
portrays natural content in Virtual Reality (VR), Augmented Reality (AR), and
Mixed Reality (MR). It is expected to be the next-gen video technology and a
prevalent use case for 5G and beyond wireless communication. Considering that
each user typically only watches a section of the volumetric video, known as
the viewport, it is essential to have precise viewport prediction for optimal
performance. However, research on this topic is still in its infancy. In the
end, this paper presents and proposes a novel approach, named Saliency and
Trajectory Viewport Prediction (STVP), which aims to improve the precision of
viewport prediction in volumetric video streaming. The STVP extensively
utilizes video saliency information and viewport trajectory. To our knowledge,
this is the first comprehensive study of viewport prediction in volumetric
video streaming. In particular, we introduce a novel sampling method, Uniform
Random Sampling (URS), to reduce computational complexity while still
preserving video features in an efficient manner. Then we present a saliency
detection technique that incorporates both spatial and temporal information
for detecting static, dynamic geometric, and color salient regions. Finally,
we intelligently fuse saliency and trajectory information to achieve more
accurate viewport prediction. We conduct extensive simulations to evaluate the
effectiveness of our proposed viewport prediction methods using state-of-the-
art volumetric video sequences. The experimental results show the superiority
of the proposed method over existing schemes. The dataset and source code will
be publicly accessible after acceptance.

###### Index Terms:

viewport prediction, volumetric video, trajectory prediction, saliency
detection, sampling

##  I Introduction

Volumetric video, also known as holographic video, allows users to fully
immerse themselves in a 3D scene and move freely in any direction, providing 6
degrees of freedom (6DoF) experience [1, 2]. Typically, users watch only a
section of a video at a time. This specific section is referred to as a
"viewport", and users have the ability to switch between viewports
unrestrictedly as they watch. Its versatility makes it suitable for
entertainment, education, and manufacturing applications, among others, and it
is expected to become the next-generation video technology [3, 4].

Point cloud video is a prevalent kind of volumetric video, presenting various
challenges due to the high bandwidth and low-latency requirements for
transmission. Tiling and adaptive streaming technologies have been proposed to
address these limitations by dividing the point cloud video into multiple
tiles [5]. Additionally, this design only requires the server to transmit the
tiles located within the user’s viewport [6]111Viewport, also referred to as
the field of view (FoV), is the portion of a video that a user actually views.
In this paper, the terms “viewport” and “FoV” are used interchangeably..
However, since an inaccurate estimate of the viewport may lead to a
deterioration in the user’s viewing experience, accurate viewport prediction
is critical for optimal point cloud video transmission, as illustrated in Fig.
1.

Figure 1: Illustration of tiling and FoV based adaptive point cloud video
streaming. Accurate FoV prediction can lead to a perfect viewing experience,
bad prediction results in wasted bandwidth or freeze.

When watching point cloud videos, users are often attracted to visually
salient content, such as vibrant hues, noticeable geometric shapes, and
motion. Furthermore, the user trajectory we recorded consists of head position
and orientation exhibits temporal coherence, which can be predicted with deep
learning techniques[7]. Consequently, the saliency features of a point cloud
video and the user’s trajectory can be used to accurately predict the user’s
viewport.

In this paper, we are motivated to study the viewport prediction problem in
point cloud video streaming, which can greatly facilitate the point cloud
video streaming. As far as we are aware, our study is the first to
comprehensively investigate viewport prediction in point cloud video streaming
while fully factoring in video saliency and trajectory information. The
research in this field is still in nascent stages, so it attracts considerable
attention from both the academic and industrial sectors. A more detailed
analysis of these related works is presented in the following section.

TABLE I: Comparison of Related Viewport Prediction Work

Reference | Video Type | Degree of Freedom | Using head trajectory | Using video saliency | Network
---|---|---|---|---|---
[8] | 360∘ video | 3DoF | ✔ | ✖ | WLR
[9] | 360∘ video | 3DoF | ✔ | ✖ | LR & KNN
[10] | 360∘ video | 3DoF | ✔ | ✔ | PanoSalNet
[11] | 360∘ video | 3DoF | ✔ | ✔ | GBVB
[12] | 360∘ video | 3DoF | ✔ | ✔ | Spherical CNN
[13] | AR | 6DoF | ✔ | ✖ | Motion model and prediction
[7] | VR | 6DoF | ✔ | ✖ | LSTM & MLP
Our | Point cloud video | 6DoF | ✔ | ✔ | LSTM & Saliency Detection

###  I-A Prior art and limitation

####  I-A1 Viewport Prediction

Efficient viewport prediction is crucial in addressing the challenges of
delivering high-bandwidth and ultra-low latency immersive videos such as 360
video and point cloud video. There are many machine learning based viewport
prediction methods. For example, linear regression (LR) is a common method
[14, 8] used for predicting the viewer’s future FoV by extrapolating motion
information from the user’s historical head trajectories. Various
modifications, such as the weighted LR [8] and K-nearest neighbor (KNN)-based
viewport prediction [9] methods, have been proposed to improve the accuracy of
LR models.

With the advancement of neural networks, these techniques have been employed
to predict the viewport. For instance, S. Yoon et al. propose a prediction
algorithm for 6DoF augmented reality (AR) to reduce the visual inconsistency
caused by the latency of the remote rendering, combining a consistent motion
model and deep prediction [13]. X. Hou et al. employ an long short term memory
(LSTM)-based body motion prediction model and a multi-layer perceptron
(MLP)-based part motion prediction model for a low-latency pre-rendered 6DoF
virtual reality (VR) system using an external tracking device [7]. However,
these methods only consider the temporal characteristics of user head motion
and do not account for the impact of video content on user attention, which
can lead to reduced prediction performance.

By incorporating video content features, viewport prediction can be
significantly improved. For example, C. Fan et al. propose a method for
predicting the user’s fixation by fusing 360-degree video content-related
features and head-mounted displays (HMDs) sensor-related features [15]. Video
content-related features include image saliency and motion maps, while sensor-
related features include HMDs orientation and position. Similarly, A. Nguyen
et al. present PanoSalNet, which explores panoramic saliency and head
orientation history to detect panoramic saliency using a self-built dataset
[10]. Aiming at immersive experience improvement, Y. Zhu et al. [11]
investigate mechanism of visual saliency and viewing behaviors and propose a
new framework, which adapts saliency prediction models for traditional 2D
images to panoramas, employing the graph convolution and feature alignment to
extract features and retain geometric information. J. Li et al. [16] propose a
spherical convolutional empowered FoV prediction method, which is a multi-
source prediction framework combining salient features extracted from
360-degree videos and limited FoV feedback information, by using a spherical
convolutional neural network.

However, research on viewport prediction of point cloud video considering both
video content and user trajectory is limited compared to 360 video or VR
video. In this study, we aim to bridge this research gap and maximize viewport
prediction performance by jointly considering point cloud video saliency and
the user’s head trajectory.

####  I-A2 Visual Saliency Detection

The detection of visual saliency is a prominent research topic in the field of
computer vision, with the goal of quickly locating the most attractive objects
or patterns in an image or video clip. This is vital to predicting the user’s
visual attention and viewport [17, 10, 18, 16, 6, 19].

Visual saliency detection can be classified into two categories: visual
saliency detection for images and visual saliency detection for videos. For
still images, deep learning techniques have improved the detection capability
of visual saliency, and various deep learning-based methods have been proposed
to address the challenge of visual saliency detection in still images. For
example, Q. Lai, et al. propose a weakly supervised visual saliency prediction
method on images using a joint loss module and a deep neural network module
[20]. M. Zhong, et al. develop a computational visual model based on support
vector regression to predict the user’s visual attention to a specific
category [21]. G. Ma, et al. revisit the problem of image salient object
detection from the semantic perspective, providing a novel network for
learning the relative semantic saliency degree for two input object proposals
[22]. V. Figueiredo et al. [23] utilize orthographic projections in 2D planes
and apply established saliency detection algorithms to create a 3D point cloud
saliency map. In contrast, X. Ding et al. [24] propose a novel approach for
detecting saliency in point clouds by jointly utilizing both local
distinctness and global rarity cues, which are supported by psychological
evidence.

The detection of saliency in videos entails a more complex challenge compared
to images. Apart from relying on spatial information, accurate saliency
prediction for videos requires the use of temporal and motion information from
consecutive video frames. Efficient use of this information remains crucial as
it significantly impacts the detection performance, and it has garnered
significant attention among researchers. Several works have been done in this
category, including Wang et al.’s novel attentive convolutional neural
network-long short term memory (CNN-LSTM) architecture, which explicitly
encodes static attention into dynamic saliency representation learning by
leveraging both static and dynamic fixation data [25]. M. Paul et al. propose
an effective and robust framework by combining spatial and motion saliency
scores using eye tracker data for video summarization[26]. Zhu et al. propose
a graph-based CNN model for 360-degree videos using graph convolution to learn
mutual information between spatial-temporal units of high-clarity vision [11].
C. Chen et al. propose a network for saliency and granularity mining that
learns temporally invariant features from video frame [27].

Traditional CNN networks are ineffective for processing unordered inputs, such
as the point cloud data. One alternative is to treat the points in 3D space as
a volumetric grid. This approach is only practical for sparse point cloud, and
CNNs are too computationally intensive for high-resolution point cloud. For
deeply learning 3D point cloud features, Z. Song et al. present a hybrid
semantic affinity learning method to capture and leverage the dependencies of
categories [28], which provides global priors and local affinities for point
features. And some researches propose to fuse features from different layers
or points to obtain more important features[29, 30, 31, 32]. Enlightened by
these investigations, we think it is necessary to capture both static and
dynamic saliency cues for point cloud video saliency detection with careful
design for point cloud characteristics.

This paper presents a new combined static and dynamic saliency detection
network that effectively handles the unordered nature of point cloud data in
videos. Our approach recognizes the importance of capturing both static and
dynamic saliency cues in such data, making it an ideal choice for point cloud
video saliency detection.

####  I-A3 Point Cloud Sampling

Point cloud sampling is often used to reduce the computational complexity of
point cloud video analysis. The current two major approaches of point cloud
sampling - heuristic and learning-based - have unignorable limitations.
Learning-based methods require large amounts of time and memory to train
complex neural networks, making them unsuitable for large-scale point cloud
videos. Heuristic methods, on the other hand, may not capture the dynamic
changes between consecutive point cloud frames.

The existing heuristic sampling methods can be categorized as follows:
Farthest point sampling (FPS) provides uniform results but has a high sampling
complexity[30]. Iterative density-based sampling (IDIS) calculates the density
of each point in the point cloud, and then takes the top K points with the
lowest density as sampling points[31]. Geometric shape-based sampling (GS)
calculates the curvature of each point in the point cloud, and then takes the
top K points with high curvature as sampling points. Voxel subsampling (VS)
divides the point cloud into 3D voxel rasters and uses the center of gravity
of all points within each voxel to represent other points[32]. Random
subsampling (RS) randomly chooses K points from the original point cloud.
Although RS is the most efficient and memory-friendly sampling method, it may
cause some loss of dynamic change information in point cloud videos.

Therefore, we propose a novel point cloud video sampling approach named
uniform random sampling (URS) which builds upon existing heuristic techniques
and offers improved performance. More detailed information on this approach is
available in Section III-A.

###  I-B Proposed solution and main contribution

This paper proposes a novel approach to address these challenges. Our approach
presents enhancements that improve the efficiency and accuracy of the saliency
detection model, thereby resulting in a more precise prediction of the user’s
viewport during point cloud video streaming.

Initially, a low-complexity tile-based URS unit is introduced, which
subdivides the tiles into equally sized smaller cubes in space. This results
in an even distribution of sampling points in each corner of the tile, thereby
ensuring almost consistent sampling results for adjacent frames. To enhance
our sampling efficiency, we randomly and indiscriminately sample the center
points within the cube. We also utilize the KNN method to sample neighboring
points around the centroids in order to preserve the spatial geometry
information. By improving the efficiency of our sampling process, the
corresponding sampling results for adjacent frames ascertain the feasibility
of the saliency detection module.

Our second proposal is a feature comparison method used to calculate local
motion intensity. The technique involves calculating global features of tiles
from adjacent frames that correspond with each other. This technique is
attained through the implementation of the temporal contrast (TC) layer. The
global features of corresponding tiles are compared with each other to obtain
the feature similarity scores. These scores are then transformed into motion
intensity using a temporal salient operator. In doing so, comparing global
features enables us to attain tile motion intensity. This comparison method
reduces computational efforts and eliminates network instability resulting
from the unordered nature of the point cloud.

We introduce an encoder to extract spatial salient features available in each
frame of the point cloud video. The encoder aggregates discrepancies between
centroids and a certain number of neighboring points. It is achieved through
the utilization of our local discrepancy catcher (LDC), which compiles color
and coordinate discrepancies in neighboring locations to extract the spatial
saliency features from point cloud videos. Factoring both brightly colored and
geometrically prominent areas’ effects in spatial dimensions on the user’s
viewport results in better spatial saliency feature extraction.

We refer to the approach of deeply adapting spatial-temporal saliency
detection and LSTM-based user trajectory estimation for multi-modal viewport
prediction as STVP. LSTM is utilized for training user head trajectory records
to predict the user’s head state (position and direction) in the subsequent
frame. The trajectory and saliency are deeply fused, fully considering the
impact of video content and the user’s own trajectory on the moving viewport,
allowing for more accurate viewport estimates.

In summary, the main contributions of this paper include:

  * •

the development of a novel and efficient sampling method (URS), which reduces
computational load while maintaining video features;

  * •

the improvement of the existing saliency detection method by incorporating
temporal and spatial information to capture dynamic, static geometric, and
color salient regions;

  * •

the deep fusion of saliency and trajectory to achieve more accurate viewport
prediction;

###  I-C Summaries of experimental results

We have conducted extensive simulations to evaluate the efficacy of our
proposed viewport prediction methods using four point cloud video sequences,
namely Soldier, Redandblack, Loot, and Longdress, from the 8i Voxelized Full
Bodies (8iVFB) dataset [33]. The evaluation indicates a viewport prediction
accuracy of over 90% for our proposed algorithm, demonstrating its superiority
over the existing schemes222The source codes and related data will be made
publicly available after paper acceptance.. In addition, our proposed URS
method plays a crucial role in reducing computational burden while
simultaneously preserving information. Concerning the neural network
component, our study shows that LDC can learn spatial saliency features
effectively through encoding local point-to-point discrepancy information.
Furthermore, the TC layer can efficiently acquire temporal saliency features
by comparing global variations.

The paper is organized as follows: we begin with Section II, where we present
the construction of a viewport prediction model for users of point cloud
videos. In Section III, we detail the principles of saliency models for point
cloud videos. Then, in Section IV, we explain the principles of LSTM-based
user head trajectory prediction and the feature fusion mechanism. In Section
V, we present the experimental setup and performance evaluation results.
Finally, we conclude the paper with a summary in Section VI.

##  II Overview of the Viewport Prediction

Figure 2: Overview of STVP. FC: Fully Connected Layer. LDC: Local Discrepancy
Catcher. URS: Uniform-Random Sampling. DP: Dropout Layer. TC: Temporal
contrast layer. MLP: Multi-Layer Perception. US: Up-Sampling. The $c$-th LDC
module takes an $B*N_{c}*D_{c}$ matrix as input that is from $B$ batches,
$N_{c}$ points with $D_{c}$-dim point feature. $F_{S}$ and $F_{T}$ are the
spatial and temporal salient features, respectively. $F_{L}$ is the predicted
trajectory of LSTM module, and is adaptively fused with $F_{S}$ and $F_{T}$ to
generate the final FoV estimation, $F_{E}$. The consecutive frames are first
reduced numbers by URS, meanwhile the spatial-temporal information of the
input data is retained as much as possible, then the spatial and temporal
saliency features are learned in the spatial and temporal saliency sub-models,
respectively. The user’s historical head state is fed into the LSTM network to
analyze and predict the user’s head state when viewing the next frame.

When drawn by conspicuous geometric shapes, vivid colors, or moving objects, a
user will cast his or her view to that specific region by slowly changing his
or her trajectory. Based on this intuition, we develop STVP, consisting of two
main models: a saliency detection model and a user head trajectory prediction
model, where the saliency detection model is subdivided into temporal saliency
and spatial saliency detection in order to identify spatial and temporal
salient regions, respectively. The detailed structure of STVP is shown in Fig.
2. The approximate principles of the saliency model and the trajectory
prediction model are described in the following.

Prerequisites: To ensure the performance of saliency detection, we have to
make the sampling results of adjacent frames similar in the time dimension to
preserve temporal information and make the sampling points aggregated in local
space to preserve spatial information. Furthermore, due to the large amount of
point cloud video data, an effective sampling method is crucial. Addressing
this issue, we propose a saliency detection model that incorporates a tile-
based URS unit.

Input: Assuming there are $T$ frames in a point cloud video, and $N^{t}_{in}$
points in frame $t$, representing the 3D scene. For each point $i$ in frame
$t$, there are coordinate attributes $p_{i,t}=\\{x_{i,t},y_{i,t},z_{i,t}\\}$,
and color attributes $a_{i,t}=\\{r_{i,t},g_{i,t},b_{i,t}\\}$.

Given consecutive 3D point cloud frames, we extract spatial features in the
current frame and capture temporal features between consecutive frames in
spatial-temporal saliency detection sub-model. In addition, the user’s head
state records $(L_{1},L_{2},...L_{l},...,L_{t-1})$ before time $t$ are input
to the trajectory prediction model to predict the head state $L_{t}$ of
current frame $t$, where the user’s head state $L_{l}$ at time $l$ is composed
of the head coordinates $(X_{l},Y_{l},Z_{l})$ and the Euler angles of the head
rotation $(\alpha_{l},\beta_{l},\gamma_{l})$, i.e.,
$L_{l}=\\{X_{l},Y_{l},Z_{l},\alpha_{l},\beta_{l},\gamma_{l}\\}$,
$L_{l}\in\mathbb{R}^{6}$.

Saliency detection: To reduce computation power consumption and improve
processing efficiency, original video frames are sampled through the URS unit
before being fed into the temporal and spatial saliency detection sub-models
in parallel. Then we propose an LDC module, which catches local color and
coordinate discrepancies, to learn spatial saliency features $F_{S}$ and a
tile-based TC layer to learn temporal saliency features $F_{T}$. The details
of saliency detection model are explained in Section III.

Head trajectory prediction and feature fusion: To predict the user’s head
state $L_{t}$ of frame $t$, we use LSTM to analyze the patterns between
historical trajectories. Then the predicted result $L_{t}$ is transformed into
LSTM-based feature $F_{L}$ by assigning different color attributes to the
points inside and outside of the viewport.

In order to achieve accurate viewport prediction, we incorporate the results
of saliency detection with those of trajectory prediction through an adaptive
fusion process. The adaptive fusion of spatial-temporal saliency features
$F_{S}$, $F_{T}$ and LSTM-based features $F_{L}$ employs an attention
mechanism to produce viewport prediction estimates $F_{E}$. Further
elaboration on the head trajectory prediction and feature fusion can be found
in Section IV.

##  III Spatial-temporal saliency detection model

In this section, we will introduce the URS unit, the spatial saliency
detection sub-model, and the temporal saliency detection sub-model.

###  III-A Uniform-Random Sampling

Reducing the number of points in a point cloud video through sampling can
yield numerous benefits, including decreased computation and communication
consumption. However, the sampling technique employed must be carefully chosen
to preserve important details inherent in the original point cloud,
particularly in terms of spatial and temporal information.

Using the proposed URS method, we select in an effective manner uniformly
spaced sampling points to preserve temporal information. Additionally, we
employ KNN to aggregate spatial information from neighboring points.

Figure 3: Detailed process of URS. The most distinct point cloud motion
between two mutually mapped tiles $t_{k,t-1}$ and $t_{k,t}$ is marked by blue
circles. The red points represent randomly selected central points within the
cube. The effect of spatial and temporal information retention is circled on
the output.

Assuming each frame is cut into $m$ tiles in the spatial dimension, and let
$t_{j,t}$ be the $j$-th tile in frame $t$. As seen in Fig.3, we establish a
mapping relationship between tiles, $t_{j,t}$ and $t_{j,t-1}$, because they
have similar spatial locations in their respective frames. This is a step in
the preparation of the TC layer to analyze changes in global information at
their respective locations. We ensure the preservation of temporal information
by having evenly distributed sampling points within adjacent tiles to ensure
that they have similar sampling results. To guarantee a uniform distribution
of the sampled points and the sampling efficiency, we further divide each tile
into $N_{c}$ smaller cubes, and randomly select the central points. Around the
central points, we search for $N_{n}$ neighbors as sampling points using KNN
to tighten the local spatial information together, where $N_{n}=N/N_{c}$, $N$
is the total number of sampling points in each tile. Finally, we can obtain
all the sampling data for $m$ tiles. Let $D_{j,t}$ be the sampled data of tile
$t_{j,t}$ in frame $t$. The implementation of TC layer requires that sampled
data with similar spatial locations have a close mapping relationship. The
degree of this closeness can be estimated by the proposed inter-frame mapping
strength, which measures the level of temporal information retention.

###  III-B Spatial Saliency Detection Sub-model

The sampling result $\\{D_{1,t},D_{2,t},...,D_{j,t},...,D_{m,t}\\}$ of frame
$t$ obtained by URS is passed to Fully Connected (FC) layer for feature
extraction, to the LDC module for spacial encoding and to the RS layer for
down-sampling.

Given the coordinate and color attributes of each point $i$ in $D_{j,t}$, we
firstly obtain the initial feature $f_{i,t}$ of point $i$ through the FC
layer, the process of feature extraction is simply expressed as:

$f_{i,t}=FC(p_{i,t}\oplus a_{i,t})$ (1)

where $\oplus$ denotes concatenation. Finally, the sampled data $D_{j,t}$ goes
through the FC layer to obtain the initial features $F_{j,t}$. Then we further
use LDC to fully integrate local coordinate and color discrepancies. The key
components of LDC are divided into three parts: Neighborhood Coding, Attention
Pooling and Dilated Residual Blocks.

Neighborhood Coding. This Neighborhood Coding unit explicitly embeds the
coordinate and color discrepancies between each point $i$ in $D_{j,t}$ and its
all neighboring points. In this way, the corresponding point features are
always aware of their relative spatial differences, facilitating the network’s
ability to learn spatial salient features more effectively.

For the coordinate and color attributes $p_{i,t}^{k}$, $a_{i,t}^{k}$ of
neighboring point $k$ around the point $i$, where
$a_{i,t}^{k}=\\{r_{i,t}^{k},g_{i,t}^{k},b_{i,t}^{k}\\}$, is the set of color
attributes of neighboring point $k$, we explicitly encode the relative
discrepancies as follows:

$d_{i,t}=0.299*r_{i,t}+0.587*g_{i,t}+0.114*b_{i,t}$ (2)
$\begin{split}de_{i,t}^{k}=&MLP[p_{i,t}\oplus
p_{i,t}^{k}\oplus(p_{i,t}-p_{i,t}^{k})\oplus\|p_{i,t}-p_{i,t}^{k}\|\\\ &\oplus
d_{i,t}\oplus
d_{i,t}^{k}\oplus(d_{i,t}-d_{i,t}^{k})\oplus\|d_{i,t}-d_{i,t}^{k}\|]\end{split}$
(3) $\hat{f_{i,t}^{k}}=re_{i,t}^{k}\oplus f_{i,t}^{k}$ (4)

where $0.299$, $0.587$ and $0.114$ are constants for converting RGB value to
grayscale $d_{i,t}$ based on the sensitivity of the human eye to red, green
and blue according to Luminance algorithm [34]. $\|{\cdot}\|$ denotes the
coordinate relative Euclidean distance.

The discrepancy encoding $de_{i,t}^{k}$ between point $i$ and neighboring
point $k$ is correspondingly connected with the initial feature $f_{i,t}^{k}$
to obtain the enhanced point feature $\hat{f_{i,t}^{k}}$. Then a set of
enhanced features of point $i$ encoded with all $K$ neighboring points
$A_{i,t}=\\{\hat{f_{i,t}^{1}},\hat{f_{i,t}^{2}},\hat{f_{i,t}^{3}}...\hat{f_{i,t}^{k}},..,\hat{f_{i,t}^{K}}\\}$
can be obtained through neighborhood coding.

Attention Pooling. This unit is used to aggregate each feature
$\hat{f_{i,t}^{k}}$ in the set of enhanced features $A_{i,t}$ as the spatial
discrepancy feature $\hat{f_{i,t}}$ of point $i$. Thus each point $i$ contains
the features of each neighboring point, which can compensate for the reduction
of both temporal and spatial information, which is caused by the replacement
of URS by RS as the downsampling technique in the following network training
process.

Given each enhanced feature $\hat{f_{i,t}^{k}}$, we first use shared function
to learn its attention score $S_{i,t}^{k}$. Then we use $Softmax$ function to
normalize attention score $S_{i,t}^{k}$, and weighted enhanced feature by the
normalized score to obtain the spatial discrepancy feature $\hat{f_{i,t}}$ as
follows:

$S_{i,t}^{k}=\gamma(\hat{f_{i,t}^{k}},W)$ (5)
$\hat{f_{i,t}}=\sum_{i=1}^{K}[\hat{f_{i,t}^{k}}*\sigma(S_{i,t}^{k})]$ (6)

where $W$ is the shared weight, $\sigma()$ is $Softmax$ function, and
$\gamma()$ means Shared MLP function.

Dilated Residual Blocks. The neighborhood coding and attention pooling can
spatially link each sampled point to $K$ neighbors. To expand the receptive
field of each point, we perform neighborhood coding and attention pooling
twice in dilated residual blocks to receive information from up to $K^{2}$
neighbors.

According to the encoding process of the LDC module above, the process of
passing the initial features $F_{j,t}$ of the sampled data $D_{j,t}$ through
the $c$-th LDC module and RS layer is simplified as:

$\hat{F_{j,t}^{c}}=R_{c}\\{L_{c}[\hat{{F_{j,t}^{c-1}}}],{\theta}_{u}]\\}$ (7)
$\hat{F_{j,t}^{0}}=L_{1}(F_{j,t})$ (8)

where $\hat{F_{j,t}^{c-1}}$ denotes the encoded features of $D_{j,t}$ through
$(c-1)$-th LDC and RS, which is passed to $c$-th LDC module as input, and the
input of $1$st LDC module, $\hat{F_{j,t}^{0}}$, is initialized according to
the Eq. 8. $L_{c}()$ represents the $c$-th LDC module, ${\theta}_{u}$ is the
learnable weight, $R_{c}$ represents the $c$-th RS layer.

The encoded features are decoded by applying MLP layers, Up-Sampling (US)
layers, and FC layers. Additionally, to prevent overfitting, a Dropout (DP)
layer is used. The output of this decoding process is a set of spatially
salient features, denoted by $F_{S}$.

###  III-C Temporal Saliency Detection Sub-model

Based on the mapping relationship of $D_{j,t}$ and $D_{j,t-1}$ extracted from
frame $t$ and $t-1$, we propose a TC module to capture temporal information by
comparing feature difference between them, as the feature difference can
reflect the degree of variation from $D_{j,t}$ to $D_{j,t-1}$.

The general structure of TC is shown in Fig. 4 and is explicitly expressed as
follows:

$Q_{t}^{c}=M(TC_{t}^{c})$ (9) $Q_{t-1}^{c}=M(TC_{t-1}^{c})$ (10)
$S_{sim}=\gamma(Q_{t}^{c}\oplus Q_{t-1}^{c})$ (11) $O_{s}=G(S_{sim})$ (12)
$C_{t}^{c}=O_{s}*TC_{t}^{c}$ (13)

where $TC_{t-1}^{c}$ and $TC_{t}^{c}$ denote the inputs of the $c$-th TC from
tile $D_{j,t}$ and tile $D_{j,t-1}$, respectively. The first TC module
initializes $TC_{t-1}^{1}$ and $TC_{t}^{1}$ as the first encoded features
$\hat{F_{j,t-1}^{1}}$ and $\hat{F_{j,t}^{1}}$. In $c$-th TC module,
$TC_{t-1}^{c}$ represents the $c$-th encoded feature, and $TC_{t}^{c}$ is the
outcome of the output from the previous $(c-1)$-th TC module, $C_{t}^{c-1}$,
after applying LDC and RS. $Q_{t}^{c}$ and $Q_{t-1}^{c}$ refer to the global
features of $TC_{t-1}^{c}$ and $TC_{t}^{c}$, respectively, computed through
the $Maxpooling$ function $M()$. $S_{sim}$ denotes the similarity score
calculated between the corresponding global features $Q_{t}^{c}$ and
$Q_{t-1}^{c}$. $O_{s}$ is the intensity of temporal saliency obtained by
applying the temporal saliency operator, $G()$, on the similarity score
$S_{sim}$. The operator is defined as $G(\cdot)=\frac{1}{1+\exp()}+1$. A lower
degree of variation and a smaller $O_{s}$ are expected when the similarity
score, $S_{sim}$, is higher between the global features $Q_{t}^{c}$ and
$Q_{t-1}^{c}$. The temporal change feature $C_{t}^{c}$ is obtained by
weighting the input feature $TC_{t}^{c}$, from frame $t$, with the temporal
saliency intensity and passing it on as input to the next TC module.

Figure 4: The structure of temporal contrast layer (inside the dashed box). We
choose two pairs of tiles mapped to each other in consecutive frame pairs to
demonstrate the principle of the TC layer, where the point clouds within one
pair of mapped tiles are relatively similar and the point clouds within the
other pair are clearly moving.

With multiple TC modules, we can efficiently capture the temporal information
between the mapping tiles, and then we obtain the temporal salient features
$F_{T}$ by decoding using MLP, US and FC layer.

##  IV User’s Head Trajectory Prediction Module and Feature Fusion Unit

This section presents the LSTM-based module for predicting head trajectory and
the feature fusion unit that utilizes attention mechanisms.

Because LSTM have the memory to learn the temporal dependence between
observations, we use it to predict the user’s head state at the next moment
$t$ based on the past user head position. The state update process of user’s
head state $L_{l}$ ($l$ is the time index) through the LSTM cell is shown in
the following equation:

$h_{l}=f(h_{l-1},L_{l})$ (14)

where $h_{l}$ is the hidden state of the LSTM cell structure at $l$-th
moments. $f()$ is the nonlinear activation function of the LSTM cell. Each
LSTM cell contains forget gate $f_{l}$, input gate $i_{l}$, output gate
$o_{l}$ and cell state $C_{l}$. The specific encoding process of LSTM cells is
as follows:

$f_{l}=\sigma(W_{f}[h_{l-1},L_{l}]+b_{f})$ (15)
$i_{l}=\sigma(W_{i}[h_{l-1},L_{l}]+b_{i})$ (16)
$o_{l}=\sigma(W_{o}[h_{l-1},L_{l}]+b_{o})$ (17)
$\tilde{C_{l}}=tanh(W_{c}[h_{l-1},L_{l}]+b_{c})$ (18)
$C_{l}=f_{l}*c_{l-1}+i_{l}*\tilde{C_{l}}$ (19) $h_{l}=o_{l}*tanh(C_{l})$ (20)

where $[h_{l-1},L_{l}]$ is the connection vector between the hidden layer
state $h_{l-1}$ and the input $L_{l}$. $W_{f}$, $W_{i}$, $W_{o}$, $W_{c}$ are
the learnable weights in the network and $b_{f},b_{i},b_{o},b_{c}$ are the
learnable biases in the network. $\sigma()$ denotes the $Sigmoid$ function.
Then hidden state $h_{l}$ and the next moment head state $L_{l+1}$ are passed
into next LSTM cell to learn the temporal dependence.

At the last LSTM cell, we get the predicted result of the user’s head state
$L_{t}$. The user’s predicted head state is projected onto the user’s
viewport, and the current frame $t$ is divided into two regions based on its
location relative to the viewport, as illustrated in Fig. 2. An RGB color
scheme is assigned to the points inside and outside the viewport, where points
within the viewport are colored white (RGB values of 255,255,255) while points
outside the field of view (FoV) are colored black (RGB values of 0,0,0). After
that, the sampled points $\\{D_{1,t},D_{2,t},...,D_{j,t},...,D_{m,t}\\}$ of
current frames $t$, is extracted as the LSTM-based features $F_{L}$.

Then, we need to integrate spatial saliency features $F_{S}$ and temporal
saliency features $F_{T}$ into saliency features $F_{ST}$, and then integrate
the saliency features $F_{ST}$ with LSTM-based features $F_{L}$ to get final
predicted estimates $F_{t}$.

Traditional feature aggregation methods such as $Maxpooling$ or
$Averagepooling$ lead to a significant loss of point information because they
aggregate features without considering their relationships.

To avoid this loss, we use the attention mechanism to adaptively fuse the
spatial-temporal saliency features of point cloud videos. The specific fusion
process is as follows:

Compute attention scores: Given the spatial saliency feature $F_{S}$ and the
temporal saliency feature $F_{T}$, we use the shared MLP followed by $Softmax$
function to calculate the attention score. It is formally defined as follows:

$\centering S_{S}=\sigma[\gamma(F_{S},W_{1})]\@add@centering$ (21)
$S_{T}=\sigma[\gamma(F_{T},W_{2})]$ (22)

where $W_{1},W_{2}$ is the learnable weight, $\sigma()$ is the $Softmax$
function, and $\gamma()$ is a proxy for shared MLP.

Feature weighting: The learned attention score can be regarded as a mask for
the automatic selection of salient features. The features are weighted to
output the saliency feature $F_{ST}$, which is represented as follows:

$F_{ST}=S_{S}\odot F_{S}+S_{T}\odot F_{T}$ (23)

where $\odot$ denotes the element-wise multiplication.

Second, we use the same fusion approach to fuse salient features with LSTM-
based features $F_{L}$:

$S_{ST}=\sigma[\gamma(F_{ST},W_{3})]$ (24) $S_{L}=\sigma[\gamma(F_{L},W_{4})]$
(25) $F_{E}=S_{ST}\odot F_{ST}+S_{L}\odot F_{L}$ (26)

where $W_{3},W_{4}$ are the learnable weight parameters, $\sigma$ is the
$Softmax$ function, $\gamma$ is a proxy for shared MLP and $\odot$ denotes the
element-wise multiplication. Finally, we get the viewport prediction feature
$F_{E}$, which divides the points into points in the viewport and points
outside the viewport.

##  V Experiment

In this section, we verify the effectiveness of our proposed prediction method
through extensive experiments.

###  V-A Experimental Setup

####  V-A1 Datasets

We choose four point cloud video sequences from the 8iVFB dataset [33], namely
"Soldier," "Redandblack," "Loot," and "Longdress," as shown in Fig. 5. These
sequences provide comprehensive coverage of the portrait characters’ entire
body and allow users to observe their movements. During the model training
phase, we allocate 900, 150, and 150 frames for training, validation, and
testing, respectively. For each video, we conduct experiments using 10, 20,
and 30 tiles ($m={10,20,30}$) to analyze their effect on viewport prediction.

Figure 5: Point cloud video sequences: Soldier, Redandblack, Loot, Longdress.
TABLE II: Performance of STVP with different numbers of tiles.

The number of tiles | MIoU | Tiling time
---|---|---
Point level MIoU | Tile level MIoU
10 | 82.09$\%$ | 88.10$\%$ | 300.0s
20 | 82.67$\%$ | 89.03$\%$ | 420.0s
30 | 83.19$\%$ | 89.90$\%$ | 930.0s

We recruit 40 student volunteers (25 males and 15 females) for our experiment
to view point cloud videos using HMDs. The experiment requires participants to
experience a 6DoF immersive experience and explore areas of interest in a way
they deem appropriate during the viewing process. The head trajectories of all
participants are meticulously recorded throughout the entire duration of the
video.

####  V-A2 Implementation

Our STVP model has been implemented using TensorFlow, and we plan to make the
experimental code publicly available on GitHub upon acceptance. The details of
the network architecture are explained in Appendix LABEL:appendix:network. All
experiments are conducted on our AI server equipped with four Nvidia Tesla
V100 GPUs.

####  V-A3 Metrics

We assess the overall performance of STVP using various metrics.

(1) Sampling performance: To evaluate the performance of our URS method, the
memory consumption, time consumption and inter-frame mapping intensity (IFMI)
are compared with other sampling competitors.

To check the IFMI of different sampling methods, we propose a metric named as
distance and color variation values (DaCVV). It calculates the coordinates and
the degree of color difference between the point $i$ and point $l$ from the
corresponding tiles, and is defined as:

$DaCVV=\frac{\|p_{i,t}-p_{l,t-1}\|}{D_{max}}+\frac{\|a_{i,t}-a_{l,t-1}\|}{C_{max}}$
(27)

where the $D_{max}$ and $C_{max}$ are the max coordination and color distance
of tile $t_{k,t}$, respectively.

Thus we can calculate the DaCVV between point $i$ from $t_{k,t}$ and point $l$
from $t_{k,t-1}$. If the DaCVV between two points is less than a certain
threshold, the two points are considered to be mapped to each other. We count
the ratio of all mapped points to all sampled points as IFMI. We try to
compare the IFMI of each sampling method for DaCVV thresholds of
$\\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9\\}$.

(a)

(b)

Figure 6: Time consumption, memory consumption of different sampling
approaches. TABLE III: Comparison of inter-frame mapping intensity for
different sampling methods

Approach DaCVV IFMI | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9
---|---|---|---|---|---|---|---|---|---
FPS | 0.29 | 0.49 | 0.67 | 0.80 | 0.87 | 0.93 | 0.97 | 0.99 | 1
IDIS | 0.27 | 0.48 | 0.66 | 0.79 | 0.85 | 0.92 | 0.98 | 0.99 | 1
VS | 0.30 | 0.51 | 0.71 | 0.86 | 0.92 | 0.95 | 0.98 | 0.99 | 1
GS | 0.31 | 0.49 | 0.64 | 0.77 | 0.85 | 0.91 | 0.97 | 0.99 | 1
RS | 0 | 0.02 | 0.18 | 0.37 | 0.55 | 0.88 | 0.97 | 0.99 | 1
URS | 0.28 | 0.49 | 0.71 | 0.78 | 0.89 | 0.93 | 0.96 | 0.99 | 1

(a) URS.

(b) RS.

(c) FPS.

(d) GS.

(e) VS.

(f) IDIS.

Figure 7: Results of different sampling methods for consecutive frames. TABLE IV: Performance comparison of different networks. | MIoU | Accuracy | Precision | Recall
---|---|---|---|---
| Point level MIoU | Tile level MIoU
PointNet++ | 52.22$\%$ | - | 69.44$\%$ | - | -
RandLANet | 61.66$\%$ | - | 76.28$\%$ | 67.68$\%$ | 87.49$\%$
BAAF-Net | 68.38$\%$ | - | 81.22$\%$ | 71.86$\%$ | 93.71$\%$
STVP-SD | 71.82$\%$ | - | 84.30$\%$ | 88.45$\%$ | 72.64$\%$
STVP | 82.09$\%$ | 88.10$\%$ | 90.17$\%$ | 83.72$\%$ | 96.80$\%$
RS++ | 58.19$\%$ | 69.86$\%$ | 74.06$\%$ | 69.39$\%$ | 80.84$\%$
IDIS++ | 59.40$\%$ | 70.90$\%$ | 79.12$\%$ | 78.12$\%$ | 79.53$\%$
LFA++ | 61.38$\%$ | 77.50$\%$ | 77.32$\%$ | 81.02$\%$ | 61.05$\%$
TSD– | 67.40$\%$ | 79.97$\%$ | 80.95$\%$ | 81.70$\%$ | 71.22$\%$

(a) The $2$th frame (front).

(b) The $2$th frame (back).

(c) The $2$th frame (side).

(d) The $307$th frame (front).

(e) The $307$th frame (back).

(f) The $307$th frame (side).

Figure 8: Comparison of predicted results and ground truth in points level.
The left, back and side columns are front, back, and side view, respectively.
In each subplot, the left side is the prediction result and the right side is
the ground truth. Figure 9: Comparison of predicted results and ground truth
of four sequences in tile-level. The top chart is the prediction chart, and
the bottom chart is the corresponding ground truth.

(2) Saliency detection: To assess the saliency detection performance of STVP,
we compare its predicted saliency output with the ground truth and other
competing models. The evaluation is carried out using four metrics: overall
accuracy (OA), precision, recall, and mean intersection over union (MIoU).

(3) Viewport prediction: We evaluate the viewport prediction performance of
STVP using the same metrics used in saliency detection evaluation. In
particular, we extend the MIoU metric into two categories for the viewport
prediction evaluation: point-level MIoU and tile-level MIoU. Point-level MIoU
measures the ratio of intersection and concatenation of point-level prediction
results with the ground truth, while tile-level MIoU measures the ratio of
intersection and concatenation of tile-level prediction results with the
ground truth.

####  V-A4 Baselines

We compare STVP with the competitors in terms of sampling and saliency
detection respectively.

(1) Baselines for sampling: The performance of URS is compared with five
commonly used competitors: FPS [30], GS, RS, IDIS [31], VS [32].

(2) Baselines for saliency detection : PointNet++ [30] RandLANet[35] and BAAF-
Net[36] have excellent performance in saliency detection, and we compare the
saliency detection performance of STVP saliency detection network (STVP-SD)
with them. Specifically, PointNet++ proposes a hierarchical ensemble
abstraction method that takes a frame of point cloud image as input, and
performs hierarchical downsampling and neighborhood group coding processes on
each video frame, and the extracted spatial features contain more and more
local information. Inspired by PointNet++, RandLANet improves the group
encoding layer by creating a local feature aggregator to encode neighborhood
coordination information, which can better learn spatial features. BAAF-Net
goes one step further and proposes a bilateral context module to extend local
coordinate encoding. The bilateral context module first fuses the local
semantic context information of the point cloud frames into the local
geometric context information and subsequently extracts local spatial features
by adapting the local geometric context information to the local semantic
context information.

####  V-A5 Ablation setup

During the experiments, we also conduct the following ablation studies for
STVP.

(1$\sim$2) RS++ and IDIS++: We replace URS of STVP with RS and IDIS,
respectively, while keeping the STVP saliency extraction modules. This is to
highlight the performance of the URS unit that can retain spatial-temporal
information by comparison.

(3) LFA++: To demonstrate the potential benefits of using our proposed LDC, we
replace LDC of STVP with LFA of RandLANet, which focuses only on the geometric
coordinate information of the neighborhood. This allows us to better
understand the performance of using the LDC, which leverages both the
neighboring coordinate information and color information to obtain more
accurate local spatial saliency, as explained in Sec. III.

(4) TSD–: We remove the temporal saliency detection (TSD–) and obtain the
viewport prediction results for this ablation experiment. By comparing the
prediction results of STVP and TSD–, we obtain the importance of the temporal
saliency model used to achieve the viewport prediction.

###  V-B Experimental Results

The effect of the number of tiles: We conjecture that the number of tiles has
some effect on the viewport prediction. We experiment with 10, 20, and 30
tiles, and the corresponding results are displayed in Table II. It can be seen
that as the number of tiles increases, the point-level MIoU increases
slightly, while the tile-level MIoU remains almost constant, but the
computation time to partition the whole point cloud video into tiles increases
significantly. Therefore, in the subsequent experiments, we set the number of
tiles to 12.

(a)  RS++

(b) IDIS++

(c) LFA++

(d) TSD–

(e) STVP

(f) Groud truth

(g)  RS++

(h) IDIS++

(i) LFA++

(j) TSD–

(k) STVP

(l) Groud truth

Figure 10: The predicted results for $311$th and $11$th frames are compared
with the ground truth at the point level and are shown on the images in the
first and second rows, respectively. The first four columns of each row
indicate the results of each frame in the ablation experiment (1), (2), (3),
and (4), the fifth image column denotes the predicted image of our complete
network STVP, and the sixth column shows the ground truth.

The efficiency of URS: To evaluate the efficiency of URS, we pick four levels
of point cloud size ($\sim 10^{3}$, $\sim 10^{4}$, $\sim 10^{5}$, and $\sim
10^{6}$ points per video frame). The point clouds are all downsampled to $1/4$
of the original number of points using FPS, VS, IDIS, GS, RS, and URS,
respectively. For each sampling method we compare their sampling time and
memory consumption. In addition, we show the results of the sampling
distribution for each sampling method separately.

The time and memory consumption of all the sampling methods are listed in Fig.
6. Experimental results indicate that when the number of sampling points
exceeds $10^{4}$, the sampling time and memory consumption for URS is second
only to RS, due to the fact that URS needs to slice the point cloud into
cubes. When the number of points is low, the time and memory consumed by each
sampling method, except RS, is similar. However, as the number of points
increases, the sampling efficiency of URS becomes more apparent.

As shown in Table. III, we also compare the IFMI of the six sampling methods
at the same DaCVV values. RS has the smallest IFMI overall. The IFMI of URS is
comparable to FPS and IDIS, slightly stronger than GS, and slightly weaker
than VS. In terms of time consumption, memory consumption, and IFMI, URS is
the best choice of the six sampling methods for point cloud video saliency
detection.

To observe the sampling effect visually, we take the corresponding tiles
$t_{k,t-1}$ and $t_{k,t}$ of consecutive frames $t-1$ and $t$ as sampling
inputs and set the number of sampling points $N_{num}=12288$ to get the result
shown as Fig. 7. The sampling result shows that URS can indeed retain temporal
information and spatial information. Although FPS, VS and IDIS retain the
temporal information between frames more completely, they consume more
sampling time and sampling memory than URS and neglect spacial local
information. RS and GS are even less preserving of temporal information than
URS.

The results of saliency detection: We compare the saliency prediction
performance of STVP-SD with PointNet++, BAFF, and RandLANet. The experimental
results are shown in Table IV, STVP-SD achieves on-par or better saliency
detection performance than state-of-the-art methods, with an MIoU of
71.82$\%$. This is because STVP-SD makes full use of the temporal information,
spatial coordinates and color information between point cloud video frames.

The results of viewport prediction: The Table IV presents the experimental
results, showing that STVP achieved high viewport prediction performance, with
a point-level MIoU of 82.09%. The accuracy at the tile level is even higher,
with an MIoU of 88.10%.

To visualize the prediction results, we extract one frame from two video
sequences to display the final viewport prediction results. The prediction
results are presented in two forms: a point-level prediction (Figure 8) and a
tile-level prediction (Figure 9). From the point-level prediction results, we
can see that although the STVP predictions are slightly different at the edge
of the viewport compared to the ground truth, their viewport ranges largely
overlap. However, their viewport edges do not resemble this defect, which is
perfectly solved by extending the predictions to the tile level. From the
tile-level prediction results, we can see that ground truth is almost the same
as the predicted result.

The result of ablation: We also compare the performance of all ablation
networks, and show the results in Table IV. The result indicates that
substituting the URS unit with either RS or IDIS leads to a reduction of
approximately 23% in the MIoU score of the prediction outcome. This shows that
URS plays a fundamental role in preserving the information of the original
point cloud video. Replacing or removing it has a large impact on the viewport
prediction. The LDC replacement also has large impact on prediction
performance. This is because both spatial color and geometric prominence
attract the user’s attention, and LDC increases color discrepancy coding
compared to LFA, which can effectively capture spatially color salient regions
and fully improve the spatial saliency prediction performance.

We also visualize the prediction results of the ablation experiments in Fig.
10. Since RS does not guarantee that the sampling results of adjacent video
frames match, RS++ leads to disconnected frames within the viewport in the
prediction results as shown in Fig. 10a and Fig. 10g. Due to the low degree of
local spatial information retention in IDIS, IDIS++ also leads to the above
situation, as shown in Fig. 10b and Fig. 10h. Meanwhile, since LFA ignores
spatially salient information, from Fig. 10c and Fig. 10i we can see that the
results of LFA++ have a salient part of the region less than the ground truth,
e.g., hair in “Redandblack” and gun in “Soldier”. The temporal detection sub-
module is created to capture the dynamic saliency region between frames.
Therefore, as shown in Fig. 10d and Fig. 10j, the result of TSD– lacks some
dynamic salient regions, e.g., hair in “Redandblack”, neck in “Soldier”,
compared with the ground truth.

##  VI Conclusion

This paper proposes a high-precision viewport prediction scheme for point
cloud video exploring viewport trajectory and video saliency information. In
particular, we propose a novel and efficient sampling method, and a new
saliency detection method incorporating temporal and spatial information to
capture dynamic, static geometric, and color salient regions. Our proposal
smartly fuses the saliency and trajectory information to achieve more accurate
viewport prediction. We have conducted extensive simulations to evaluate the
efficacy of our proposed viewport prediction methods using state-of-the-art
point cloud video sequences. The experimental results demonstrate the
superiority of the proposal over existing schemes.

## ACKNOWLEDGMENT

This work is supported in part by grants from the National Natural Science
Foundation of China (52077049), the Anhui Provincial Natural Science
Foundation (2008085UD04).

## References

  * [1] D. Mieloch, P. Garus, M. Milovanović, J. Jung, J. Y. Jeong, S. L. Ravi, and B. Salahieh, “Overview and efficiency of decoder-side depth estimation in mpeg immersive video,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 9, pp. 6360–6374, 2022.
  * [2] R. Mekuria, K. Blom, and P. Cesar, “Design, implementation, and evaluation of a point cloud codec for tele-immersive video,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 27, no. 4, pp. 828–842, 2016\.
  * [3] Z. Liu, Q. Li, X. Chen, C. Wu, S. Ishihara, J. Li, and Y. Ji, “Point cloud video streaming: Challenges and solutions,” _IEEE Network_ , vol. 35, no. 5, pp. 202–209, 2021.
  * [4] B. Han, Y. Liu, and F. Qian, “Vivo: Visibility-aware mobile volumetric video streaming,” in _Proceedings of the 26th Annual International Conference on Mobile Computing and Networking_ , ser. MobiCom ’20. New York, NY, USA: Association for Computing Machinery, 2020\.
  * [5] T. Stockhammer, “Dynamic adaptive streaming over http –: Standards and design principles,” in _Proceedings of the Second Annual ACM Conference on Multimedia Systems_ , ser. MMSys ’11. New York, NY, USA: Association for Computing Machinery, 2011, p. 133–144.
  * [6] J. Li, C. Zhang, Z. Liu, R. Hong, and H. Hu, “Optimal volumetric video streaming with hybrid saliency based tiling,” _IEEE Transactions on Multimedia_ , 2022.
  * [7] X. Hou and S. Dey, “Motion prediction and pre-rendering at the edge to enable ultra-low latency mobile 6dof experiences,” _IEEE Open Journal of the Communications Society_ , vol. 1, pp. 1674–1690, 2020.
  * [8] F. Qian, L. Ji, B. Han, and V. Gopalakrishnan, “Optimizing 360 video delivery over cellular networks,” in _Proceedings of the 6th Workshop on All Things Cellular - Operations, Applications and Challenges, AllThingsCellular@MobiCom 2016, New York City, New York, USA, October 3-7, 2016_ , M. K. Marina and U. C. Kozat, Eds. ACM, 2016, pp. 1–6.
  * [9] Y. Ban, L. Xie, Z. Xu, X. Zhang, Z. Guo, and Y. Wang, “Cub360: Exploiting cross-users behaviors for viewport prediction in 360 video adaptive streaming,” in _2018 IEEE International Conference on Multimedia and Expo (ICME)_ , 2018, pp. 1–6.
  * [10] A. Nguyen, Z. Yan, and K. Nahrstedt, “Your attention is unique: Detecting 360-degree video saliency in head-mounted display for head movement prediction,” in _Proceedings of the 26th ACM International Conference on Multimedia_ , ser. MM ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 1190–1198.
  * [11] Y. Zhu, G. Zhai, Y. Yang, H. Duan, X. Min, and X. Yang, “Viewing behavior supported visual saliency predictor for 360 degree videos,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 7, pp. 4188–4201, 2022.
  * [12] J. Li, L. Han, C. Zhang, Q. Li, and Z. Liu, “Spherical convolution empowered viewport prediction in 360 video multicast with limited fov feedback,” _ACM Trans. Multimedia Comput. Commun. Appl._ , vol. 19, no. 1, jan 2023.
  * [13] S. Yoon, H. j. Lim, J. H. Kim, H.-S. Lee, Y.-T. Kim, and S. Sull, “Deep 6-dof head motion prediction for latency in lightweight augmented reality glasses,” in _2022 IEEE International Conference on Consumer Electronics (ICCE)_ , 2022, pp. 1–6.
  * [14] L. Xie, Z. Xu, Y. Ban, X. Zhang, and Z. Guo, “360probdash: Improving qoe of 360 video streaming using tile-based http adaptive streaming,” in _Proceedings of the 25th ACM International Conference on Multimedia_ , ser. MM ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 315–323.
  * [15] C.-L. Fan, J. Lee, W.-C. Lo, C.-Y. Huang, K.-T. Chen, and C.-H. Hsu, “Fixation prediction for 360° video streaming in head-mounted virtual reality,” in _Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video_ , ser. NOSSDAV’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 67–72.
  * [16] J. Li, L. Han, C. Zhang, Q. Li, and Z. Liu, “Spherical convolution empowered viewport prediction in 360 video multicast with limited fov feedback,” _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , 2022.
  * [17] L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual attention for rapid scene analysis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 20, no. 11, pp. 1254–1259, 1998.
  * [18] T. Judd, K. Ehinger, F. Durand, and A. Torralba, “Learning to predict where humans look,” in _2009 IEEE 12th International Conference on Computer Vision_ , 2009, pp. 2106–2113.
  * [19] R. Cong, J. Lei, H. Fu, M.-M. Cheng, W. Lin, and Q. Huang, “Review of visual saliency detection with comprehensive information,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 10, pp. 2941–2959, 2019.
  * [20] Q. Lai, T. Zhou, S. Khan, H. Sun, J. Shen, and L. Shao, “Weakly supervised visual saliency prediction,” _IEEE Transactions on Image Processing_ , vol. 31, pp. 3111–3124, 2022.
  * [21] M. Zhong and L. Shen, “Learning to predict what humans look at: Computational visual attention model for specific category,” in _2022 International Symposium on Electrical, Electronics and Information Engineering (ISEEIE)_ , 2022, pp. 1–7.
  * [22] G. Ma, S. Li, C. Chen, A. Hao, and H. Qin, “Rethinking image salient object detection: Object-level semantic saliency reranking first, pixelwise saliency refinement later,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 4238–4252, 2021.
  * [23] V. F. Figueiredo, G. L. Sandri, R. L. de Queiroz, and P. A. Chou, “Saliency maps for point clouds,” in _2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP)_ , 2020, pp. 1–5.
  * [24] X. Ding, W. Lin, Z. Chen, and X. Zhang, “Point cloud saliency detection by local and global feature fusion,” _IEEE Transactions on Image Processing_ , vol. 28, no. 11, pp. 5379–5393, 2019.
  * [25] W. Wang, J. Shen, F. Guo, M.-M. Cheng, and A. Borji, “Revisiting video saliency: A large-scale benchmark and a new model,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4894–4903.
  * [26] M. Paul and M. Musfequs Salehin, “Spatial and motion saliency prediction method using eye tracker data for video summarization,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 6, pp. 1856–1867, 2019.
  * [27] C. Chen, M. Ye, M. Qi, J. Wu, Y. Liu, and J. Jiang, “Saliency and granularity: Discovering temporal coherence for video-based person re-identification,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 9, pp. 6100–6112, 2022.
  * [28] Z. Song, L. Zhao, and J. Zhou, “Learning hybrid semantic affinity for point cloud segmentation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 7, pp. 4599–4612, 2022.
  * [29] L. Zhao and W. Tao, “Jsnet++: Dynamic filters and pointwise correlation for 3d point cloud instance and semantic segmentation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 33, no. 4, pp. 1854–1867, 2023\.
  * [30] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 5105–5114.
  * [31] F. Groh, P. Wieschollek, and H. P. A. Lensch, _Flex-Convolution: Million-Scale Point-Cloud Learning Beyond Grid-Worlds_. Computer Vision – ACCV 2018, 2019.
  * [32] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4490–4499.
  * [33] E. d’Eon, B. Harrison, T. Myers, and P. Chou, “8i voxelized full bodies—a voxelized point cloud dataset, document iso,” _IEC JTC1/SC29 Joint WG11/WG1 (MPEG/JPEG), WG11M40059/WG1M74006, Geneva_ , 2017.
  * [34] W. K. Pratt, _Digital Image Processing (2nd Ed)_. Digital image processing (2nd ed.), 1987.
  * [35] Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, N. Trigoni, and A. Markham, “Randla-net: Efficient semantic segmentation of large-scale point clouds,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 11 105–11 114.
  * [36] S. Qiu, S. Anwar, and N. Barnes, “Semantic segmentation for real point cloud scenes via bilateral augmentation and adaptive fusion,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 1757–1767.