# Novel View Video Prediction Using a Dual Representation

Sarah Shiraz, Krishna Regmi, Shruti Vyas, Yogesh S. Rawat, Mubarak Shah
Center for Research in Computer Vision (CRCV), University of Central Florida

###### Abstract

We address the problem of novel view video prediction; given a set of input
video clips from a single/multiple views, our network is able to predict the
video from a novel view. The proposed approach does not require any priors and
is able to predict the video from wider angular distances, upto 45 degree, as
compared to the recent studies predicting small variations in viewpoint.
Moreover, our method relies only on RGB frames to learn a dual representation
which is used to generate the video from a novel viewpoint. The dual
representation encompasses a view-dependent and a global representation which
incorporates complementary details to enable novel view video prediction. We
demonstrate the effectiveness of our framework on two real world datasets:
NTU-RGB+D and CMU Panoptic. A comparison with the State-of-the-art novel view
video prediction methods shows an improvement of 26.1% in SSIM, 13.6% in PSNR,
and 60% in FVD scores without using explicit priors from target views.

Fig. 1: Novel-view Video Prediction. Given a set of video clips from
single/multiple input viewpoints, the proposed framework can generate a video
clip from a novel viewpoint. In the above example, given two input views, our
network is able to generate videos from multiple viewpoints, video from five
unseen viewpoints are shown here.

Index Terms—  Novel View Synthesis, Dual Representation, Video Prediction

##  1 Introduction

Novel view prediction finds interesting applications in robotics, graphics and
augmented reality [1, 2, 3]. Given one or more input views, the task is to
predict the seen observations from a novel unseen view. Ample research on
novel view prediction has been done for image based methods [4, 5], however
novel view prediction for videos is much more challenging and relatively
unexplored. The image-based approaches, when directly applied to videos,
result in temporally incoherent videos as they do not model temporal dynamics.

Fig. 2: Overall architecture of the proposed network. The Video Encoder
$V_{E}$ takes multiple input clips $c_{i}$, $i$ $\in$ 1,2,.., N from different
viewpoints, $v_{i}$ to learn the encoded features, $e_{i}$. The Global
Representation Block, $V_{G}$ uses the absolute view parameters of the input,
$v_{i}$. The features $e_{i}$ are aggregated with view embeddings from view
embedder, VE using a 3D convLSTM block. The Query Network, Q retrieves
features, $r^{\prime}_{g}$ according to the query view $v_{q}$. The view
dependent block, $V_{T}$ on the other hand uses relative view information,
using both partial input view $v^{p}_{i}$ and partial query view $v^{p}_{q}$,
which adds fine view-dependent details to the video. The dual representation,
$r$ thus obtained by aggregating the retrieved global representation,
$r^{\prime}_{g}$ and view dependent representation, $r_{t}$ is used to
synthesizes the query view video, $c_{q}^{{}^{\prime}}$.

In this paper, we address the problem of novel view video prediction without
using explicit estimation of 3D structures and geometry based transformations.
Most of the video prediction works [6, 7, 8] in the literature are mainly
limited to single viewpoint with the aim of improved dynamics and sharper
frames. Only recently, the authors in [9] tackled the task of novel view video
prediction by proposing the use of depth maps and human pose information of
the novel view as priors. However, the availability of these priors from the
query viewpoint is a big assumption which may not always be valid and requires
extra computation.

Considering this limitation, we propose a novel learning based method which
generates human action videos from a novel view without any priors and of
better video quality. Proposed method takes video clips from multiple
viewpoints and utilizes a two-stream approach for learning the dual
representations: global and view-dependent. The global representation focuses
on the general features, such as scene structure, whereas the view-dependent
representation focuses on finer details specific to the query view. We
evaluate our approach with extensive ablations and report state-of-the-art
results on two real world datasets, CMU Panoptic [10] and NTU-RGB+D [11].
Contributions:  (1) We propose a novel-view video prediction framework which
can generate videos from unseen query viewpoints. (2) The proposed network
integrates both global as well as view-dependent representation for effective
novel-view video prediction. (3) We provide extensive evaluation of our
approach on two real-world datasets, CMU Panoptic [10] and NTU-RGB+D [11],
achieving significant improvement over the existing methods without using any
prior information about the query views. We report 26.1% improvement in SSIM,
13.6% improvement in PSNR and 60% improvement in FVD scores over state-of-the-
art viewLSTM [9] on CMU-Panoptic Dataset [10].

##  2 Related Work

Early works in multi-view prediction by Cooke et al. [12] and [13] create a
novel view video by exploiting the scene geometry information from multiple
sources. Similar works by Domański et al. [14] discuss novel view video
prediction from multiview video sources. [15] estimates the 3D mesh of the
person present in the query view and transfer the textures from the 2D images
to the mesh. Recently, Lakhal [9] guided their network to synthesize novel
view videos by feeding skeleton and depth priors from the query view. Such
priors are strong cues that influence the quality of synthesized videos.
Learning to synthesize the novel view video without any prior information from
the query view is very challenging and our work attempts to solve this
problem. We do so by learning holistic as well as view-dependent
representations from multiple input video clips for effective novel-view video
prediction.

##  3 Method

Given a set of video clips $c_{i}$, $i$ $\in$ 1,2,.., N from a single or
multiple input views, our goal is to synthesize a video clip from a novel
viewpoint. We propose a deep neural network which learns a dual internal
representation, $r$ by assimilating global and view dependent information
required for a novel view generation. Global information is learned in the
form of a global representation $r_{g}$ which encodes the scene as a whole
whereas, view-dependent representation $r_{t}$ helps recover the finer details
specific to the query view. As shown in Figure 2, input to the proposed
network is a set of $N$ video clips from different view points along with the
respective view information, and the output is a video clip from the query
view. The view information is in the form of a vector, $v$ of dimension
$N\times d$ where d is the number of view parameters for a viewpoint. The
$c_{q}^{{}^{\prime}}$ is the output clip from the query view. The network can
be divided into four parts: a Video Encoder, $V_{E}$, Representation Learning
Blocks $V_{G}$ & $V_{T}$, and a Video Decoder $V_{D}$. Each part is explained
as follow:

###  3.1 Video Encoder $V_{E}$

Video Encoder is a multi-view encoder that takes $N$ video clips separately
from multiple views and learns features of each video clip in parallel. We use
a 3D CNN based encoder network which is a modified version of an Inflated 3D
convNet [16] architecture. The encoder shares weights for all the input video
clips. The input to the Encoder is $c_{i}$ where $i\in{[1,2,..,N]}$ and the
output is the encoded feature $e_{i}$.

###  3.2 Global Representation Block $V_{G}$

The goal of this block is to transform the video features $e_{i}$ in a way
such that we have a global representation representing the dynamics of the
overall scene. To achieve this, video encoding $e_{i}$ are first concatenated
with the respective view information, resulting in $e_{i}^{g}$, and then all
features are aggregated using a recurrent network. The viewpoint, $v_{i}$, is
a $d$-dimensional vector that contains viewpoint information i.e. camera
location coordinates, viewpoint angle, principal point offset, focal length
etc. It is passed through a View Embedder which performs linear interpolation
on it before combining with the video features $e_{i}$. Next,
$e_{i}^{g},i\in{[1,2,..,N]}$ are aggregated together using a recurrent network
consisting of a bidirectional convLSTM followed by a unidirectional convLSTM
to learn the global representation $r_{g}$.

###  3.3 View-Dependent Block $V_{T}$

The goal of the View-Dependent Block $V_{T}$ is to transform the features of
video clips from multiple input viewpoints to the features of a query
viewpoint. To do so, it learns the features specific to the query view and
help in adding finer details to the synthesized video. To achieve this, we use
relative view information i.e. partial view vector $v^{p}_{i}$, comprising of
only viewpoint angles and camera location of the input views $v^{p}_{i}$ and
the query view $v^{p}_{q}$. The partial viewpoint facilitates the model to
focus more on the input view embeddings which are the most similar to query
view. Specifically, we apply two fully-connected layers on $v^{p}_{i}$ and
$v^{p}_{q}$ to learn the relative view features $e_{i}^{t}$, as shown in
Figure 2. Similar to aggregation of embeddings in $V_{G}$ we use a 3D
convLSTM, in particular a bidirectional convLSTM followed by a unidirectional
convLSTM to learn the view dependent representation $r_{t}$. However, the
weights of 3D convLSTM block are not shared with $V_{G}$.

###  3.4 Video Decoder $V_{D}$

The goal of the Video Decoder, $V_{D}$, is to integrate the view dependent
representation with the global representation to synthesize a video clip from
the query view $v_{q}$. To achieve this, it uses a convoluional-based query
network, $Q$, to extract the query view specific features $r^{\prime}_{g}$
from the global representation $r_{g}$ and then combines it with view
dependent representation $r_{t}$ to get the combined representation $r$.
Finally, we use another 3D convNet which uses $r$ and transforms the features
to the image space resulting in a synthesized video clip
$c_{q}^{{}^{\prime}}$.

###  3.5 Loss Function

We use a combination of reconstruction, adversarial and perceptual loss as
shown in Equation 1 to train our network.

$\mathcal{L}=\lambda_{r}{L}_{r}+\lambda_{p}{L}_{p}+\lambda_{adv}{L}_{adv}$ (1)

where $L_{r}$, $L_{p}$ and $L_{adv}$ are reconstruction, perceptual and
adversarial losses respectively. $L_{p}$ compares the features extracted from
conv5_2 layer of VGG-19 [17] for ground truth and predicted video. We
empirically set $\lambda_{p}$ = $\lambda_{adv}$ = 0.01 and $\lambda_{r}$ = 1.
To improve the realism of the synthesized video clip, we employ a
discriminator which (a) distinguishes real video clips from the synthetically
generated ones, and (b) it recognizes realistic motion between frames. The
loss term is implemented by the minimax loss [18].

Fig. 3: Qualitative Results on CMU Panoptic dataset; showing comparison with
state-of-the-art method viewLSTM [9] with single-view input setup. We show two
samples and the resolution is $56\times 56.$

##  4 Experiments

###  4.1 Experimental Setup

Datasets  We evaluated our network on following two datasets. (1) CMU Panoptic
Dataset [10] is a large-scale multi-activity real-world dataset with a total
of 521 viewpoints. The cameras are installed in 20 hexagonal panels, with 24
cameras in each panel. The provided viewpoint information vector, $v_{i}$,
consists of camera location coordinates (3D), principal point offset (2D),
focal length (2D), distortion coefficients (5D), horizontal-pan (1D) and
vertical-pan (1D), therefore $d=14$. (2) NTU-RGB+D [11] dataset contains 60
action classes recorded from 3 viewpoints: at horizontal angles of -45
degrees, 0 degree, 45 degrees. We use the cross subject evaluation split of
40,320 and 16,560 clips for training and testing respectively. The viewpoint
information vector, $v_{i}$, for this dataset, consists of camera height (1D),
camera distance (1D), horizontal-pan (1D) and vertical-pan (1D), viewpoint
angle (1D), therefore $d=5$.

###  4.2 Novel View Video Prediction: CMU Panoptic

CMU panoptic [10] has a large number of views which makes it a lucrative
dataset for cross-view video prediction. Considering this, we experiment on a
large-scale setting, using a maximum of 72 views for training and testing. We
examine the impact of number of training videos on the quality of synthesized
videos and compare with SOTA.

Large-scale Multiview Experiment. In this setup, we used 72 camera viewpoints.
Specifically, we select 3 panels (Number 4, 5 and 17) and use all 24 views
from each panel, totalling 72 views. We use 56 viewpoints for training, and 16
viewpoints for the testing with a split of 6,244 training and 1,132 test
samples. Training with so many viewpoints can be challenging due to
computation cost. So for each iteration we randomly selected, six views from
one of the panels, five input views and a query view. Testing followed a
similar setup. For each panel, we fix the input viewpoints from view 1 to view
5 $i.e.$ ($v_{1}$,$v_{2}$,$v_{3}$,$v_{4}$,$v_{5}$) and test on the remaining
19 viewpoints $i.e.$ the query view points are from $v_{6}$ to $v_{24}$. SSIM
[19] and PSNR for frame quality evaluation and Frechet Video Distance (FVD)
[20] to measure the video quality.

Fig. 4: Qualitative results on NTU-RGB+D Dataset [11] with multi-input view setup, resolution 56 $\times$ 56\. The first two rows show clips from input views $v_{1}$ and $v_{2}$. The synthesized output from the query view $v_{q}$ is shown in the third row followed by the corresponding ground truth frames on the last row.  Table 1: Quantitative results for Large-scale Multiview experiment (72 views), averaged over 19 query views of each of the three panels of CMU Panoptic at $56\times 56$ resolution. Views ($v_{1}$,$v_{2}$,$v_{3}$,$v_{4}$,$v_{5}$) are input views and views $v_{6}$ to $v_{24}$ are query (output) views.  Panels | Metrics
---|---
| SSIM($\uparrow$) | PSNR($\uparrow$) | FVD($\downarrow$)
4 | 0.837 | 25.15 | 11.52
5 | 0.848 | 24.9 | 13.42
17 | 0.845 | 25.46 | 13.31

Table 4 shows that the proposed method is successful at synthesizing high
quality video clips from multiple query views. Owing to the weight sharing
nature of the proposed approach, we are able to change the number of testing
input clips irrespective of training. Figure Novel View Video Prediction Using
a Dual Representation shows qualitative results with two fixed input views and
for different query views. The predicted video frames capture the motion and
viewpoints correctly.

Comparison with state-of-the-art

We compare our results with a video based method viewLSTM [9] which uses
skeleton prior from the query view and a frame-based method SynSin [21], both
trained from scratch. For training, we use three viewpoints, however for a
fair comparison with baselines (use one input view), we tested our network
employing a single input view setup. Table 6 shows comparison between these
methods in terms of SSIM, PSNR and FVD scores. As shown in Table 6, we report
26.1%, 13.6% and 60% improvement in SSIM, PSNR and FVD scores respectively
over viewLSTM [9], for clips length T=16. Similar improvements are observed
when compared with SynSin [21] quantitatively. For visual comparison, some
video frames are shown in Figure 3. Our model is able to get better quality
(sharpness), colors, motion as well as background details. Although, there is
ample scope of improvement as compared to ground truth, finer details are much
more formed with the proposed approach.

Table 2: Quantitative comparison of our method with state-of-the-art on CMU Panoptic dataset at $56\times 56$ resolution for different lengths of synthesized video frames. | Metrics
---|---
Methods | SSIM($\uparrow$) | PSNR($\uparrow$) | FVD($\downarrow$)
| T=16 | T=24 | T=32 | T=48 | T=16 | T=16
[9] | 0.681 | 0.667 | 0.588 | 0.488 | 22.96 | 22.87
[21] | 0.744 | 0.752 | 0.748 | 0.735 | 21.53 | 17.62
Ours | 0.859 | 0.834 | 0.809 | 0.782 | 26.08 | 9.15
Table 3: SSIM, PSNR and FVD scores for Network Ablations on NTU-RGB+D Dataset [11] using View Dependent (VD) stream, Global Representation (GR) stream and the Full model (Full) on $56\times 56$ resolution with multi-view input setup. The table shows the average scores for all input-output view combinations.  Networks | Metrics
---|---
| SSIM($\uparrow$) | PSNR($\uparrow$) | FVD($\downarrow$)
VD (tested) | 0.335 | 12.58 | 18.93
GR (tested) | 0.639 | 17.45 | 15.68
VD (trained) | 0.563 | 16.22 | 16.77
GR (trained) | 0.792 | 20.05 | 13.45
Proposed | 0.817 | 20.77 | 12.31

###  4.3 Novel View Video Prediction: NTU-RGB+D

Since the NTU-RGB+D dataset [11] consists of three camera viewpoints, our
experiment uses two views as input and the third view as query. Qualitative
results along with the ground truth are visualized in Figure 4. Consistent
with SSIM and PSNR values Table 7 (Proposed), the network is able to
synthesize the persons in the frames as well as the background details of the
query viewpoint correctly. We show frames at three different time steps and
observe that the synthesized frames are able to capture the motion similar to
the ground truth which is also supported by low FVD scores. As ablations
studies, we evaluate the role of global and view-dependent blocks separately.
First we test the fully trained network by using one block at a time and call
them VD(tested) and GR(tested). Secondly, we train both blocks separately and
call VD(trained) and GR(trained). The results are shown in Table 7 confirming
the effect of the two blocks together produces better output.

##  5 Conclusion

We propose a novel approach for unseen view video prediction via a dual
representation. The dual representation comprises of global and view dependent
representations which enables novel view video prediction. The proposed
approach takes single/multiple clips as input for video prediction from the
novel view without any priors. Using only RGB videos, our evaluation on two
real world datasets shows an improvement over state-of-the-art.

## References

  * [1] Steven M Seitz and Charles R Dyer,  “Physically-valid view synthesis by image interpolation,”  in Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV’95). IEEE, 1995, pp. 18–25.
  * [2] Ismael Daribo and Béatrice Pesquet-Popescu,  “Depth-aided image inpainting for novel view synthesis,”  in 2010 IEEE International workshop on multimedia signal processing. IEEE, 2010, pp. 167–170.
  * [3] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros,  “View synthesis by appearance flow,”  in European conference on computer vision. Springer, 2016, pp. 286–301.
  * [4] Kyle Olszewski, Sergey Tulyakov, Oliver Woodford, Hao Li, and Linjie Luo,  “Transformable bottleneck networks,”  in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 7648–7657.
  * [5] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein,  “Scene representation networks: Continuous 3d-structure-aware neural scene representations,”  in Advances in Neural Information Processing Systems, 2019, pp. 1119–1130.
  * [6] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba,  “Generating videos with scene dynamics,”  in Advances In Neural Information Processing Systems, 2016, pp. 613–621.
  * [7] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz,  “Mocogan: Decomposing motion and content for video generation,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1526–1535.
  * [8] Simone Palazzo, Concetto Spampinato, P D’Oro, Daniela Giordano, and Mubarak Shah,  “Generating synthetic video sequences by explicitly modeling object motion,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 0–0.
  * [9] Mohamed Ilyes Lakhal, Oswald Lanz, and Andrea Cavallaro,  “View-lstm: Novel-view video synthesis through view decomposition,”  in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 7577–7587.
  * [10] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh,  “Panoptic studio: A massively multiview system for social motion capture,”  in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3334–3342.
  * [11] Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang,  “Ntu rgb+ d: A large scale dataset for 3d human activity analysis,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1010–1019.
  * [12] Eddie Cooke, Peter Kauff, and Thomas Sikora,  “Multi-view synthesis: A novel view creation approach for free viewpoint video,”  Signal Processing: Image Communication, vol. 21, no. 6, pp. 476–492, 2006.
  * [13] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh,  “Neural volumes: Learning dynamic renderable volumes from images,”  arXiv preprint arXiv:1906.07751, 2019.
  * [14] M Domański, M Gotfryd, and K Wegner,  “View synthesis for multiview video transmission,”  .
  * [15] Mohamed Ilyes Lakhal, Davide Boscaini, Fabio Poiesi, Oswald Lanz, and Andrea Cavallaro,  “Novel-view human action synthesis,”  arXiv preprint arXiv:2007.02808, 2020.
  * [16] Joao Carreira and Andrew Zisserman,  “Quo vadis, action recognition? a new model and the kinetics dataset,”  in proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6299–6308.
  * [17] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial nets,”  in Advances in neural information processing systems, 2014, pp. 2672–2680.
  * [19] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli,  “Image quality assessment: from error visibility to structural similarity,”  IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  * [20] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly,  “Towards accurate generative models of video: A new metric & challenges,”  arXiv preprint arXiv:1812.01717, 2018.
  * [21] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson,  “Synsin: End-to-end view synthesis from a single image,”  in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [22] J Deng, W Dong, R Socher, LJ Li, K Li, and L Imagenet Fei-Fei,  “A large-scale hierarchical image database. computer vision and pattern recognition, 2009. cvpr 2009,”  in IEEE Conference on, pp. 248–255.
  * [23] Mohamed Ilyes Lakhal, Oswald Lanz, and Andrea Cavallaro,  “Pose guided human image synthesis by view disentanglement and enhanced weighting loss,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 0–0.
  * [24] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool,  “Pose guided person image generation,”  in Advances in Neural Information Processing Systems, 2017, pp. 406–416.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,” 2015.

Novel View Video Prediction Using a Dual Representation
Supplementary Material

##  6 Additional Network Details

Fig. 5: Qualitative results on CMU Panoptic dataset for 72 views experiment
with five input views showing three different samples. The resolution is
$56\times 56$.  Fig. 6: SSIM and PSNR scores vs the number of input views.
Higher number of views help better prediction quality. The plots show scores
for 5 randomly chosen query views.

Video encoder is a modified I3D network[16] where the modifications are: (i)
remove max pooling layer after block 2, (ii) for max pooling in block 3 & 4,
stride is set to 2$\times$1$\times$1, (iii) Add 1$\times$1$\times$1
convolutional layer before average pooling, to reduce the number of feature
maps to 256. Finally, $V_{E}$ is initialized with I3D weights pre-trained on
ImageNet[22].

View Embedder, VE is a trilinear interpolation layer which takes the view
information ($v_{i}$ or $v_{q}$) of size (1$\times$1$\times d$), where, $d$ is
the number of view dimensions.and interpolates the spatial size to match with
that of the encoder features in case of the View Dependent Block and the
Global Representation Block ( i.e. $v_{i}$ and $v_{i}-v_{q}$) and convLSTM
features in the case of Video Decoder ($v_{q}$) respectively, as we can see
from the Network Diagram in the manuscript.

View Dependent Block, $V_{T}$ consists of two fully-connected layers of size
$d^{p}\times d^{p}$ and $2d^{p}\times d^{p}$ where $d^{p}$ is the number of
view dimensions in $v_{i}^{p}$ or $v_{q}^{p}$. Next, it uses a bidirectional
convLSTM with 64 3 $\times$ 3 $\times$ 3 kernels; followed by a unidirectional
convLSTM with 128 3 $\times$ 3 $\times$ 3 filters in the convolutions. The
output features from the Video Encoder, $e_{i}^{{}^{\prime}}$ and the relative
view embeddings from the View Embedder are concatenated channelwise and passed
as input to this block.

Global Representation Block also consists of a bidirectional convLSTM followed
by a unidirectional convLSTM. Here, the first convLSTM consists of 128 3
$\times$ 3$\times$ 3 filters whereas the second one uses 256 3 $\times$
3$\times$ 3 filters.

Query Network is a single-layer convolutional network with 256 filters and
filter size of 5 $\times$ 5 $\times$ 5.

Video Decoder has 8 convolutional layers (combined with upsampling layers)
with kernel sizes 5$\times$ 5 $\times$ 5 and 256, 256, 256, 256, 256, 128, 64,
3 filters respectively. The second last layer uses kernel size 3 $\times$ 3
$\times$ 3\. The final layer is the Tanh activation.

Discriminator is a 6-layered 3D CNN with 3 $\times$ 3$\times$ 3 kernels and
strides of 2 $\times$ 2 $\times$ 2\. The number of filters in each layer is
[16, 32, 64, 128, 256, 256] respectively. After the last convolution, we add a
fully connected layer followed by Sigmoid activation.

##  7 Additional Results

Table 4: Quantitative results (SSIM, PSNR and FVD scores) for 72 views experiment on CMU Panoptic dataset on $56\times 56$ resolution. For each panel, input views are fixed to ($v_{1}$,$v_{2}$,$v_{3}$,$v_{4}$,$v_{5}$) and we test output views from $v_{6}$ to $v_{24}$.  Query | Panel 4 | Panel 5 | Panel 17
---|---|---|---
View | SSIM($\uparrow$) | PSNR($\uparrow$) | FVD($\downarrow$) | SSIM($\uparrow$) | PSNR($\uparrow$) | FVD($\downarrow$) | SSIM($\uparrow$) | PSNR($\uparrow$) | FVD($\downarrow$)
$v_{6}$ | 0.814 | 24.46 | 7.58 | 0.852 | 26.79 | 11.79 | 0.855 | 25.94 | 11.80
$v_{7}$ | 0.795 | 24.56 | 11.64 | 0.843 | 25.66 | 13.17 | 0.820 | 26.06 | 13.17
$v_{8}$ | 0.794 | 24.02 | 10.49 | 0.842 | 25.94 | 16.01 | 0.834 | 26.15 | 16.04
$v_{9}$ | 0.813 | 24.7 | 10.66 | 0.827 | 25.16 | 17.65 | 0.795 | 25.18 | 14.23
$v_{10}$ | 0.829 | 25.58 | 10.91 | 0.828 | 24.99 | 14.80 | 0.807 | 25.69 | 14.82
$v_{11}$ | 0.836 | 25.36 | 11.98 | 0.823 | 24.21 | 12.72 | 0.809 | 24.33 | 12.67
$v_{12}$ | 0.871 | 26.50 | 8.24 | 0.807 | 23.17 | 10.92 | 0.784 | 24.17 | 10.89
$v_{13}$ | 0.834 | 25.09 | 10.80 | 0.832 | 23.61 | 13.33 | 0.805 | 24.62 | 13.32
$v_{14}$ | 0.811 | 24.17 | 15.01 | 0.808 | 23.26 | 11.86 | 0.789 | 24.25 | 11.85
$v_{15}$ | 0.827 | 24.7 | 8.24 | 0.844 | 24.43 | 16.60 | 0.797 | 23.90 | 16.60
$v_{16}$ | 0.852 | 26.48 | 12.69 | 0.860 | 25.06 | 10.84 | 0.813 | 24.62 | 10.87
$v_{17}$ | 0.868 | 26.80 | 11.01 | 0.835 | 23.76 | 10.50 | 0.808 | 24.31 | 10.48
$v_{18}$ | 0.846 | 27.43 | 10.44 | 0.817 | 24.82 | 9.86 | 0.810 | 24.14 | 9.84
$v_{19}$ | 0.829 | 25.01 | 12.25 | 0.836 | 23.90 | 13.71 | 0.816 | 24.60 | 13.70
$v_{20}$ | 0.820 | 25.02 | 13.26 | 0.851 | 25.68 | 13.97 | 0.827 | 24.95 | 13.97
$v_{21}$ | 0.822 | 25.06 | 12.14 | 0.811 | 24.60 | 14.91 | 0.832 | 25.83 | 14.89
$v_{22}$ | 0.810 | 24.82 | 10.08 | 0.839 | 24.44 | 15.04 | 0.804 | 24.65 | 15.04
$v_{23}$ | 0.824 | 25.15 | 15.53 | 0.839 | 24.73 | 14.05 | 0.811 | 24.66 | 14.02
$v_{24}$ | 0.835 | 25.61 | 15.94 | 0.823 | 23.65 | 13.31 | 0.750 | 20.59 | 14.70
Fig. 7: Qualitative Results on CMU Panoptic dataset showing synthesized frames
employing five input views (v1 through v5) and randomly selected query views
from the remaining views ($v_{6}$ \- $v_{24}$) for each panel. We show 3
samples and the resolution is $56\times 56$.

###  7.1 CMU Panoptic Dataset [10]

Large-scale Multiview Experiment.  We provide additional qualitative results
on CMU Panoptic Dataset.

Figure 5 depicts the frames at time t = 0, 8 and 15 for five input views as
well as the synthesized frames and the corresponding ground truth frames from
the query view.

The detailed quantitative evaluation is provided in Table 4. The evaluation is
conducted per panel separately. For each panel, the viewpoints from $v_{1}$ to
$v_{5}$ are fixed as input views and the output video clips for the remaining
19 viewpoints ($v_{6}$ to $v_{24}$) are synthesized. The SSIM, PSNR and FVD
scores are provided in Table 4. The average scores for panel 4, 5 and 17 are

For the same configuration, we provide additional qualitative results in
Figure 7 for randomly selected query views from the pool of viewpoints from
$v_{6}$ to $v_{24}$. It can be observed that video frames and motion are
equally good for each of the query views given the five input views as shown.

Table 5: SSIM and PSNR scores for investigating the effect of varying the number of input views. We randomly select 9 query views $v_{k}$ from panel 4 and increase the input views from 1 to 5. | Query View | Input Views
---|---|---
Metrics | $v_{k}$ | $(v_{1})$ | ($v_{1},v_{2}$) | ($v_{1},v_{2},v_{3}$) | ($v_{1},v_{2},v_{3},v_{4}$) | ($v_{1},v_{2},v_{3},v_{4},v_{5}$)
| $v_{6}$ | 0.789 | 0.803 | 0.808 | 0.812 | 0.814
| $v_{10}$ | 0.806 | 0.818 | 0.820 | 0.822 | 0.829
| $v_{11}$ | 0.796 | 0.815 | 0.823 | 0.824 | 0.836
| $v_{12}$ | 0.858 | 0.867 | 0.870 | 0.871 | 0.871
SSIM($\uparrow$) | $v_{13}$ | 0.807 | 0.816 | 0.820 | 0.819 | 0.834
| $v_{14}$ | 0.776 | 0.781 | 0.784 | 0.784 | 0.811
| $v_{16}$ | 0.841 | 0.844 | 0.847 | 0.850 | 0.852
| $v_{17}$ | 0.846 | 0.856 | 0.857 | 0.858 | 0.868
| $v_{20}$ | 0.786 | 0.797 | 0.809 | 0.812 | 0.820
| $v_{6}$ | 23.51 | 24.06 | 24.27 | 24.41 | 24.46
| $v_{10}$ | 24.48 | 24.87 | 25.00 | 25.02 | 25.58
| $v_{11}$ | 24.17 | 24.92 | 25.28 | 25.30 | 25.36
| $v_{12}$ | 25.60 | 26.14 | 26.35 | 26.46 | 26.50
PSNR($\uparrow$) | $v_{13}$ | 24.82 | 25.16 | 25.25 | 25.22 | 25.09
| $v_{14}$ | 23.34 | 23.54 | 23.63 | 23.80 | 24.17
| $v_{16}$ | 25.98 | 26.14 | 26.32 | 26.40 | 26.48
| $v_{17}$ | 25.90 | 26.56 | 26.73 | 26.80 | 26.80
| $v_{20}$ | 23.99 | 24.29 | 24.64 | 24.83 | 25.02
Table 6: Quantitative comparison with state-of-the-art viewLSTM[9] on CMU Panoptic dataset at $56\times 56$ resolution and single-input setup. Our method performs significantly better than SOTA viewLSTM[9] on all three evaluation metrics for all input-output view combinations.  |  | Input View $\xrightarrow{}$ Output View | Average
---|---|---|---
Method | Metrics |  $v_{1}$ $\xrightarrow{}$ $v_{2}$ |  $v_{1}$ $\xrightarrow{}$ $v_{3}$ |  $v_{2}$ $\xrightarrow{}$ $v_{1}$ |  $v_{2}$ $\xrightarrow{}$ $v_{3}$ |  $v_{3}$ $\xrightarrow{}$ $v_{1}$ |  $v_{3}$ $\xrightarrow{}$ $v_{2}$ | Score
|  SSIM ($\uparrow$) | 0.675 | 0.684 | 0.686 | 0.673 | 0.690 | 0.680 | 0.681
[9] |  PSNR ($\uparrow$) | 23.03 | 23.16 | 22.56 | 23.07 | 22.69 | 23.24 | 22.96
|  FVD ($\downarrow$) | 22.43 | 25.67 | 20.62 | 25.73 | 20.32 | 22.48 | 22.87
|  SSIM ($\uparrow$) | 0.740 | 0.751 | 0.753 | 0.748 | 0.742 | 0.732 | 0.744
[21] |  PSNR ($\uparrow$) | 21.33 | 21.75 | 22.53 | 21.41 | 21.44 | 20.70 | 21.53
|  FVD ($\downarrow$) | 18.23 | 18.98 | 16.23 | 17.95 | 17.88 | 16.42 | 17.62
|  SSIM ($\uparrow$) | 0.862 | 0.853 | 0.852 | 0.861 | 0.858 | 0.876 | 0.859
Ours |  PSNR ($\uparrow$) | 26.37 | 25.48 | 25.85 | 25.58 | 26.34 | 26.87 | 26.08
|  FVD ($\downarrow$) | 8.51 | 10.12 | 9.21 | 9.98 | 8.93 | 8.20 | 9.15

Effect of Multiple Input Views.

This analysis is conducted to understand the impact of multiple input views on
the quality of the synthesized videos. The number of input clips is increased
from 1 to 5 to synthesize the video from the query view and report the results
in Figure 6. As expected, with an increase in the number of input views the
video quality improves. This validates that the network is able to capture and
aggregate the features pertinent to each view and exploit them to synthesize
the query view.

The SSIM and PSNR scores are reported for the synthesized video frames in
Table 5. The input and query views are selected from Panel 4 for this
analysis. We observe that the scores improve as the number of input view
increases from single to multiple views. This is possible since the multiple
input views are successful at providing greater scene details useful to learn
to synthesize the query view.

Higher Resolution Experiment For the same setup, we also conduct experiment on
224 $\times$ 224 resolution. The qualitative results are shown in Figure 8.
The model is able to preserve fine details and produce sharper frames with an
SSIM and PSNR score of 0.812 and 24.55, respectively.

Fig. 8: Qualitative Results on CMU Panoptic dataset with multi-view input
setup. We show three samples and the resolution is $224\times 224.$

Comparison with viewLSTM[9].  Here, we provide additional qualitative and
quantitative evaluation of our proposed approach with state-of-the-art
viewLSTM [9] in Figure 9 and Table 6 respectively. The experimental details
are provided in the manuscript in Section 5.1.

In Figure 9, we show three samples from different input viewpoints, each at
three time instances, t = 0, 8 and 15. The actors in the images are performing
different actions. Qualitatively, our proposed method is successful at
synthesizing better images with sharper foreground-background boundaries and
also capture the motion of the person along the temporal sequence.

The detailed quantitative result in Table 6 provides the performance of our
method and the state-of-the-art viewLSTM [9] for view transformation between
the Panels 2,4 and 10 (Node 10). The SOTA method uses the query view skeleton
priors to synthesize the query RGB frames, whereas our method doesn’t utilize
the skeleton information from the query view. We observe that our proposed
method outperforms the existing SOTA by huge margins for all evaluation
metrics.

Fig. 9: Qualitative Results showing the comparison of our proposed method with
state-of-the-art method viewLSTM [9] on CMU Panoptic dataset. We show 3
samples, each with frames at time t =0, 8 and 15; and the resolution is
$56\times 56$.

###  7.2 NTU-RGB+D Dataset [11]

Refer Section 4.2 in the manuscript for the summarized quantitative analysis
on network ablations. Here, we provide the detailed network and loss ablation
in terms of quantitative results and qualitative visualizations on NTU-RGB+D
Dataset along with the comparison with existing works.

Fig. 10: Qualitative results showing Network Ablations on two samples of NTU-
RGB+D Dataset [11], showing the comparison between View Dependent (tested),
Global Representation(tested) and Full Model at $56\times 56$ resolution for
multiple input view setup.  Fig. 11: Qualitative results showing Network
Ablations on two samples of NTU-RGB+D Dataset [11], showing the comparison
between View Dependent (trained), Global Representation (trained) and Full
Model at $56\times 56$ resolution for single input view setup.

Network Ablations. We performed two types of ablations for evaluating the role
of the global and view-dependent representations. First, we tested the fully
trained network by using only one representation at a time. Figure 10 shows
the testing results with global representation $r_{g}$ and/or view-dependent
representation $r_{t}$. With global representation $r_{g}$, the foreground and
background are almost formed however view-dependent representation $r_{t}$ is
not meaningful alone. The $r_{t}$ helps by filling in the view specific finer
details and only makes sense along with the global representation. It is
visibly obvious that best results are obtained with the dual representation
$r$. The detailed quantitative evaluation for single and multi-input setup is
shown in Table 7 (Row 1-2).

Table 7: SSIM, PSNR and FVD scores for Network Ablations on NTU-RGB+D Dataset [11] using View Dependent (VD) stream, Global Representation (GR) stream and the full model (Full) on $56\times 56$ resolution. The left panel shows the results for multi-input view setup, whereas the right panel shows single-input view setup results.  |  | Multi-Input Setup | Avg | Single Input Setup | Avg
---|---|---|---|---|---
Network | Metric | ($v_{1},v_{2}$) $\xrightarrow{}$ $v_{3}$ | ($v_{2},v_{3}$) $\xrightarrow{}$ $v_{1}$ | ($v_{1},v_{3}$) $\xrightarrow{}$ $v_{2}$ | Score |  $v_{1}$ $\xrightarrow{}$ $v_{2}$ |  $v_{1}$ $\xrightarrow{}$ $v_{3}$ |  $v_{2}$ $\xrightarrow{}$ $v_{1}$ |  $v_{2}$ $\xrightarrow{}$ $v_{3}$ |  $v_{3}$ $\xrightarrow{}$ $v_{1}$ |  $v_{3}$ $\xrightarrow{}$ $v_{2}$ | Score
| SSIM($\uparrow$) | 0.313 | 0.352 | 0.340 | 0.335 | 0.323 | 0.327 | 0.329 | 0.316 | 0.315 | 0.323 | 0.323
VD(tested) | PSNR($\uparrow$) | 12.67 | 13.20 | 11.88 | 12.58 | 12.09 | 12.52 | 12.59 | 11.92 | 11.89 | 12.11 | 12.18
| FVD($\downarrow$) | 18.20 | 18.48 | 18.03 | 18.23 | 19.23 | 19.93 | 18.96 | 18.92 | 17.78 | 17.67 | 18.74
| SSIM($\uparrow$) | 0.634 | 0.647 | 0.637 | 0.639 | 0.634 | 0.627 | 0.629 | 0.624 | 0.628 | 0.631 | 0.629
GR(tested) | PSNR($\uparrow$) | 17.23 | 17.72 | 17.40 | 17.45 | 16.12 | 16.96 | 16.66 | 17.02 | 16.67 | 16.31 | 16.62
| FVD($\downarrow$) | 15.45 | 15.82 | 15.76 | 15.68 | 15.39 | 15.87 | 16.02 | 16.28 | 16.71 | 15.39 | 15.94
| SSIM($\uparrow$) | 0.548 | 0.579 | 0.563 | 0.563 | 0.539 | 0.517 | 0.548 | 0.547 | 0.537 | 0.519 | 0.534
VD(trained) | PSNR($\uparrow$) | 16.13 | 16.21 | 16.32 | 16.22 | 15.86 | 15.62 | 15.59 | 15.59 | 15.63 | 15.747 | 15.67
| FVD($\downarrow$) | 16.84 | 16.57 | 16.92 | 16.77 | 17.48 | 17.37 | 16.77 | 17.11 | 16.90 | 17.47 | 17.18
| SSIM($\uparrow$) | 0.783 | 0.814 | 0.780 | 0.792 | 0.781 | 0.777 | 0.804 | 0.774 | 0.799 | 0.773 | 0.784
GR(trained) | PSNR($\uparrow$) | 19.99 | 20.67 | 19.50 | 20.05 | 19.56 | 19.83 | 20.2 | 19.67 | 20.04 | 19.23 | 19.755
| FVD($\downarrow$) | 13.25 | 13.54 | 13.57 | 13.45 | 13.69 | 13.17 | 12.12 | 13.78 | 13.17 | 13.89 | 13.33
| SSIM($\uparrow$) | 0.801 | 0.834 | 0.805 | 0.813 | 0.804 | 0.793 | 0.828 | 0.783 | 0.815 | 0.790 | 0.802
Full | PSNR($\uparrow$) | 20.65 | 21.40 | 20.34 | 20.79 | 20.44 | 20.48 | 20.97 | 20.17 | 20.63 | 20.02 | 20.45
| FVD($\downarrow$) | 12.75 | 11.25 | 12.93 | 12.31 | 11.01 | 11.04 | 11.76 | 12.78 | 11.94 | 12.28 | 11.80
Table 8: SSIM and PSNR scores for different combination of losses on NTU Dataset[11] $56\times 56$ resolution. $L_{2}$ is the Reconstruction loss, $L_{p}$ is the Perceptual loss and $L_{adv}$ is the Adversarial loss. The left panel shows the results for multi-input view setup, whereas the right panel shows single-input view setup results. | Losses | Multi-Input Setup | Avg | Single-Input Setup | Avg
---|---|---|---|---|---
Metrics | $L_{2}$ | $L_{p}$ | $L_{adv}$ | ($v_{1},v_{2}$) $\xrightarrow{}$ $v_{3}$ | ($v_{2},v_{3}$) $\xrightarrow{}$ $v_{1}$ | ($v_{1},v_{3}$) $\xrightarrow{}$ $v_{2}$ | Score |  $v_{1}$ $\xrightarrow{}$ $v_{2}$ |  $v_{1}$ $\xrightarrow{}$ $v_{3}$ |  $v_{2}$ $\xrightarrow{}$ $v_{1}$ |  $v_{2}$ $\xrightarrow{}$ $v_{3}$ |  $v_{3}$ $\xrightarrow{}$ $v_{1}$ |  $v_{3}$ $\xrightarrow{}$ $v_{2}$ | Score
| ✓ |  |  | 0.699 | 0.723 | 0.704 | 0.708 | 0.689 | 0.704 | 0.653 | 0.678 | 0.651 | 0.670 | 0.674
|  | ✓ | ✓ | 0.737 | 0.775 | 0.745 | 0.752 | 0.703 | 0.714 | 0.601 | 0.710 | 0.662 | 0.662 | 0.675
SSIM | ✓ | ✓ |  | 0.791 | 0.833 | 0.796 | 0.806 | 0.773 | 0.774 | 0.778 | 0.768 | 0.766 | 0.749 | 0.768
($\uparrow$) | ✓ |  | ✓ | 0.784 | 0.825 | 0.789 | 0.799 | 0.783 | 0.776 | 0.798 | 0.761 | 0.815 | 0.787 | 0.786
| ✓ | ✓ | ✓ | 0.801 | 0.834 | 0.805 | 0.813 | 0.804 | 0.793 | 0.828 | 0.783 | 0.815 | 0.790 | 0.802
| ✓ |  |  | 17.56 | 18.55 | 17.89 | 18.00 | 17.25 | 18.27 | 16.92 | 17.27 | 16.55 | 17.01 | 17.21
|  | ✓ | ✓ | 18.96 | 19.94 | 18.78 | 19.22 | 17.85 | 18.464 | 15.82 | 18.39 | 17.34 | 17.34 | 17.53
PSNR | ✓ | ✓ |  | 19.69 | 21.63 | 20.38 | 20.72 | 19.76 | 20.24 | 19.72 | 18.03 | 19.31 | 18.93 | 19.29
($\uparrow$) | ✓ |  | ✓ | 20.19 | 21.06 | 19.91 | 20.38 | 19.76 | 20.00 | 20.15 | 18.58 | 19.55 | 18.8 | 19.47
| ✓ | ✓ | ✓ | 20.65 | 21.40 | 20.34 | 20.79 | 20.44 | 20.48 | 20.97 | 20.17 | 20.63 | 20.02 | 20.45
Fig. 12: Qualitative results on NTU-RGB+D Dataset[11] showing comparison with
VDG [23], PG2 [24], ResNet [25] and viewLSTM [9]. We show two samples and the
resolution is $112\times 112$ with single-view input setup.

Secondly, we trained the network with only one representation at a time to
investigate the impact of each branch in our architecture. To examine the role
of View Dependent block, we employ View Dependent block $V_{T}$, in
conjunction with Video Encoder $V_{E}$, and Video Decoder $V_{D}$ (i.e. Global
Representation block $V_{G}$ is removed). To examine the role of Global
Representation block, we employ Global Representation block $V_{G}$, in
conjunction with Video Encoder $V_{E}$, and Video Decoder $V_{D}$ (i.e. View
Dependent block $V_{T}$ is removed).

The final experiment is performed on the full model $i.e.$ $V_{T}$ and $V_{G}$
combined. The qualitative results for network ablations are presented in
Figure 11. The first row shows the frames from input clip, followed by the
synthesized outputs for View Dependent (VD) stream, Global Representation (GR)
stream and the full model in rows two, three and four respectively. Finally,
the ground truth frames are shown in the last row. The outputs from the
network with View Dependent stream contains visible artifacts in the frames.
The network with Global Representation stream shows improvements in visual
quality of the output over the network with View Dependent branch only.
Finally, the full model is able to add finer details like arm and leg (Figure
11, right panel) in the synthesized output.

Table 7 (Row 3-4) shows detailed quantitative results for network ablations
for single and multiple input-view settings. The quantitative results are
consistent for both single and multiple input-view setups, and for all input
and output view combinations. The results show that View Dependent and Global
Representation streams learn complementary features that are aggregated by the
full model (Table 7 Row 5) and generate better results.

Table 9: SSIM scores comparison with existing frame based methods[23, 24] and state-of-the-art video-based method, viewLSTM [9] on  NTU-RGB+D dataset at 112 $\times$ 112 resolution with single-view input setup. The scores for viewLSTM [9] are computed for the videos synthesized using both depth and skeleton priors from the query view. Metric | Methods
---|---
| [23] | [24] | [9] | Ours
SSIM($\uparrow$) | 0.554 | 0.560 | 0.783 | 0.791
Fig. 13: Qualitative results on NTU-RGB+D Dataset[11] showing comparison with
viewLSTM [9]. We show two samples and the resolution is $112\times 112$ with
single-view input setup.

Comparison with Existing Works.

In order to compare with the existing novel view prediction methods, we test
our network to synthesize a query view video using a single view video as
input. We input multiple copies of the same input clip, as explained earlier
with CMU Panoptic dataset. The experiment is conducted at 112 $\times$ 112
resolution. We compare with a state-of-the-art video-based method, viewLSTM
[9] and two frame-based methods [23, 24].

The quantitative comparison is shown in Table 9. It can be observed that SSIM
with proposed approach is comparable to ViewLSTM [9], without using any priors
from the query view. ViewLSTM [9], on the other hand, uses multiple priors
from the query view, i.e both depth and skeleton information. The frame-based
methods [23, 24] were reportedly trained to predict smaller view-point
changes. When trained on NTU-RGB+D there are larger view point changes and our
method predicts better quality videos. Also, these methods are unable to
capture the temporal coherency between the synthesized frames.

Table 10: SSIM, PSNR and FVD scores for different combination of losses on NTU-RGB+D Dataset [11] on $56\times 56$ resolution.  Losses | Metrics
---|---
$L_{2}$ | $L_{p}$ | $L_{adv}$ | SSIM($\uparrow$) | PSNR($\uparrow$) | FVD($\downarrow$)
✓ |  |  | 0.708 | 18.00 | 15.42
| ✓ | ✓ | 0.752 | 19.22 | 13.96
✓ | ✓ |  | 0.806 | 20.72 | 13.28
✓ |  | ✓ | 0.799 | 20.38 | 13.76
✓ | ✓ | ✓ | 0.817 | 20.77 | 12.31

Figure 13 shows a qualitative comparison between the outputs synthesized by
viewLSTM [9] (second row) and our proposed method (third row) on NTU-RGB+D
dataset. The samples are shown for three time steps and for two samples. As
seen with the SSIM scores, our method produces sharper foreground objects and
better background details.

We provide more qualitative results comparing with our baselines VGD [23], PG2
[24], ResNet [25] and viewLSTM [9] in Figure 12. As can be seen, the frame-
bases baselines VDG and PG2 are not able to produce even the scene structure
correctly. ResNet, which is used as a video-based baseline, generates correct
scene structure but has visible artifacts present. VDNet produces better
results but the video quality is low. Finally, we can observe that the
proposed method perfectly generates the background as well as the actor
performing the action with better motion as well.

Loss Ablations.  Figure 14 shows the qualitative results obtained by employing
different losses. Using only the reconstruction loss produces very blurry
results. Adversarial and perceptual loss combined ($L_{p}+L_{adv}$) helps in
sharpening the frames but does not get the color information and background
correctly. Adding perceptual loss with reconstruction loss ($L_{2}+L_{p}$)
gets the background information well, but the structures of bodies are not
defined. $L_{2}+L_{adv}$ also produces sharp results with correct background
but lacks in producing correct body structures. Finally, combining
$L_{2}+L_{p}+L_{adv}$, we get improved results compared to other losses.

Table 8 shows detailed quantitative results for loss ablations for single
input-view settings. The results are consistent for different input and output
view combinations. The use of all three losses helps to synthesize query view
outputs better than using only one or two loss combinations as validated by
the table.

Fig. 14: Qualitative Results for loss ablations on NTU Dataset; We show frames
obtained by employing different losses. We obtained the the best visual
results employing all three losses ($L_{2}+L_{p}+L_{adv}$); here $L_{2}$ :
Reconstruction Loss, $L_{p}$ : Perceptual Loss, $L_{adv}$ : Adversarial Loss.