# Multi-Scale Video Frame-Synthesis Network with Transitive Consistency Loss

Zhe Hu
Hikvision
<EMAIL_ADDRESS>Yinglan Ma
Adobe
<EMAIL_ADDRESS>Lizhuang Ma
East China Normal University
<EMAIL_ADDRESS>
###### Abstract

Traditional approaches to interpolate/extrapolate frames in a video sequence
require accurate pixel correspondences between images, e.g., using optical
flow. Their results stem on the accuracy of optical flow estimation, and could
generate heavy artifacts when flow estimation failed. Recently methods using
auto-encoder has shown impressive progress, however they are usually trained
for specific interpolation/extrapolation settings and lack of flexibility and
generality for more applications. Moreover, these models are usually heavy in
terms of of model size which constrains the application on mobile devices. In
order to reduce these limitations, we propose a unified network to
parameterize the interest frame position and therefore infer
interpolate/extrapolate frames within the same framework. To achieve this, we
introduce a transitive consistency loss to better regularize the network. We
adopt a multi-scale structure for the network so that the parameters can be
shared across multi-layers. Our approach avoids expensive global optimization
of optical flow methods, and is efficient and flexible for video
interpolation/extrapolation applications. Experimental results have shown that
our method performs favorably against state-of-the-art methods.

##  1 Introduction

Video frame synthesis, including interpolation and extrapolation, is a classic
problem in computer vision and has attracted much interest recently due to the
trend of unsupervised/self-supervised learning of video representation. Frame
interpolation has been used in numerous applications such as temporal
upsampling, frame rate conversion and view synthesis. Frame extrapolation, on
the other hand, is related to predicting motion and it is a critical problem
for learning interaction with the physical world, e.g., robots, autonomous
driving of cars and drones.

Video frame synthesis itself is a challenging problem in the exist of moving
deformable objects, object occlusion, illumination change, camera movement,
and etc.. Traditional solutions to frame synthesis first compute dense
correspondences, mostly optical flow, and then render image via
correspondence-based image warping. Such methods heavily rely on
computationally expensive global optimization, due to inherent ambiguities in
computing correspondences.

To avoid explicit estimation of dense correspondences, recent methods
formulate frame interpolation [31, 32] or extrapolation [9, 42, 5] as a
convolution process and estimate the convolution using the neural networks. To
speed up the convolution process with spatially-varying kernels, a spatially-
adaptive separable convolution approach has been presented for video frame
interpolation [32], and obtain impressive interpolation results . However,
they still pose the correspondence estimation as a separate step for the video
interpolation process.

Most of the work that bypass correspondence estimation treat interpolation
problem alone or extrapolation problem alone in a specific setting, e.g.,
interpolating the frame right in the middle of two input frames, extrapolating
next frame with a certain interval. This is because existing networks are
specifically designed for a fixed temporal position prediction, and lacks of
generality and flexibility. In fact, if we consider a continuous sequence of
video frames in temporal domain, either interpolating or extrapolating frames
based on a subset of frames can be viewed as a unified problem that predicts
frames at certain positions, and therefore should be handled as a whole. This
is naturally solved in optical-flow-based methods, in which
interpolation/extrapolation are handled along flow directions. Thus, how to
elegantly formulate and solve the problem with a unified neural network
becomes a critical problem.

In this work, we propose a unified end-to-end network for frame synthesize
(interpolate/extrapolate) by parameterizing the temporal position of the
interest frame. To our best knowledge, this is the first work to synthesize
video frames, interpolating/extrapolating at any time ratio, in a single
network (without re-training for other ratio settings). The proposed Multi-
Scale Frame-Synthesis Network (MSFSN) progressively reconstruct interest
frames in a coarse-to-fine manner. We introduce a technique by enforcing
transitive properties to enable training interpolation/extrapolation tasks in
a single network. Experimental results show that our network is smaller than
autoencoder based methods, e.g., [31, 32, 24], while generating comparable
reconstruction accuracy. Our network architecture naturally enables parameter
sharing across pyramid levels, since different pyramid levels share the same
input/output format and purpose. By sharing parameters across levels, we
obtain a compact and flexible model that more levels can be stacked during
test to accommodate higher capacity of deeper networks.

---
Figure 1: Example of our multi-scale network. Each level is a sub-network
consisting of $D$ residual blocks [14] as shown in left bottom. Parameters are
shared across pyramid levels, since different pyramid levels (except the
coarsest level) share the same input/output format and purpose. We only show a
network with 3 pyramid levels for simplicity.

##  2 Related Work

The proposed frame-synthesis GAN aims on interpolate/extrapolate video frames
with a unified framework. We adopt a transitive consistency loss for
leveraging the problems. Thus, we focus on reviewing work that are relevant to
our problem and approach.

Video frame interpolation/extrapolation Video frame interpolation is a classic
topic in computer vision and video processing. Traditional frame interpolation
methods estimate dense correspondence, mostly optical flow, between input
frames and then interpolate one or more intermediate frames by warping the
input images [1, 41]. The performance of these methods heavily stem on
estimation accuracy of optical flow and require special processing to reduce
artifacts. Other than optical flow, approaches along view synthesis
perspective have been explored [45, 26]. Meyer et al. [29] developed a phase-
based interpolation method that represents motion in the format of pixel phase
shift and therefore render intermediate frames by modifying pixel phase. This
phase-based method often produces impressive interpolation results, but could
fail to preserve high-frequency details under large temporal changes.

Recent approaches formulate frame interpolation/extrapolation with a single
convolution step that combines motion estimation and frame synthesis [42, 5,
32, 31, 9]. These methods generate impressive results by estimating spatially-
varying kernels within the network and convolve them with input frames to
synthesize a new frame. Since these methods involves heavy convolution with
large spatially variant kernels, they suffer from high computational load and
memory for high-resolution videos. Niklaus et al. [32] propose a method to
speed up the final convolution process via approximating 2D kernels with
separable 1D kernel, and could process 1080p videos in one pass. Recently, an
approach has been proposed to output dense voxel flows through a network and
use them to generate interpolated frames [24]. The generated 3D voxel flows
encode the motion variance in temporal domain and generate intermediate frames
via trilinear interpolation.

Image reconstruction Our work is also inspired by the recent progress of
network methods in image enhancement and reconstruction [10, 18, 23, 4, 7, 21,
20, 30, 8]. Pixel-wise loss functions such as MSE struggle to reconstruct
high-frequency details, and therefore minimizing MSE usually lead to over-
smooth results [27, 18]. Features extracted from a pre-trained VGG network
instead of per-pixel error has been proposed to render visually pleasing
images [3, 18]. And perceptually more realistic results can be obtained via
combining those with generative adversarial networks (GANs) [12] for image
generation tasks [43, 27, 22].

Transitive property Transitive property is a property of equivalence, and it
has been used to regularize structured data for a long time in the literature.
Forward-backward consistency check has proved to be efficient in depth
estimation [15] and optical flow [40]. Language translation can also be
improved via consistency check with back translation [2, 13]. More recently,
higher-order cycle consistency has been used in image-to-image translation
[44]. In this work, we are introducing a transitive consistency loss to
facilitate training process for video representation.

##  3 Frame-Synthesis Generative Adversarial Network

Our goal is to learn a mapping function $G:S\times S\times T\rightarrow S$ in
image domain $S$ and time domain $T$, for frame interpolation/extrapolation.
We denote a triplet $(x_{t_{1}},x_{t_{2}},t_{p})$, where $x_{t_{i}}\in S,\quad
i=1,2$ represents an observed frame at timestamp $t_{i}$, and $t_{p}$ is the
timestamp of our interest frame. Without loss of generality, we assume
$t_{1}<t_{2}$. We denote video frame distribution as $x_{t}\sim p_{data}$. The
mapping predicts the interest frame $y_{t_{p}}=G(x_{t_{1}},x_{t_{2}},t_{p})$
at timestamp $t_{p}$, and $y_{t_{p}}\in S$. If $t_{1}<t_{p}<t_{2}$, the
problem is a frame interpolation problem. If $t_{p}<t_{1}$ or $t_{p}>t_{2}$,
the problem becomes a frame extrapolation problem.

###  3.1 Network Architecture

We construct our network using a multi-scale structure [6, 33, 20] as shown in
Figure 1. Each level is a sub-network consisting of $D$ residual blocks [14].
We adopt a modified residual block by removing batch normalizations. We will
discuss the effect of the block number in Section 4.

Our model takes a triplet $(x_{t_{1}},x_{t_{2}},t_{p})$ as input and
progressively predicts the interpolated/extrapolated frames from coarse levels
to fine levels. Let $S$ and $y_{t_{p}}^{s}$ represent number of pyramid levels
and the predicted frame at level $s,s\in\\{1,2,\dots,S\\}$ (from coarse to
fine, level $S$ is the final reconstruction level). Let $x_{t_{1}}$ and
$x_{t_{2}}$ be $m\times n$ images that are multiples of $2^{S-1}$.

At each pyramid level $s$, we feed downsampled frames
$x_{t_{1}}^{s},x_{t_{2}}^{s}$ of size $m/2^{S-s}\times n/2^{S-s}$ and an
initial interpolated/extrapolated image $y_{t_{p}}^{s-1}\uparrow$ upsampled
from previous level. We predict the frame $y_{t_{p}}^{s}$ at level $s$ by
passing input triplet $(x_{t_{1}}^{s},x_{t_{2}}^{s},y_{t_{p}}^{s-1}\uparrow)$
through our sub-network. In this work, we use a multi-scale structure of 4
levels and 3 sub-networks $\\{N_{s}\\}_{s=2}^{4}$ are
$N_{s}:(x_{t_{1}}^{s},x_{t_{2}}^{s},y_{t_{p}}^{s-1}\uparrow)=y_{t_{p}}^{s}$.
The sub-network at coarsest level $s=1$ takes $(x_{t_{1}}^{1},x_{t_{2}}^{1})$
as input, and is denoted as
$N_{1}:(x_{t_{1}}^{1},x_{t_{2}}^{1})=y_{t_{p}}^{1}$.

Parameter sharing across pyramid levels The entire network is a cascade of
sub-networks with the same structure at levels except the coarsest level 1. We
propose to share the network parameters across those pyramid levels because
the sub-networks at these levels share the same structure and the task (i.e.,
predicting the interest frames with a lower-resolution version as input). As
shown in Figure 1, we share the parameters of the sub-networks across all the
pyramid levels. As a result, the number of network parameters is independent
of the number of levels. We can use one single set of parameters to predict
large motion by increasing the number of pyramid levels and this is discussed
in Section 4.

###  3.2 Loss Function

Considering the supreme benefit of adversarial training on synthesizing
realistic texture [21, 44], we formulate our model in a GAN framework. For the
mapping function $G$, we introduce an adversarial discriminator $D$, where $D$
aims to distinguish between images and generated images by $G$ [12]. Our
objective contains four terms: 1) pixel reconstruction loss; 2) feature
reconstruction loss to encourage similarity in feature representation; 3)
adversarial loss for matching the distribution of generated images to the data
distribution in the target feature domain; 4) transitive consistency loss to
enhance the mapping function $G$ with more constraints.

|  |
---|---|---
Figure 2: Example of transitive property. We aim to learn the frame synthesis
mapping $G:S\times S\times T\rightarrow S$. Let
$x_{t_{1}},x_{t_{p}},x_{t_{2}}$ denote three frames capturing the same scene
in a video, and the mapping $y_{t_{p}}=G(x_{t_{1}},x_{t_{2}},t_{p})$. The
frames should lie on a manifold that represents the images of the scene,
specifically a curve in the manifold that represents the frames in the video.
To further regularize the mappings, we introduce a transitive consistency loss
that captures the transitive mappings $G(y_{t_{p}},x_{t_{2}},t_{1})\approx
x_{t_{1}}$ and $G(x_{t_{1}},y_{t_{p}},t_{2})\approx x_{t_{2}}$.

Pixel reconstruction loss We adopt a per-pixel difference loss in $\ell_{1}$
norm as

$\mathcal{L}_{pix}(G)=\mathbb{E}_{p_{data}}[|G(x_{t_{1}},x_{t_{2}},t_{p})-y_{t_{p}}|_{1}],$
(1)

which is also suggested in [39, 32] to reduce blur effect rather than
$\ell_{2}$ norm.

Feature reconstruction loss Inspired by perceptual loss functions used in
image reconstruction and style transfer networks [10, 8, 18], we use a feature
reconstruction loss:

$\mathcal{L}_{feat}(G)=\mathbb{E}_{p_{data}}[\|\phi(G(x_{t_{1}},x_{t_{2}},t_{p}))-\phi(y_{t_{p}})\|_{2}].$
(2)

This loss is to encourage the feature of the output image to match that of the
target image, and we use a similar loss network $\phi$ as [10, 18] based on
16-layer VGG network [37] pretrained on ImageNet [35].

Adversarial loss We apply the adversarial loss [12] to the mapping function
$G$ and its discriminator $D$ as:

$\displaystyle\mathcal{L}_{GAN}(G,D)$ $\displaystyle=$
$\displaystyle\mathbb{E}_{p_{data}}[log_{x_{t_{p}}\ }D(x_{t_{p}})]$
$\displaystyle+$
$\displaystyle\mathbb{E}_{p_{data}}[log(1-D(G(x_{t_{1}},x_{t_{2}},t_{p}))],$

where $G$ tries to generate image $G(x_{t_{1}},x_{t_{2}},t_{p})$ that looks
similar to the frame $x_{t_{p}}$ at timestamp $t_{p}$, while $D$ aims to
distinguish between generated image $G(x_{t_{1}},x_{t_{2}},t_{p})$ and real
sample $x_{t_{p}}$. In other words, $G$ and $D$ are trained to optimize the
objective in a minmax manner.

Adversarial training, in theory, can learn a discriminator $D$ to help the
mapping function G that produces outputs identically distributed as target
domains $S$ when $G$ is a stochastic function) [12]. However, with large
enough network capacity, there are countless network with different parameter
sets that can map a set of inputs to the target domain, with each of them
mapping to a different image in the target domain. Thus, the adversarial loss
alone cannot guarantee that the learned function maps an input
$(x_{t_{1}},x_{t_{2}},t_{p})$ to the desired output $x_{t_{p}}$.

Transitive consistency loss Since we aim at training the network for generic
frame synthesis at any time ratio, special handling on the training loss is
needed. To further regularize the mapping function, we argue that the learned
mapping function for frame interpolation/extrapolation should be transitive-
consistent: given a mapping

$y_{t_{p}}=G(x_{t_{1}},x_{t_{2}},t_{p}),$ (4)

we should have transitive mappings for $x_{t_{1}}$ and $x_{t_{2}}$ giving
$y_{t_{p}}$ as an input,

$\displaystyle G(y_{t_{p}},x_{t_{2}},t_{1})$ $\displaystyle\approx$
$\displaystyle x_{t_{1}},$ $\displaystyle G(x_{t_{1}},y_{t_{p}},t_{2})$
$\displaystyle\approx$ $\displaystyle x_{t_{2}}.$ (5)

We call this transitive consistency, and an illustration is shown in Figure 2.
Therefore, we can formulate this behavior using a transitive consistency loss:

$\displaystyle\mathcal{L}_{tran}(G)$ $\displaystyle=$
$\displaystyle\mathbb{E}_{p_{data}}[|G(x_{t_{1}},y_{t_{p}},t_{2})-x_{t_{2}}|_{1}]$
(6) $\displaystyle+$
$\displaystyle\mathbb{E}_{p_{data}}[|G(y_{t_{p}},x_{t_{2}},t_{1})-x_{t_{1}}|_{1}].$

In preliminary experiments, we also tried replacing the $\ell_{1}$ norm in
this loss with an $\ell_{2}$, but did not observe improved performance.

Another way to formulate transitive consistency is directly using the observed
image $x_{t_{p}}$ instead of predicted image $y_{t_{p}}$ in (6), as

$\displaystyle\mathcal{L}_{tran}(G)$ $\displaystyle=$
$\displaystyle\mathbb{E}_{p_{data}}[|G(x_{t_{1}},x_{t_{p}},t_{2})-x_{t_{2}}|_{1}]$
(7) $\displaystyle+$
$\displaystyle\mathbb{E}_{p_{data}}[|G(x_{t_{p}},x_{t_{2}},t_{1})-x_{t_{1}}|_{1}].$

This is similar to have all the triplet permutation in the same training
batch, which is enforcing structured input for training.

Objective function By combining adversarial loss and transitive consistency
loss, our final objective function becomes:

$\displaystyle\mathcal{L}(G,D)=\mathcal{L}_{pix}(G)+\lambda_{feat}\mathcal{L}_{feat}(G)+$
(8)
$\displaystyle\lambda_{GAN}\mathcal{L}_{GAN}(G,D)+\lambda_{tran}\mathcal{L}_{tran}(G),$

where $\lambda_{\ast}$ are the weights to balance different components. Thus,
our goal is to solve:

$(G^{\star},D^{\star})=\arg\min_{G}\max_{D}\mathcal{L}(G,D).$ (9)

In Section 4, we compare our method of the full objective, against ablations
of the transitive consistency loss $\mathcal{L}_{tran}$ and its alternatives,
and show that transitive consistency loss play a critical role in obtaining
high-quality results.

###  3.3 Implementation and Training Details

In practice, instead of feeding the time stamp $t_{p}$ directly for mapping
$G(x_{t_{1}},x_{t_{2}},t_{p})$, we feed a ratio
$r_{p}(t_{1},t_{2})=(t_{p}-t_{1})/(t_{2}-t_{1})$ representing the relative
temporal position comparing to $t_{1}$ and $t_{2}$. Similarly, temporal ratios
$(t_{1}-t_{p})/(t_{2}-t_{p})$ and $(t_{2}-t_{1})/(t_{p}-t_{1})$ are used for
mappings $G(y_{t_{p}},x_{t_{2}},t_{1})$ and $G(x_{t_{1}},y_{t_{p}},t_{2})$
respectively. The ratio is formulated as a channel of the input image size and
concatenated to the input images.

In the proposed MSFSN, all convolutional layers have 64 filters with the size
of $5\times 5$, and they are initialized using the Xavier initialization [11].
We use the pixel-shuffle technique [36] in the upsampling layer. We use the
leaky rectified linear units (LeakyReLUs) [25] with a negative slope of 0.2 as
the non-linear activation function. For loss function, we set
$\lambda_{feat}=2*10^{-5}$, $\lambda_{GAN}=5*10^{-2}$ to balance with the
pixel reconstruction loss $\mathcal{L}_{pix}$ according to [21]. The parameter
$\lambda_{tran}$ is set to be $0.2$, and we determine the parameter via a
coarse-to-fine cross-validation on a small validation dataset, with respect to
the reconstruction accuracy and convergence speed.

The networks are trained using Adam optimization [19] with $\beta_{1}=0.9$ and
$\beta_{2}=0.999$. We use a batch size of 8, a patch size of 128 and 200
iterations per epoch for training. At each pass, three nearby frames are
sampled from the videos with randomly selected intervals. To guarantee that
each training data contains enough information, we rule out smooth patches
based on variance. We use a learning rate of $1*10^{-4}$ for the initial
generator training and decay it during the adversarial training until the
network converges. All our networks are trained on a single Nvidia P40 GPU.

We include various types of data augmentation during training: 1) randomly
rotate images by {0, 90, 180, 270} degrees; 2) randomly flip images
horizontally or vertically; 3) randomly crop patches of the same resolution of
the training input; 4) adding additive Gaussian noise sampled uniformly from
$N(0,0.1)$. During test, images are padded with mirror reflection such that
their sizes are multiplies of $2^{S-1}$.

##  4 Model Analysis

In this section, we first validate the contributions of different components
of the proposed network at an interpolation ratio 0.5 (interpolating middle
in-between frame). We then discuss the effect of multiple in-between frame
interpolation. For validation experiments, we use the GOPRO dataset [30], in
which the videos are captured using a GOPRO camera at 240 fps. Since the
videos are captured under significant camera movement, object motion and
illumination change at a high sampling rate, the dataset well fits our
purpose. We split GOPRO dataset into training/test sets (2103/1111 frames) and
train our model on the training set while evaluating on test set.

Adversarial training To validate the influence of the adversarial training, we
train the proposed model with only generator (Ours-Gen) and compare it with
the original model with the adversarial discriminator. We test this using the
setting described in Section 5 and show results in Table 4. The proposed model
does not outperform Ours-Gen in PSNR, but is able to render visually pleasing
images. As shown in Figure 3, our proposed method generate sharper images but
the corresponding PSNR is slightly lower than those from Ours-Gen. We note
that our method removes some artifacts existed in the input images, e.g., the
blocky artifacts in the example, thus is at a disadvantage in quantitative
comparison.

|  |
---|---|---
|  |
(a) Ground truth | (b) Ours-Gen | (c) Ours
Figure 3: Example of frame interpolation on UCF-101 dataset [38]. We compare
the proposed network with that trained without adversarial training (Ours-
Gen).

Pyramid depth Since the sub-networks in our model share same parameters, we
can easily apply different numbers of pyramid levels during test with one
trained model. We train our model of 4 pyramid levels on GOPRO training set
and test the performance of different numbers of pyramid levels on GOPRO test
dataset. Moreover, we evaluate the results on different intervals: 1/2/3 frame
intervals, meaning that the temporal distances between the interpolated frames
and the pairs of input frames are 1/2/3 frames. The quantitative results are
shown in Table 1. As shown in the table, stacking more levels in general leads
to better performance (exception is 4-level structure with which the model is
trained). This is because long-range motion can be better handled by delving
into lower downsampled images.

Pyramid depth/ Time interval | GOPRO (PSNR/SSIM)
---|---
1 frame | 2 frames | 3 frames
3 levels | 32.56/0.90 | 31.98/0.88 | 31.39/0.81
4 levels | 34.76/0.91 | 34.24/0.90 | 33.56/0.85
5 levels | 34.25/0.91 | 33.71/0.90 | 33.04/0.84
6 levels | 34.58/0.91 | 34.04/0.90 | 33.35/0.84
Table 1: Quantitative evaluation on the number of pyramid levels. We train the model of 4 pyramid levels on GOPRO training set [30] and evaluate the performance of different numbers of pyramid levels on GOPRO test set. The results are evaluated for frame interpolation at 1/2/3 frame intervals. | #Params | Size (MB)
---|---|---
DVF | 13,413,440 | 53.7
AdapSC | 21,667,716 | 86.7
Ours (12) | 9,878,617 | 39.5
Ours (9) | 7,419,481 | 29.7
Table 2: Comparisons on parameter number and model size. We compare the
proposed method of sub-network depth 9 and 12 with DVF [24] and AdapSC [32].

Sub-network depth Each of the sub-network contains $D$ residual blocks and we
explore the performance influence on the sub-network depth, i.e., number of
residual blocks. We train the proposed model of 4 pyramid levels with
different depth, $D=5,9,12$ at each level, and show the performance on
interpolation task in Table 3. The models are trained on GOPRO training set
and evaluated on GOPRO test dataset.

In general, deeper networks perform better than shallower ones at the expense
of increased training time and computational cost. We use $D=9$ in our model
for most of our experiments to compromise between accuracy and speed.

Network depth/ Time interval | #Parameters | GOPRO (PSNR/SSIM)
---|---|---
1 frame | 2 frames | 3 frames
5 | 4,140,633 | 34.43/0.90 | 33.82/0.88 | 33.13/0.81
9 | 7,419,481 | 34.76/0.91 | 34.24/0.90 | 33.56/0.85
12 | 9,878,617 | 34.80/0.91 | 34.27/0.90 | 33.65/0.85

Table 3: Quantitative evaluation on the sub-network depth at each level. We
build MSFSN with different sub-network depth by varying the numbers of
residual blocks. The results are evaluated for frame interpolation task on the
UCF-101 dataset [38].

Transitive consistency loss. To validate the effectiveness of the transitive
consistency loss, we compare with the proposed network with alternatives, and
show the training curve in terms of PSNR metric instead of loss curve, as the
loss function has been changed when removing transitive consistency loss. We
conduct experiments by comparing following alternatives on a validation
dataset under 1200 epochs (evaluate every 10 epochs): 1) no transitive
consistency loss $\mathcal{L}_{tran}$, but increasing the weight for pixel
reconstruction loss to 1.4, to match the loss level when using transitive
consistency loss; 2) using transitive consistency loss in (6); 3) using
transitive consistency loss in (7); 4) using a temporal total variation (TV)
loss

$\mathbb{E}_{p_{data}}[|G(x_{t_{1}},x_{t_{2}},t_{p})-x_{t_{1}}|_{1}]+\mathbb{E}_{p_{data}}[|G(x_{t_{1}},x_{t_{2}},t_{p})-x_{t_{2}}|_{1}],$
(10)

instead of $\mathcal{L}_{tran}$; 5) using a weighted temporal TV loss based on
the temporal distance as

$\displaystyle\frac{2*|t_{2}-t_{p}|}{|t_{1}-t_{p}|+|t_{2}-t_{p}|}\mathbb{E}_{p_{data}}[|G(x_{t_{1}},x_{t_{2}},t_{p})-x_{t_{1}}|_{1}]+$
(11)
$\displaystyle\frac{2*|t_{1}-t_{p}|}{|t_{1}-t_{p}|+|t_{2}-t_{p}|}\mathbb{E}_{p_{data}}[|G(x_{t_{1}},x_{t_{2}},t_{p})-x_{t_{2}}|_{1}],$

to better fit generalized synthesis tasks. We adopt the same settings for the
comparison, thus each alternative is viewing the same amount of data at an
epoch. We note that each epoch using transitive consistency loss, 2) or 3),
takes around 2x computational time comparing to those without using transitive
properties. As shown in Figure 4(a), the methods by enforcing transitive
properties, both using 2) and 3), converge to better solutions comparing to
those without transitive properties. The network using transitive properties
(7) performs slightly better in convergence, and we use it for the experiments
in Section 5. Without transitive loss, the images and time ratio of frame
synthesis in each batch are randomly selected and could easily bias the
training when the batch number is small and constrained by the memory size.
Thus, the method using 1) oscillates heavily at the beginning and does not
converge to a good minimum. The network with temporal TV does not perform well
as those using transitive properties. The reason could be that ground-truth
frames are not favored by temporal TV in these tasks, especially when large
appearance variation happens in videos. We note that it is possible to apply
this loss to other frame interpolation/extrapolation networks to enable them
for generic synthesis tasks.

|
---|---
(a) | (b)
Figure 4: (a) Training curves of transitive consistency loss and its
alternatives in terms of PSNR value. (b) User study results on frame
interpolation. We compare our method with state-of-the-art methods, FlowNet2
[17], EpicFlow [34], DVF [24], AdapSC [32] and the ground truth (GT).

##  5 Experimental Results

In this section, we compare the proposed MSFSN with several state-of-the-art
methods on benchmark datasets. We present the quantitative and qualitative
comparison in terms of interpolation and extrapolation. Finally, we discuss
the limitation of the proposed method.

###  5.1 Interpolation

We compare our approach against several methods, including optical flow
techniques: EpicFlow [34] and FlowNet2 [17]; and frame synthesis methods:
BeyondMSE [27], DVF [24] and AdapSC [32]. We carry out extensive experiments
on public benchmark datasets: UCF-101 [38] and THUMOS-15 [16]. UCF-101 and
THUMOS-15 contain videos with object motions in relatively low resolution. To
synthesize the interpolated images given the estimated flow fields, we apply
the interpolation algorithm used in the Middlebury interpolation benchmark
[1].

As shown in Figure 6, the correspondence-based methods would generate images
with ringing or blurry artifact on the regions where correspondence estimation
fails, e.g., front of motor boat, while our network is able to render pleasing
results. We present quantitative comparisons on the benchmark datasets in
Table 4. Our approach performs better than flow-based methods EpicFlow [34]
and FlowNet2 [17], and comparable to frame synthesis network DVF [24]. The
method AdapSC [32] usually generate sharper results than our results, due to
their correspondence nature via local filtering. That is, when the
correspondences/spatially-varying kernels are accurately estimated, they can
generate sharp results as the input images.

Method | UCF-101 | THUMOS-15
---|---|---
PSNR | SSIM | PSNR | SSIM
BeyondMSE | 32.8 | 0.93 | 32.3 | 0.91
EpicFlow | 34.2 | 0.95 | 33.9 | 0.94
FlowNet2 | 34.0 | 0.94 | 33.8 | 0.94
DVF | 35.8 | 0.95 | 35.4 | 0.95
AdapSC | 36.2 | 0.95 | 36.4 | 0.96
Ours-Gen | 36.0 | 0.95 | 35.5 | 0.95
Ours | 35.8 | 0.95 | 35.2 | 0.95
Table 4: Performance (PSNR and SSIM) of video frame interpolation on UCF-101 and THUMOS-15 datasets.  Method | UCF-101 | THUMOS-15
---|---|---
PSNR | SSIM | PSNR | SSIM
BeyondMSE | 30.6 | 0.90 | 30.2 | 0.89
EpicFlow | 31.3 | 0.92 | 31.0 | 0.92
FlowNet2 | 31.8 | 0.92 | 31.7 | 0.92
DVF | 32.7 | 0.93 | 32.2 | 0.92
Ours-Gen | 32.8 | 0.93 | 32.2 | 0.92
Ours | 32.4 | 0.93 | 31.9 | 0.92
Table 5: Performance (PSNR and SSIM) of video frame prediction on UCF-101 and
THUMOS-15 datasets.

Model size We include comparisons with other frame-interpolation networks on
the number of parameters in Table 2. Unlike the traditional encoder-decoder
structure, the proposed network enables parameter sharing as each sub-network
level share the same purpose. The model compression would help reducing memory
for model storage during inference, without losing much accuracies, and it
could be useful for smartphone applications and edge-device processing in IoT
applications.

Interpolating multiple frames Unlike those networks trained on specific
interpolation settings, e.g., interpolating middle in-between frame, our
method is capable of synthesizing frames at any in-between position, without
computing correspondences. We show comparison with flow-based method [17] and
frame interpolation network [32] on interpolating multiple in-between frames.
To compare with the network [32] which can only interpolate middle in-between
frame, we evaluate on the scenario of interpolating three equally spaced
frames. In this case, we obtain their results by two-stage interpolation,
which is first interpolating middle frame and then interpolating the other two
using the middle frame as an input. As shown in Figure 5, our method renders
sharp results while maintains the straight road lines while others cannot.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
(a) First input frame | (b) Interpolated frame 1 | (c) Interpolated frame 3 | (d) Second input frame
Figure 5: Example of multiple frame interpolation from KITTI dataset [28]. For
this example, we interpolate three equally spaced frames in-between and show
comparison with FlowNet2 [17] and AdapSC [32]. The first to third rows are
results from FlowNet2, AdapSC and ours respectively. Here we show first and
third interpolated frames for comparison.

User study To better understand the visual quality of different methods, we
conducted a user study on the interpolated frames, comparing with state-of-
the-art methods, FlowNet2 [17], EpicFlow [34], DVF [24], AdapSC [32] and the
ground truth (GT). We develop a web-based system to display and collect study
results. The system provides two side-by-side results at a time, one from our
method and the other from a randomly selected method. Each pair of results is
randomly selected and placed. There are 42 subjects participated in the user
study and each of them was asked to select on 20 pairs of comparison. As shown
in Figure 4(b), our method is preferred over FlowNet2, EpicFlow and DVF, and
obtain comparable results comparing with AdapSC.

###  5.2 Extrapolation

The trained model can be directly applied to video extrapolation tasks without
fine tuning. Most of the interpolation networks mentioned are trained for a
specific setting, and therefore cannot apply directly to the extrapolation
cases. We compare with state-of-the-art methods on UC-101 and THUMOS-15
datasets and present quantitative results in Table 5. Qualitative comparisons
are given in Figure 7.

|  |
---|---|---
|  |
Ground truth | EpicFlow | FlowNet2
|  |
|  |
DVF | AdapSC | Ours
|  |
|  |
Ground truth | EpicFlow | FlowNet2
|  |
|  |
DVF | AdapSC | Ours
Figure 6: Examples of video frame interpolation from UCF-101 dataset [38]. We compare with several state-of-the-art methods, including EpicFlow [34], FlowNet2 [17], DVF [24] and AdapSC [32].  |  |
---|---|---
|  |
Input frame 1 | Input frame 2 | Ground truth
|  |
|  |
| FlowNet2 | Ours
|  |
|  |
Input frame 1 | Input frame 2 | Ground truth
|  |
|  |
| FlowNet2 | Ours
Figure 7: Examples of video frame extrapolation from UCF-101 dataset [38]. We
compare with FlowNet2 [17].

###  5.3 Limitation and Discussion

While our network is capable to handle large motion, the results from our
method are usually more blurry than than corresponding-based method [32], as
they fuse correponding pixels through a local window filtering. We will
explore the framework to combine the benefit of corresponding-based method
into the unified framework in our future work. Another limitation of the
proposed network is that the synthesized frame is assumed to follow the motion
momentum, which is constrained and parameterized using one temporal variable.
How to generalize the motion patterns in the network will also be an
interesting research direction.

##  6 Conclusion

In this paper, we propose a unified deep neural network for video frame
synthesis. The proposed model progressively predicts interpolated/extrapolated
frames in a coarse-to-fine manner. We introduce a transitive consistency loss
to facilitate the network training and enable the network for both
interpolation and extrapolation capabilities. By sharing parameters across
pyramid levels, the network is compact and is practical to use in devices
where memory and computation power are limited. The proposed model can be
easily extend to scenarios, e.g., long-range motion, where deeper pyramid is
needed. Extensive evaluations on benchmark datasets demonstrate that the
proposed model performs favorably against state-of-the-art algorithms.

## References

  * [1] S. Baker, D. Scharstein, J. Lewis, S. Roth, M. J. Black, and R. Szeliski.  A database and evaluation methodology for optical flow.  International Journal Computer Vision, 92(1):1–31, 2011.
  * [2] R. W. Brislin.  Back-translation for cross-cultural research.  Journal of cross-cultural psychology, 1(3):185–216, 1970.
  * [3] J. Bruna, P. Sprechmann, and Y. LeCun.  Super-resolution with deep convolutional sufficient statistics.  arXiv preprint arXiv:1511.05666, 2015.
  * [4] H. C. Burger, C. J. Schuler, and S. Harmeling.  Image denoising: Can plain neural networks compete with BM3D.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2392–2399. IEEE, 2012.
  * [5] B. De Brabandere, X. Jia, T. Tuytelaars, and L. Van Gool.  Dynamic filter networks.  In Neural Information Processing Systems, 2016.
  * [6] E. L. Denton, S. Chintala, R. Fergus, et al.  Deep generative image models using a laplacian pyramid of adversarial networks.  In Neural Information Processing Systems, pages 1486–1494, 2015\.
  * [7] C. Dong, C. C. Loy, K. He, and X. Tang.  Image super-resolution using deep convolutional networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295–307, 2016.
  * [8] A. Dosovitskiy and T. Brox.  Generating images with perceptual similarity metrics based on deep networks.  In Neural Information Processing Systems, pages 658–666, 2016.
  * [9] C. Finn, I. Goodfellow, and S. Levine.  Unsupervised learning for physical interaction through video prediction.  In Neural Information Processing Systems, pages 64–72, 2016.
  * [10] L. A. Gatys, A. S. Ecker, and M. Bethge.  A neural algorithm of artistic style.  arXiv preprint arXiv:1508.06576, 2015.
  * [11] X. Glorot and Y. Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010.
  * [12] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Neural Information Processing Systems, pages 2672–2680, 2014\.
  * [13] D. He, Y. Xia, T. Qin, L. Wang, N. Yu, T. Liu, and W.-Y. Ma.  Dual learning for machine translation.  In Neural Information Processing Systems, pages 820–828, 2016.
  * [14] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [15] H. Hirschmuller.  Stereo processing by semiglobal matching and mutual information.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):328–341, 2008.
  * [16] H. Idrees, A. R. Zamir, Y.-G. Jiang, A. Gorban, I. Laptev, R. Sukthankar, and M. Shah.  The THUMOS challenge on action recognition for videos in the wild.  Computer Vision and Image Understanding, 155:1–23, 2017.
  * [17] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  arXiv preprint arXiv:1612.01925, 2016.
  * [18] J. Johnson, A. Alahi, and L. Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In Proceedings of European Conference on Computer Vision, pages 694–711. Springer, 2016.
  * [19] D. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [20] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang.  Deep laplacian pyramid networks for fast and accurate super-resolution.  arXiv preprint arXiv:1704.03915, 2017.
  * [21] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  arXiv preprint arXiv:1609.04802, 2016.
  * [22] C. Li and M. Wand.  Combining markov random fields and convolutional neural networks for image synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2479–2486, 2016.
  * [23] C. Li and M. Wand.  Precomputed real-time texture synthesis with markovian generative adversarial networks.  In Proceedings of European Conference on Computer Vision, pages 702–716. Springer, 2016.
  * [24] Z. Liu, R. Yeh, X. Tang, Y. Liu, and A. Agarwala.  Video frame synthesis using deep voxel flow.  arXiv preprint arXiv:1702.02463, 2017.
  * [25] A. L. Maas, A. Y. Hannun, and A. Y. Ng.  Rectifier nonlinearities improve neural network acoustic models.  In Proceedings of International Conference on Machine Learning, volume 30, 2013.
  * [26] D. Mahajan, F.-C. Huang, W. Matusik, R. Ramamoorthi, and P. Belhumeur.  Moving gradients: a path-based method for plausible image interpolation.  ACM Transactions on Graphics (TOG), 28(3):42, 2009.
  * [27] M. Mathieu, C. Couprie, and Y. LeCun.  Deep multi-scale video prediction beyond mean square error.  arXiv preprint arXiv:1511.05440, 2015.
  * [28] M. Menze and A. Geiger.  Object scene flow for autonomous vehicles.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [29] S. Meyer, O. Wang, H. Zimmer, M. Grosse, and A. Sorkine-Hornung.  Phase-based frame interpolation for video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1410–1418, 2015.
  * [30] S. Nah, T. H. Kim, and K. M. Lee.  Deep multi-scale convolutional neural network for dynamic scene deblurring.  arXiv preprint arXiv:1612.02177, 2016.
  * [31] S. Niklaus, L. Mai, and F. Liu.  Video frame interpolation via adaptive convolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [32] S. Niklaus, L. Mai, and F. Liu.  Video frame interpolation via adaptive separable convolution.  In Proceedings of IEEE International Conference on Computer Vision, 2017.
  * [33] A. Ranjan and M. J. Black.  Optical flow estimation using a spatial pyramid network.  arXiv preprint arXiv:1611.00850, 2016.
  * [34] J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid.  Epicflow: Edge-preserving interpolation of correspondences for optical flow.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1164–1172, 2015.
  * [35] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  International Journal Computer Vision, 115(3):211–252, 2015.
  * [36] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1874–1883, 2016.
  * [37] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [38] K. Soomro, A. R. Zamir, and M. Shah.  UCF101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [39] N. Srivastava, E. Mansimov, and R. Salakhudinov.  Unsupervised learning of video representations using lstms.  In Proceedings of International Conference on Machine Learning, pages 843–852, 2015.
  * [40] N. Sundaram, T. Brox, and K. Keutzer.  Dense point trajectories by gpu-accelerated large displacement optical flow.  In Proceedings of European Conference on Computer Vision, pages 438–451. Springer, 2010.
  * [41] M. Werlberger, T. Pock, M. Unger, and H. Bischof.  Optical flow guided tv-l1 video interpolation and restoration.  In EMMCVPR, pages 273–286. Springer, 2011.
  * [42] T. Xue, J. Wu, K. L. Bouman, and W. T. Freeman.  Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks.  In Neural Information Processing Systems, 2016.
  * [43] X. Yu and F. Porikli.  Ultra-resolving face images by discriminative generative networks.  In Proceedings of European Conference on Computer Vision, pages 318–333. Springer, 2016.
  * [44] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  arXiv preprint arXiv:1703.10593, 2017.
  * [45] C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski.  High-quality video view interpolation using a layered representation.  In ACM Transactions on Graphics (TOG), volume 23, pages 600–608. ACM, 2004.