# NIO: Lightweight neural operator-based architecture for video frame
interpolation

Hrishikesh Viswanath
<EMAIL_ADDRESS>Md Ashiqur Rahman
<EMAIL_ADDRESS>Rashmi Bhaskara
<EMAIL_ADDRESS>Aniket Bera
<EMAIL_ADDRESS>
###### Abstract

We present, NIO \- Neural Interpolation Operator, a lightweight efficient
neural operator-based architecture to perform video frame interpolation.
Current deep learning-based methods rely on local convolutions for feature
learning and require a large amount of training on comprehensive datasets.
Furthermore, transformer-based architectures are large and need dedicated GPUs
for training. On the other hand, NIO, our neural operator-based approach
learns the features in the frames by translating the image matrix into the
Fourier space by using Fast Fourier Transform (FFT). The model performs global
convolution, making it discretization invariant. We show that NIO can produce
visually-smooth and accurate results and converges in fewer epochs than state-
of-the-art approaches. To evaluate the visual quality of our interpolated
frames, we calculate the structural similarity index (SSIM) and Peak Signal to
Noise Ratio (PSNR) between the generated frame and the ground truth frame. We
provide the quantitative performance of our model on Vimeo-90K dataset, DAVIS,
UCF101 and DISFA+ dataset.

##  1 Introduction

Figure 1: Comparison of Performance of Deep Learning based models on Vimeo90K
dataset against the number of epochs. The proposed NIO model achieves
comparable accuracy within just 10 epochs.

Video frame interpolation is the process of creating several in-between frames
from the set of available frames. This is a challenging problem as it involves
understanding the geometric structures of images and predicting the positions
of multiple objects within an image while taking into account the varying
velocities of the objects and the time step of the frames. In addition, a
video interpolation system should be easy to run on commonly used devices and
be able to operate on edge hardware and low-power/compute devices.

Many applications make use of video frame interpolation, such as apps that
generate movies and panoramic views from visually similar frames and
applications that run on the network edge that may need to recover lost frames
due to network issues, restore broken videos [5, 26, 20]. Recent works focus
on increasing the video frame rate to improve gaming and real-time video
streaming experience. The other applications include medical imaging [1, 18],
restoring compressed videos [11] and generating virtual frames in 3D spaces
[47, 50]. Most of these applications, especially video streaming applications,
and gaming environments need the interpolation algorithm running on the
network edge. Several cutting-edge models have been developed, and they
produce interpolated frames that, on average, have a structural similarity of
98$\%$ with the expected output. However, these models require long duration
of training on large datasets, and is therefore difficult to train them on
edge devices. In this paper, we present a fast, lightweight neural operator-
based architecture for interpolating video frames.

Current deep learning-based interpolation models are generally variations of
Convolutional Neural Networks [36, 34, 6], which construct the missing frames
by extracting the semantics and structural information present in the input
images using the appropriate filters or kernels. These strategies are
ineffective in capturing the position and orientations of objects within the
image. Moreover, Convolution neural network-based models rely on local
convolution for feature learning and require large amounts of training data
and take a long time to converge. Furthermore, convolution is a costly
operation (O($n^{3}$)). There have been attempts to solve this issue, for
example, video interpolation using cyclic frame generation technique [29], but
is not very accurate.

Optical Flow based techniques [23], which capture the motion of objects have
also been applied for frame interpolation. In this technique, the apparent
motion of the pixels is captured and labeled as a flow vector. Using the
estimated values of these flow vectors for each pixel, a missing frame can be
generated. The limitations imposed by inadequate kernel sizes in CNN-based
methods are resolved by flow-based techniques and the frames can also be
generated at a higher frequency per second, resulting in a smoother video.
Optical flow-based methods fail while dealing with noisy frames due to the
lack of necessary pixel information. It has also been observed that a
combination of both kernel-based and optical flow-based methods [7] with Deep
Learning techniques like Transformers [46], and GANs, have low frame
interpolation errors and also provide a great frame rate for a smooth video.
However, GAN-based architectures suffer from modal collapse and thus cannot be
generalized in an arbitrary fashion. They rely heavily on the distribution of
input and would require re-training for a new distribution if the input space
is changed. Transformer-based architectures have been shown to be very
efficient [46]. However, due to their massively complex architecture, they
require a lot of computing power and powerful GPUs to train. Long training
times are an additional downside.

In this paper, we define the problem of video interpolation from a physics
perspective. The problem can be defined as predicting the trajectories of
objects, each moving with a different velocity in continuous space, similar to
optical flow, but we present a way to capture the flow information efficiently
using kernels. This problem is similar to predicting the trajectory of wind
currents or ocean currents, for which neural operators [22, 10, 42] have been
shown to be exceptionally efficient, beating state-of-the-art weather
prediction models [40]. We present a lightweight Fourier Neural Operator-based
architecture - NIO - Neural Interpolation Operator and show empirically that
with less training (5-10 epochs), the model achieves state-of-the-art results.
We quantify the quality of the results using the Peak Signal to Noise Ratio
(PSNR) and the Structural Similarity Index and show that the SSIM of the
generated frames is structurally similar to the ground truth.

The NIO network has a sequence of spectral-convolution layers, which perform
convolution after translating the input to the Fourier space. The low-
frequency regions are retained and the high-frequency ones are discarded
because high-frequency points are specific to the particular input and may
lead to overfitting as shown in [25]. The spectral-convolution structure of
the NIO network is similar to an autoencoder network where the encoder layers
contract the input space to capture the key information about the distribution
of the input. The decoder expands it back to its original input space.

In this work we have three main contributions:

  * •

We present a lightweight efficient neural operator-based architecture to
perform video frame interpolation whose performance is comparable to state-of-
the-art interpolation models. To the best of our knowledge, NIO is the first
to propose a spatially-continuous architecture to solve this problem.

  * •

We show that this model takes fewer epochs to converge than other CNN and GAN-
based architectures of comparable size (5-10 epochs). We show that it has a
better time complexity per convolution $(O(n\log n))$.

  * •

We prove that the model is resolution invariant and can be trained on lower
resolution images but can be applied to higher resolution images.

  * •

Lastly, we show that NIO can generalize well on the unseen data by testing it
on DAVIS-90 [3], UCS101 [30] and DISFA+ datasets. [32].

##  2 Literature Review

Existing video interpolation methods focus on creating techniques that can
accurately capture an object’s motion while accounting for occlusions and
blurry frames. To deal with blurry frames, the pyramid module proposed in [44]
successfully restores the inputs and synthesizes the interpolated frame. Many
other methods focus on estimating the bidirectional optical flow in the input
frames [27, 23, 13, 43, 35] and these methods are usually trained using Deep
Learning models like Neural Networks [15], Generative Adversarial Networks
[49], Long short-term memory (LSTM) [12], Encoder-Decoder [4, 39]. However,
they fail to generate smooth interpolation of the frames when dealing with
large motions and occlusion in the input frames.

Bao et al. [2] introduced Depth Aware Flow projection layer that uses depth
maps in combination with optical flow to suppress the effect of occluded
objects. In addition to bidirectional optical flow estimation, [34] uses
pixel-wise contextual information to generate high-quality intermediate frames
and also effectively handles occlusion. To capture non-linear motion among
frames a bidirectional pseudo-three-dimensional warping layer is used in [31]
that uses motion estimation and depth-related occlusion estimation to learn
the interpolation. A ST-MFNet [8] uses 3-D CNNs, and Spatio-Temporal GANs to
estimate intermediate flows to capture large and complex motions.

The optical-flow estimating models are accurate, but their calculations are
expensive and their designs complicated. An alternate technique was kernel-
based, [45, 41, 48, 29], that uses filters to learn features from the input
frames in order to synthesize an intermediate frame. These models are end-to-
end trainable but fail to capture motion and pixel information beyond the
kernel size. To overcome these limitations and also to handle other major
issues like occlusion Choi et al. [7] proposed an architecture that uses a
layer called PixelShuffle combined with channel attention that is comprised of
Convolutional layers and Relu. The PixelShuffle layer in [7] downsizes the
input frames to capture relevant feature information and upscales the feature
maps in later stages to generate the missing frame and is a replacement for
flow-estimation networks. A similar model that uses transformers has been
proposed in [19], which uses a visual transformer module that analyzes input
frame features to refine the interpolated prediction and also introduces an
image attention module to deal with occlusion. To avoid the additional
computation overheads of having an image attention module, [46] uses
transformers along with local attention which is extended to a spatial
temporal domain to compute the interpolation.

Figure 2: Abstract view of the NIO Model. The CONV represents the
interpolation layer that applies the shared weight to the input frames and
creates a hybrid input to the UNO. The second section represents the
SpectralConv2d layers of the UNO model. The Fourier modes are progressively
reduced from 24 to 4 as the frame gets downsampled.

To overcome the issue of restricted kernel size in kernel-based methods [36]
introduces Adaptive Separable Convolution that uses deep convolutional neural
networks and runs pairs of 1-D kernels on the input frames to capture large
motions. In addition to kernel-based, there are phase-based methods like [33]
that use per-pixel phase modification to compute a smooth interpolated frame
without having to perform any flow estimation.

The above-discussed methods show exceptional results on well-known datasets
like Vimeo-90K, but a major drawback is that these models take too long to
converge. For example, DAIN [2] takes 5+ days to converge on an NVIDIA Titan X
(Pascal) GPU for around 40 epochs and a batch size of 2; CAIN [7] is trained
for 200 epochs with a batch size of 32 and takes around 4 days to converge on
a single Titan Xp GPU. In addition to the long training time, these models are
responsible for tuning too many parameters, for example, 24 million for DAIN,
and 42.8 million for CAIN. Some prior works including [9] attempt to reduce
the number of parameters and generate a lightweight-driven network. The
architecture proposed in [9] uses bidirectional encoding (BE) with channel
attention and low dimensional channels, which results in a reduction in the
number of parameters for tuning. However, the model has to tune 7.05 million
parameters, whereas our proposed NIO model tunes 3.43 million parameters.

##  3 Our Approach

The proposed model NIO takes as input a couplet of frames, $I_{0}$ and
$I_{1}$, and generates one intermediate frame $I_{0.5}$. If $I^{\prime}_{0.5}$
is the ground truth and $\mathcal{N}$ is an operator that represents the NIO
architecture then,

$I_{0.5}=\mathcal{N}(I_{0},I_{1})$ (1)

The interpolated frame quality is measured as PSNR( $I^{\prime}_{0.5}$,
$I_{0.5}$ ) and SSIM( $I^{\prime}_{0.5}$, $I_{0.5}$ ).

The model contains two components - The interpolation layer which performs a
linear operation on the two frames and couples them together with a common
weight matrix. The second component is the UNO network, as proposed in Rahman
et al [42], which is a neural operator network. It performs convolution
operations in the Fourier Space. Our model, NIO, initially contracts the input
space to extract the key features of the image, which is equivalent to
performing a dimensionality reduction. The encoder is followed by the decoder
layers which expand space back to its original size. The input takes two
channels, with each channel corresponding to an input frame while the output
has a single channel corresponding to the output frame.

###  3.1 NIO Architecture

The model initially takes as input multiple frames ($I_{0}$ and $I_{1}$) and
recursively applies the convolution kernel with a single weight matrix to
generate a hybrid frame and projects this into Fourier Space using FFT. The
layer only preserves low-frequency Fourier models and ignores high-frequency
modes, which are too specific to the particular input and if these modes are
learned, they overfit to the input as shown in Li et al.[25]. After applying
the weights, the tensors are projected back into the spatial domain and non-
linear activation is applied. The primary advantage of Fourier layers is that
they are discretization invariant. This means that they can learn function
mappings that are discretized in arbitrary ways since the parameters are
learned in Fourier space. Another advantage of the Fourier Transform is that
it is fast. Convolution when applied in Fourier space is quasi-linear [25],
therefore learning weights is significantly faster in this space as opposed to
the spatial domain. The convolution becomes point-wise multiplication in the
Fourier space. This entire process of applying the Fast Fourier Transform,
followed by the linear transformation and the inverse Fourier Transform is a
single global convolution. It is represented as follows

$(K(a;\phi)v_{t})(x)=\mathcal{F}^{-1}(\mathcal{F}(K_{\phi}).\mathcal{F}(v_{t}))(x)$
(2)

The $K$ represents the Kernel operator parameterized by $\phi$ and
$\mathcal{F}$ and $\mathcal{F}^{-1}$ represent the Fourier Transform and
Inverse Fourier Transform and $x$ is the input. $v_{t}$ is the weight matrix.

####  3.1.1 Frames as continuous functions

The input to the model is a set of frames, which are images. Neural Operators
have been traditionally applied to solve Partial differential equations or
PDEs where the input is the discretization of a continuous vector field. These
frames on the other hand have distinct geometric features, sharp edges and
discontinuities. The continuous element in these frames is the motion of the
constituent object, which is the function that the operator learns. The
operator needs to learn the complex motions of distinct geometric entities in
the given frame.

####  3.1.2 Operator Learning

As proposed in [25], the neural operator learns the mapping between two
infinite dimensional spaces from a finite collection of input-output pairs. In
our case, for frame interpolation, the infinite spaces are the functions
corresponding to the trajectories of the constituent objects while the input-
output pairs are the triplets representing micro-movement samples.

Let $\mathcal{A}$ and $\mathcal{U}$ be the two such function spaces. Any frame
$a$ is sampled from $\mathcal{A}$ and the corresponding output frame $u$ is
sampled from $\mathcal{U}$. The neural operator $\mathcal{G}$, parameterized
by theta, learns the mapping between $a$ and $u$ and is an approximation of
the function mapping $\mathcal{G}$ from $\mathcal{A}$ to $\mathcal{U}$. The
operator is a sequence of $\mathcal{G}_{i}$, for each layer
$i\in\\{1,2,...N\\}$, each of which is a linear integral operator followed by
non-linear activation. The kernel operation is given by the following equation

$\mathcal{G}_{i}v_{i}(x)=\sigma(\int\kappa_{i}(x,y)v_{i}d\mu_{i}(y)+W_{i}v_{i}(x))$
(3)

$\kappa_{i}$ is the kernel function, which along with the integral operator,
performs global linear operation. $W$ performs point-wise linear operation.
$\sigma$ is the non-linear activation. Iterative updates are performed as
denoted by the equation below

$v_{t+1}(x)=\sigma(Wv_{t}(x)+(K(a;\phi)v_{t})(x))$ (4)

In the above equation, $\sigma$ is the non-linear activation and $W$ is the
linear transformation and $K$ is the kernel integration. The Kernel
integration operator is replaced by the point-wise multiplication operator in
the Fourier space and the equation is reduced as follows

$(K(a;\phi)v_{t})(x)=\mathcal{F}^{-1}(\mathcal{F}(K_{\phi}).\mathcal{F}(v_{t}))(x)$
(5)

where $\mathcal{F}$ denotes the Fourier Transform, which is given by the
following equation

$(\mathcal{F}f)_{j}(k)=\int_{D}f_{j}(x)e^{-2i\pi\langle x,k\rangle}dx$ (6)

$\mathcal{F}^{-1}$ denotes the inverse Fourier Transform and is given by the
equation

$(\mathcal{F}^{-1}f)_{j}(x)=\int_{D}f_{j}(k)e^{2i\pi\langle x,k\rangle}dk$ (7)

The above equations represent Fourier Transform in continuous space. However,
images are discrete and therefore, Fast Fourier Transforms are applied.

Figure 3: The figure illustrates the resolution invariance of our NIO Model.
The model was trained on downsampled low-resolution 100x100 images, the same
resolution as the leftmost image in (a). The other two images in (a) are
200x171 and 448x256 respectively. The network was never previously trained on
images of those sizes. In row (b), the image on the right is the high-
resolution image predicted by the model. The image in the middle is the
downsampled image, whose resolution is comparable to the training dataset for
the model while the image on the left has been downsampled twice.

The Fast Fourier transform is defined as follows

$(\hat{\mathcal{F}}f)_{l}(k)=\sum_{x_{1}=0}^{s_{1}-1}...\sum_{x_{d}=0}^{s_{d}-1}f_{l}(x_{1}...x_{d})e^{-2i\pi\Sigma_{j=1}^{d}\frac{x_{j}k_{j}}{s_{j}}}$
(8)

Inverse Fast Fourier Transform is given by

$(\hat{\mathcal{F}}^{-1}f)_{l}(x)=\sum_{k_{1}=0}^{s_{1}-1}...\sum_{k_{d}=0}^{s_{d}-1}f_{l}(k_{1}...k_{d})e^{2i\pi\Sigma_{j=1}^{d}\frac{x_{j}k_{j}}{s_{j}}}$
(9)

The convolution operation in the Fourier space with the weight tensor $W$,
then becomes point-wise multiplication and is denoted by the following
equation

$(W(\mathcal{F}v_{t}))_{k,l}=\sum_{j=1}^{d_{v}}W_{k,l,j}(\mathcal{F}v_{t})_{k,j}$
(10)

####  3.1.3 Discretization Invariance and Quasi-linear Complexity

The Fourier layers can learn functions that are discretized in any arbitrary
way since they are learned in the Fourier space. This allows for zero-shot
super-resolution.

The multiplication is done on the lower $K$ Fourier modes, which is less than
$n/2$, where $n$ is the resolution of the image. This has a complexity of
$O(K)$. The Fourier Transform and the inverse Fourier Transform operations are
the costliest operations, each being $O(nlogn)$.

Figure 4: Comparison of Ground Truth and Generated images in Davis-90, DISFA+,
UCF101 and Vimeo90K. Row (a) represents the generated images and row (b)
represents the ground truth.

####  3.1.4 Interpolation Layer

The first layer of the network is the interpolation layer which applies the
linear operator to the individual frames and combines them. This layer
captures and aligns the common features of the two images together. The
intuition behind this is token mixing where an image is considered to be a
continuous sequence of tokens [10]. But in this case, the image itself is not
the token but the trajectory of the objects within the two frames is
continuous and a single set of weights aim to capture this information.

####  3.1.5 UNO Structure

The encoder section has 4 contraction layers, which reduce the input size from
100x100 to 8x8. The number of Fourier modes retained in each layer are 24, 16,
8 and 4 respectively. The decoder section is symmetric and expands from 8x8 to
100x100, with 4, 8, 16 and 24 Fourier modes respectively. As opposed to using
regular convolution to perform downsampling and upsampling, our NIO model uses
spectral convolution, to perform feature extraction and dimensionality
reduction in Fourier space which is followed by GELU activation function.

Let the input be represented as $a(x)$. A point-wise operator, parameterized
by $\theta$ is applied to $a(x)$ initially. The equation is given as follows:

$v_{0}(x)=P_{\theta}(a(x))$ (11)

This point-wise operator is the lifting operator and is represented by the
neural network. A sequence of such operators is applied to reduce the
dimensions of the input. This is denoted as follows

$\mathcal{G}_{i}:\\{v_{i}:\mathcal{D}_{i}\rightarrow\mathbb{R}^{d_{v_{i}}}\\}\rightarrow\\{v_{i+1}:\mathcal{D}_{i+1}\rightarrow\mathbb{R}^{d_{v_{i}+1}}\\}$
(12)

The Decoder layers comprise a sequence of non-linear integral operators.
However, they also include skip connections. The function which is passed to
the operator is the concatenation of $v_{L_{1+i}}$ and $v_{L_{1-i}}$ vectors.
The concatenation is represented by the following equation

$v^{\prime}_{L_{1+i}}(x)=[v^{T}_{L_{1-i}}\oplus v^{T}_{L_{1+i}}]^{T},\forall
x\in\mathcal{D}_{L_{1+i}}$ (13)

The last output is projected using a projection operator $Q$ and the final
output frame $u(x)$ is denoted by the equation

$u(x)=Q_{\theta}(v_{L+1}(x)),\forall x\in\mathcal{D}_{L+1}$ (14)

###  3.2 Frame Generation

The intermediate frame is generated after applying a sequence of spectral
convolution layers that perform global convolution in Fourier space. Let the
input frames be represented as $I_{i}$ and $I_{j}$ and the initial weight
matrix be represented as $V_{f}$. Let the interpolation layer be represented
with $I$. Let the encoder layer be represented as $E$ and the decoder layer be
represented as $D$. Let the weights be represented as $W$ and bias be
represented with $B$. The pipeline is as follows

$I(I_{i},I_{j})=V_{f}*I_{i}+V_{f}*I_{j}+B$ (15)
$E_{i}(x_{i})=\sigma(\mathcal{F}^{-1}(W_{i}\mathcal{F}(E_{i-1}(x_{i-1}))))$
(16)
$D_{i}(x_{i})=\sigma(\mathcal{F}^{-1}(W_{i}\mathcal{F}(D_{i-1}(x_{i-1})+E_{i-1}(x_{i-1}))))$
(17) $I_{0.5}=D(E(I(I_{i},I_{j})))$ (18)

###  3.3 Training

####  3.3.1 Dataset

Our model, NIO, was trained and tested on these three frequently used
benchmark datasets (Vimeo90K, DAVIS, UCF101) as well as one specialty dataset
(DISFA+) which focuses on videos with human faces.

Model | Parameters (M) | Epochs | Vimeo90K | DAVIS | UCF101 | DISFA+
---|---|---|---|---|---|---
|  |  | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
ToFlow [51] | 1.4 | - | 33.73 | 0.968 | - | - | 34.58 | 0.966 | - | -
IFRNET-S [21] | 2.8 | 300 | 35.59 | 0.978 | - | - | 35.28 | 0.969 | 38.85 | 0.961
VFIT-S [46] | 7.5 | 100 | 36.48 | 0.976 | 27.92 | 0.885 | 33.36 | 0.971 | 39.25 | 0.964
SoftSplat [35] | 7.7 | - | 35.76 | 0.972 | 27.42 | 0.878 | 35.39 | 0.952 | 38.33 | 0.954
RIFE [14] | 9.8 | 25 | 35.62 | 0.978 | - | - | 35.28 | 0.969 | 38.84 | 0.961
BMBC [37] | 11.0 | - | 34.76 | 0.965 | 26.42 | 0.868 | 35.15 | 0.968 | - | -
ABME [38] | 18.1 | - | 36.18 | 0.980 | - | - | 35.38 | 0.969 | - | -
SepConv [52] | 21.6 | - | 33.60 | 0.944 | 26.21 | 0.857 | 34.78 | 0.966 | 38.70 | 0.959
AdaCof [24] | 21.8 | 50 | 34.47 | 0.973 | 26.49 | 0.866 | 34.90 | 0.968 | 38.98 | 0.961
DAIN [2] | 24.0 | 40 | 33.35 | 0.945 | 26.12 | 0.870 | 34.99 | 0.968 | 35.00 | 0.956
QVI [28] | 29.2 | 200 | 35.15 | 0.971 | 27.17 | 0.874 | 32.89 | 0.970 | - | -
SuperSloMo [16] | 39.6 | 500 | 32.90 | 0.957 | 25.65 | 0.857 | 32.33 | 0.960 | - | -
FLAVR [17] | 42.4 | 200 | 36.30 | 0.975 | 27.44 | 0.874 | 33.33 | 0.971 | - | -
CAIN [7] | 42.8 | 200 | 34.76 | 0.970 | 27.21 | 0.873 | 34.91 | 0.969 | - | -
NIO (Ours) | 3.43 | 10 | 36.80 | 0.978 | 23.99 | 0.917 | 36.54 | 0.970 | 38.84 | 0.954
Table 1: The table showing the quantitative performance on Vimeo90K, DAVIS,
UCF101 and DISFA+ dataset. The FNO and NIO models are compared against the
quantitative performance of other models, as presented in Kong et al. [21] and
Shi et al. [46]

  * •

Vimeo90K Dataset [51] The Vimeo90K dataset is built from 89,800 clips taken
from the video streaming site Vimeo. It contains a large variety of scenes.
For this project, the triplet dataset was used. The dataset contains 73,171
3-frame sequences, all of which have a resolution of 448x256.

  * •

Davis-90 The modified Densely Annotated Video Segmentation dataset contains
frames from various scenes. These frames are partitioned into triplet sets and
used for testing the performance of the model.

  * •

UCF101 Dataset The preprocessed UCF101 dataset is a collection of scenes that
have been partitioned into triplets. This dataset is also used for testing the
models.

  * •

DISFA+ Dataset The DISFA+ or the Denver Intensity of Spontaneous Facial Action
Database consists of a large set of facial expression sequences, both posed
and non-posed. The dataset has multiple subjects of different ethnicities
exhibiting various facial expressions and is a comprehensive dataset to study
micro facial expressions. This dataset was chosen due to the increase in the
prevalence of video meetings and social media videos, many of which
predominantly features human faces.

The DISFA+ dataset was processed into triplets and the model was trained to
predict the second frame from the first and third frames. The Vimeo90K dataset
was used to provide a comparison benchmark against other deep learning-based
interpolation approaches while the DISFA+ dataset was used to predict facial
expressions from up close. This served as a test to determine the ability of
neural operator-based models to interpolate minute facial muscle movements.
The frames were resized to 100x100 due to memory and GPU constraints.
Furthermore, the dataset was normalized using Min-Max Normalizer to bring
about uniformity to the different color-schemes used in the videos.

Figure 5: This figure provides a visual comparison of results across various
models. NIO achieves visually comparable results with other models.

####  3.3.2 Hyperparameters

The models were built using Pytorch and trained on A30 GPUs. The Fourier modes
used for the initial layer are 4 and 16. The batch size was set to 10 and the
learning rate was set to 0.0001. The training was done using Cuda 11.0. Adam
optimizer was used with a weight decay of 0.0001. The loss function used for
training was mean squared error (MSE) or L2 Loss.

##  4 Experiments

The model was trained on the Vimeo90K dataset for 50 epochs, with an initial
learning rate of $1e^{-4}$. The input was normalized with Min-Max normalizer
and the batch size was set to 10. The dataset was split in 80:20 ratio for
training and validation. Each input was a couplet of frames while the expected
ground truth was the intermediate frame. The images were resized to 100x100
and were converted to grayscale. The models were tested against the Vimeo90K
dataset, DISFA+ dataset and the DAVIS dataset.

Figure 6: Loss trend in the First Epoch for every 250th batch of Vimeo90K
Dataset.

###  4.1 Evaluation

In this section, we provide a comparison benchmark against the other state-of-
the-art models with SSIM and PSNR as the evaluation metrics. The other state-
of-the-art methods use CNN layers and use convolution in real space, which is
an O($n^{3}$) operation, while the NIO model performs pointwise multiplication
in Fourier space, making it an O($n\log n)$ operation. The operation is also a
global convolution and helps the model converge in fewer epochs, compared to
other models with comparable parameters. The model is shown to perform at
least as well as other models of comparable size.

###  4.2 Ablation Study

We built a lighter model - NIO-Small (NIO-S), with 1.18 million parameters and
trained the model on the Vimeo-90K dataset. In the NIO-S model, the decoder is
replaced with regular convolution2D and the encoder retains the channels
instead of downsampling. The NIO-S model was not as accurate as the NIO model,
but it was faster and took half as much time per epoch. On the A30 GPU, the
NIO model took 5 and a half minutes per epoch while the NIO-S model took 3
minutes and 39 seconds per epoch on average. This model only had three
spectral convolution layers. The two models approached an SSIM of 0.90 within
the first epoch.

Model | Spectral | Time/Epoch | DISFA+
---|---|---|---
| Layers | fixed batch | PSNR | SSIM
NIO-S | 4 | 00:03:39 | 40.22 | 0.987
NIO | 8 | 00:05:43 | 38.84 | 0.954
Table 2: The table showing the differences between NIO-S and NIO models

####  4.2.1 Normalization

Min-Max normalization was applied to the dataset as a preprocessing step,
reducing the range of values that a pixel can take. The models performed
slightly worse on the Vimeo90K dataset when the input was normalized. However,
these models performed better on the other datasets after being trained on the
normalized Vimeo90K dataset, thus showing that normalization as a
preprocessing step helps the models generalize better on data they have not
seen before.

Model | UCF101
---|---
| PSNR | SSIM
NIO | 36.54 | 0.970
NIO + Norm | 36.84 | 0.954
Table 3: The table showing the differences between Normalizing the input and
not normalizing the input

##  5 Conclusion

In this paper, we have presented a Neural Operator-based architecture for
performing frame interpolation. The model is lightweight, resolution
invariant, and discretization invariant, and takes less time than conventional
CNN-based models per epoch since convolution is replaced with pointwise
multiplication and generalizes well on datasets that it has not seen before.
The model has proven effective in capturing information in tiny regions of the
image (tokens) and generalizing well in larger images. The NIO model has only
been trained on a triplet dataset with consecutive frames. However, it remains
to be seen how well it performs when more frames are dropped and the available
frames are far apart from each other. Secondly, the resolution invariance is
an important property of the neural operator and it remains to be seen whether
this can be used to improve the resolution of the images. In the future, we
also want to experiment with videos with heavy motion or with occluded
subjects.

## References

  * [1] Sharib Ali, Felix Zhou, Adam Bailey, Barbara Braden, James E East, Xin Lu, and Jens Rittscher.  A deep learning framework for quality assessment and restoration in video endoscopy.  Medical image analysis, 68:101900, 2021.
  * [2] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.  Depth-aware video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3703–3712, 2019.
  * [3] Sergi Caelles, Jordi Pont-Tuset, Federico Perazzi, Alberto Montes, Kevis-Kokitsi Maninis, and Luc Van Gool.  The 2019 davis challenge on vos: Unsupervised multi-object segmentation.  arXiv:1905.00737, 2019.
  * [4] Xiongtao Chen, Wenmin Wang, and Jinzhuo Wang.  Long-term video interpolation with bidirectional predictive network.  In 2017 IEEE Visual Communications and Image Processing (VCIP), pages 1–4. IEEE, 2017.
  * [5] Harry Cheng, Yangyang Guo, Jianhua Yin, Haonan Chen, Jiafang Wang, and Liqiang Nie.  Audio-driven talking video frame restoration.  IEEE Transactions on Multimedia, 2021.
  * [6] Xianhang Cheng and Zhenzhong Chen.  Video frame interpolation via deformable separable convolution.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10607–10614, 2020.
  * [7] Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee.  Channel attention is all you need for video frame interpolation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10663–10671, 2020.
  * [8] Duolikun Danier, Fan Zhang, and David Bull.  St-mfnet: A spatio-temporal multi-flow network for frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3521–3531, 2022.
  * [9] Xiangling Ding, Pu Huang, Dengyong Zhang, and Xianfeng Zhao.  Video frame interpolation via local lightweight bidirectional encoding with channel attention cascade.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1915–1919. IEEE, 2022.
  * [10] John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro.  Adaptive fourier neural operators: Efficient token mixers for transformers.  arXiv preprint arXiv:2111.13587, 2021.
  * [11] Gang He, Chang Wu, Lei Li, Jinjia Zhou, Xianglin Wang, Yunfei Zheng, Bing Yu, and Weiying Xie.  A video compression framework using an overfitted restoration neural network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 148–149, 2020.
  * [12] Mengshun Hu, Jing Xiao, Liang Liao, Zheng Wang, Chia-Wen Lin, Mi Wang, and Shin’ichi Satoh.  Capturing small, fast-moving objects: Frame interpolation via recurrent motion enhancement.  IEEE Transactions on Circuits and Systems for Video Technology, 2021\.
  * [13] Ping Hu, Simon Niklaus, Stan Sclaroff, and Kate Saenko.  Many-to-many splatting for efficient video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3553–3562, 2022.
  * [14] Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou.  Rife: Real-time intermediate flow estimation for video frame interpolation.  arXiv preprint arXiv:2011.06294, 2020.
  * [15] Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou.  Real-time intermediate flow estimation for video frame interpolation.  In European Conference on Computer Vision, pages 624–642. Springer, 2022.
  * [16] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz.  Super slomo: High quality estimation of multiple intermediate frames for video interpolation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9000–9008, 2018.
  * [17] Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran.  Flavr: Flow-agnostic video representations for fast frame interpolation.  arXiv preprint arXiv:2012.08512, 2020.
  * [18] Alexandros Karargyris and Nikolaos Bourbakis.  Three-dimensional reconstruction of the digestive wall in capsule endoscopy videos using elastic video interpolation.  IEEE transactions on Medical Imaging, 30(4):957–971, 2010.
  * [19] Hannah Halin Kim, Shuzhi Yu, Shuai Yuan, and Carlo Tomasi.  Cross-attention transformer for video interpolation.  arXiv preprint arXiv:2207.04132, 2022.
  * [20] Tae Hyun Kim, Mehdi SM Sajjadi, Michael Hirsch, and Bernhard Scholkopf.  Spatio-temporal transformer network for video restoration.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 106–122, 2018.
  * [21] Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang, and Jie Yang.  Ifrnet: Intermediate feature refine network for efficient frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1969–1978, 2022.
  * [22] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Neural operator: Learning maps between function spaces.  arXiv preprint arXiv:2108.08481, 2021.
  * [23] Ravi Krishnamurthy, John W Woods, and Pierre Moulin.  Frame interpolation and bidirectional prediction of video using compactly encoded optical-flow fields and label fields.  IEEE transactions on circuits and systems for video technology, 9(5):713–726, 1999.
  * [24] Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Daehyun Pak, Yuseok Ban, and Sangyoun Lee.  Adacof: Adaptive collaboration of flows for video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5316–5325, 2020.
  * [25] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Fourier neural operator for parametric partial differential equations.  arXiv preprint arXiv:2010.08895, 2020.
  * [26] Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool.  Vrt: A video restoration transformer.  arXiv preprint arXiv:2201.12288, 2022.
  * [27] Xiaozhang Liu, Hui Liu, and Yuxiu Lin.  Video frame interpolation via optical flow estimation with image inpainting.  International Journal of Intelligent Systems, 35(12):2087–2102, 2020.
  * [28] Yihao Liu, Liangbin Xie, Li Siyao, Wenxiu Sun, Yu Qiao, and Chao Dong.  Enhanced quadratic video interpolation.  In European Conference on Computer Vision, pages 41–56. Springer, 2020.
  * [29] Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, and Yung-Yu Chuang.  Deep video frame interpolation using cyclic frame generation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8794–8802, 2019.
  * [30] Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala.  Video frame synthesis using deep voxel flow.  In Proceedings of the IEEE international conference on computer vision, pages 4463–4471, 2017.
  * [31] Yao Luo, Jinshan Pan, and Jinhui Tang.  Bi-directional pseudo-three-dimensional network for video frame interpolation.  IEEE Transactions on Image Processing, 2022.
  * [32] Mohammad Mavadati, Peyten Sanger, and Mohammad H Mahoor.  Extended disfa dataset: Investigating posed and spontaneous facial expressions.  In proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1–8, 2016.
  * [33] Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, and Alexander Sorkine-Hornung.  Phase-based frame interpolation for video.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1410–1418, 2015.
  * [34] Simon Niklaus and Feng Liu.  Context-aware synthesis for video frame interpolation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1701–1710, 2018.
  * [35] Simon Niklaus and Feng Liu.  Softmax splatting for video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5437–5446, 2020.
  * [36] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive separable convolution.  In Proceedings of the IEEE International Conference on Computer Vision, pages 261–270, 2017.
  * [37] Junheum Park, Keunsoo Ko, Chul Lee, and Chang-Su Kim.  Bmbc: Bilateral motion estimation with bilateral cost volume for video interpolation.  In European Conference on Computer Vision, pages 109–125. Springer, 2020.
  * [38] Junheum Park, Chul Lee, and Chang-Su Kim.  Asymmetric bilateral motion estimation for video frame interpolation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14539–14548, 2021.
  * [39] Minho Park, Sangmin Lee, and Yong Man Ro.  Video frame interpolation via exceptional motion-aware synthesis.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1958–1962. IEEE, 2020.
  * [40] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al.  Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators.  arXiv preprint arXiv:2202.11214, 2022.
  * [41] Tomer Peleg, Pablo Szekely, Doron Sabo, and Omry Sendik.  Im-net for high resolution video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2398–2407, 2019.
  * [42] Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadenesheli.  U-no: U-shaped neural operators.  arXiv preprint arXiv:2204.11127, 2022.
  * [43] Lars Lau Rakêt, Lars Roholm, Andrés Bruhn, and Joachim Weickert.  Motion compensated frame interpolation with a symmetric optical flow constraint.  In International Symposium on Visual Computing, pages 447–457. Springer, 2012.
  * [44] Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, and Zhiyong Gao.  Blurry video frame interpolation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5114–5123, 2020.
  * [45] Zhihao Shi, Xiaohong Liu, Kangdi Shi, Linhui Dai, and Jun Chen.  Video frame interpolation via generalized deformable convolution.  IEEE Transactions on Multimedia, 24:426–439, 2021.
  * [46] Zhihao Shi, Xiangyu Xu, Xiaohong Liu, Jun Chen, and Ming-Hsuan Yang.  Video frame interpolation transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17482–17491, 2022.
  * [47] Aljoscha Smolic, Karsten Muller, Kristina Dix, Philipp Merkle, Peter Kauff, and Thomas Wiegand.  Intermediate view interpolation based on multiview video plus depth for advanced 3d video systems.  In 2008 15th IEEE International Conference on Image Processing, pages 2448–2451. IEEE, 2008.
  * [48] Haoyue Tian, Pan Gao, and Xiaojiang Peng.  Video frame interpolation based on deformable kernel region.  arXiv preprint arXiv:2204.11396, 2022.
  * [49] Quang Nhat Tran and Shih-Hsuan Yang.  Video frame interpolation via down–up scale generative adversarial networks.  Computer Vision and Image Understanding, 220:103434, 2022.
  * [50] Hung-Ming Wang, Chun-Hao Huang, and Jar-Ferr Yang.  Depth maps interpolation from existing pairs of keyframes and depth maps for 3d video generation.  In Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pages 3248–3251. IEEE, 2010.
  * [51] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman.  Video enhancement with task-oriented flow.  International Journal of Computer Vision, 127(8):1106–1125, 2019\.
  * [52] Zhifeng Zhang, Li Song, Rang Xie, and Li Chen.  Video frame interpolation using recurrent convolutional layers.  In 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), pages 1–6. IEEE, 2018.