# SportsSloMo: A New Benchmark and Baselines for
Human-centric Video Frame Interpolation

Jiaben Chen
UC San Diego
<EMAIL_ADDRESS>Work mainly done when Jiaben Chen was an intern at
Northeastern University. Huaizu Jiang
Northeastern University
<EMAIL_ADDRESS>
###### Abstract

Human-centric video frame interpolation has great potential for improving
people’s entertainment experiences and finding commercial applications in the
sports analysis industry, _e.g_., synthesizing slow-motion videos. Although
there are multiple benchmark datasets available in the community, none of them
is dedicated for human-centric scenarios. To bridge this gap, we introduce
SportsSloMo, a benchmark consisting of more than 130K video clips and 1M video
frames of high-resolution ($\geq$720p) slow-motion sports videos crawled from
YouTube. We re-train several state-of-the-art methods on our benchmark, and
the results show a decrease in their accuracy compared to other datasets. It
highlights the difficulty of our benchmark and suggests that it poses
significant challenges even for the best-performing methods, as human bodies
are highly deformable and occlusions are frequent in sports videos. To improve
the accuracy, we introduce two loss terms considering the human-aware priors,
where we add auxiliary supervision to panoptic segmentation and human
keypoints detection, respectively. The loss terms are model agnostic and can
be easily plugged into any video frame interpolation approaches. Experimental
results validate the effectiveness of our proposed loss terms, leading to
consistent performance improvement over 5 existing models, which establish
strong baseline models on our benchmark. The dataset and code can be found at:
https://neu-vi.github.io/SportsSlomo/.

##  1 Introduction

Video frame interpolation (VFI) is a technique that synthesizes intermediate
frames from input images, enhancing the clarity of content that may be
difficult to see otherwise. It has broad applications in areas such as slow-
motion video generation [26], novel view synthesis [37], video compression
[70], cartoon and rendered content generation [35, 4], etc. In recent years,
we have witnessed significant advances in this field, due in large part to the
development of various benchmarks [1, 62, 55, 63, 76, 13, 61].

Humans are important subject of most of the videos nowadays. With the
widespread use of mobile devices, people can easily record and share their
daily experiences with family, friends, and colleagues. Meanwhile, live
broadcasts of sporting events attract a large audience. Automatically
generated slow-motion videos can create a more immersive and engaging
experience for users by highlighting details and valuable moments of their
lives that may be missed in real-time. Therefore, improving video frame
interpolation results for human-centric videos has great potential for
enhancing user experiences in entertainment. Human-centric video frame
interpolation approaches can also be beneficial in various industries.
Athletes and coaches, for example, can use slow-motion synthesis to identify
flaws in techniques, highlight areas for improvement, and gain a more detailed
understanding of how different factors may contribute to the success.

Figure 1: Human-centric video frame interpolation results. We propose human-
aware auxiliary losses to improve interpolation accuracy at motion boundaries.

A new benchmark. Although various benchmarks are available for video frame
interpolation, none of them is dedicated to human-centric scenarios. To bridge
this gap and to foster the research in this important direction, we create a
new dataset, SportsSloMo, focusing on high-resolution ($\geq$720p) slow-motion
sports videos crawled from YouTube under the Common Creative Licence. We
collected videos from different sports games, such as football, basketball,
baseball, hockey, etc. Since a video may contain advertisement, transition
frames, changes of shot, and non-slow-motion content, we carefully curate the
data to remove such unwanted content and finally split each long video into a
set of short slow-motion clips of 9 frames. The first and last frames are used
as input and the rest 7 intermediate frames as ground truths for training and
evaluating VFI models. In total, our benchmark has 130K video clips and more
than 1M video frames. Compared with other existing datasets, as shown in Table
1, our proposed SportsSloMo benchmark is the largest one so far, with high
resolution and focus on human-centric scenarios.

While SportsSloMo is primarily designed for human-centric video frame
interpolation in this paper, we believe it may also be useful for other tasks
such as video super-resolution [6, 36], group activity recognition [64, 79,
78], and dynamic view synthesis [37, 16, 72]. By releasing this dataset to the
entire community, we hope to empower researchers to explore innovative
applications and encourage technical advancement in the field.

Figure 2: Visualization of human keypoints [75] and panoptic segmentation
masks [9].

Benchmarking existing approaches. To facilitate development and evaluation of
human-centric video frame interpolation methods, we re-train state-of-the-art
approaches [26, 34, 22, 24, 27] using their publicly released code on our
SportsSloMo dataset. As the human bodies are highly deformable and occlusions
are frequent in sports videos, the accuracy of all the approaches decrease
compared on other datasets. For instance, the best-performing approach on
SportsSloMo, EBME [27] produces a PSNR score of 30.15, compared with 36.19 on
the Viemo90K [76] and 30.64 on the hard split of the SNU-FILM [13] benchmarks,
respectively. It highlights the difficulty of our benchmark and suggests
significant challenges need to be addressed.

Baseline models. To improve the existing VFI models on our benchmark, we
introduce two loss terms, considering the human-aware priors. We propose loss
terms based on human segmentation in the panoptic setting [9] and human
keypoints estimation [75] as extra supervision for intermediate frame
synthesis. Fig. 2 shows a visualization of detected human keypoints and
segmentation masks in our dataset. Human segmentation masks indicate where
human body boundaries are, which is helpful for reducing ghost effects around
the motion boundaries. At the same time, human keypoints estimation can also
tell us where each body part is, enforcing coherent motion trajectories in the
synthesized video frames. Specifically, we feed a synthesize intermediate
frame as input to pre-trained panoptic segmentation and keypoints detection
models and use the consistency between the output and the results obtained
from the real intermediate frame as supervision. As shown in Fig. 1, by
supervising with our proposed auxiliary loss terms, we improve the
interpolation quality at motion boundaries with less blurry results in
scenarios with large motion and occlusion.

Both of these loss terms are model agnostic and can be easily integrated into
any video frame interpolation approach. Experimental results show that they
can consistently improve the accuracy of 5 existing approaches, leading to
strong baselines on our benchmark.

Table 1: Comparisons of different benchmark datasets for video frame interpolation. (#inter. frames indicates the number of intermediate frames to synthesize.) Dataset | #clips | #images | #inter. | resolution | human-
---|---|---|---|---|---
frames | centric
UCF101 [62] | 0.4K | 1K | 1 |  256$\times$256 | ✓
Adobe240fps [63] | 0.1K | 80K | 7 |  1280$\times 720$ |  ✗
Vimeo90K [76] | 70K | 220K | 1 |  448$\times$256 |  ✗
SNU-FILM [13] | 1.2K | 3.6K | 1 |  1280$\times 720$ |  ✗
X4K1000FPS [61] | 4.4K | 286K | 7 |  4096$\times$2160 |  ✗
SportsSloMo | 130K | 1183K | 7 |  1280$\times$720 | ✓

To sum up, this paper makes the following contributions:

  * •

We introduce SportsSloMo, a new benchmark dataset consisting of a large amount
of slow-motion sports videos. This is the first high-resolution dataset
dedicated for human-centric video frame interpolation, supporting synthesis of
multiple intermediate frames, to our best knowledge.

  * •

We benchmark the state-of-the-art approaches on the new benchmark, where the
accuracy of all the approaches decrease, indicating the challenges of the
human-centric video frame interpolation task.

  * •

We propose two auxiliary loss terms, which take the human priors into account
for video frame interpolation and can easily be plugged into existing video
frame interpolation approaches. Experimental results validate their
effectiveness, yielding strong baseline models on our new benchmark.

##  2 Related Work

###  2.1 Benchmark datasets

Existing publicly available datasets already provide a valuable resource for
developing and evaluating video frame interpolation methods. We would briefly
introduce these datasets and reveal some limitations in this section. Table. 1
shows a comparison between existing datasets and our proposed SportsSloMo
dataset.

The SNU-FILM [13] dataset is a widely-used benchmark for VFI evaluation,
containing 1240 frame triplets of 1280 $\times$ 720 resolution. And it is
divided into four different parts, namely, Easy, Medium, Hard, and Extreme
according to motion magnitude. The Middlebury benchmark [1] is another widely
used dataset, image resolution in this dataset is around $640\times 480$.
However, it is commonly only used to evaluate VFI methods for 8 sequences.
UCF101 [62] is originally a dataset for human action recognition, containing a
variety of human actions. With the test set constructed by [41], it is also
used to evaluate VFI methods, containing 379 triplets of 256 $\times$ 256
frame size. Nevertheless, its scale is rather small and its resolution is also
low. The Adobe240fps dataset [63], originally for video deblurring, is another
widely used dataset for VFI. It is consisted of high frame-rate videos (240
fps) with a resolution of 1280 $\times$ 720, yet the videos are from only 118
clips.

The mostly used dataset for training and evaluating VFI methods is the
Vimeo90K dataset [76]. It contains 73171 frame triplets from 14777 video clips
extracted from real-life video clips with a fps $\leq 30$. Nevertheless, one
of its main drawback is the low resolution of 448 $\times$ 256 obtained by
downscaling the original high resolution frames. Moreover, as VFI methods have
been rapidly improving in recent years, their performance on the widely-used
Vimeo90K dataset has approached saturation. To further advance the state-of-
the-art in video frame interpolation, a new dataset with bigger scale, higher
resolution and more challenging scenarios is necessary. X4K1000FPS is a
recently released high frame rate (1000 fps) with 4K spatial resolution to
promote the study of VFI for very high resolution videos.

As a result, none of existing datasets contains rich human-centric data with
high resolution and big scale. To bridge this gap, we introduce the
SportsSloMo dataset, a human-centric VFI dataset with 130K video clips and 1M
video frames at 240 fps with a resolution of $1280\times 720$, aiming to
foster research in human-centric VFI.

###  2.2 Video frame interpolation methods

Existing VFI methods could be generally classified into flow-agnostic and
flow-based methods.

Flow-agnostic approaches model VFI without explicit intermediate motion
representation. _Phase-based_ methods [46, 45] directly predict the phase
decomposition of the intermediate frame, but can only handle motion within a
limited range. _Kernel-based_ methods are the mainstream approach in this
category, which typically aims to estimate intermediate frames by learning
adaptive kernels to convolve input frames [51, 52]. Over the years, numerous
improvements are proposed in this field, including using deformable
convolution [10, 11], formulating interpolated motion estimation as
classification [54], blending deep features [18], introducing dual-frame
adversarial loss [34], performing channel attention [13] and utilizing 3D
space-time convolutions [30]. Recently, Shi et al. introduced a Transformer-
based framework to model long-range dependencies with the aid of attention
mechanisms. By directly hallucinating pixel values, these methods tend to
generate blurry results and artifacts, especially in fast-moving scenes [42].

Figure 3: The SportsSloMo dataset. (a) Sampled frames from the dataset,
covering various sports categories and challenging human-centric content for
VFI; (b) The histogram of flow magnitude of all pixels in the dataset; (c) The
histogram of mean flow magnitude of all images in the dataset. Figure 4:
Distribution of different sports categories in our SportsSloMo benchmark.

Flow-based approaches currently serve as a promising direction in VFI.
Generally speaking, flow-based method takes a paradigm of a two-stages
pipeline: (1) flow estimation, and (2) frame synthesis. They first estimate
optical flow between input frames, and then synthesize intermediate frames
using image warping [25]. As a representative work, SuperSlomo [26] by Jiang
et al. adopted a skip-connected U-Net to estimate bi-directional optical flows
under the assumption of linear motion. Quadratic [74] and cubic [12, 66]
trajectory assumptions have also been made to approximate intermediate motion.
Recent work has explored various techniques to improve intermediate flow
estimation and interpolation accuracy, including forward-warping via softmax
splatting [50, 22], voxel flow [41], cycle consistency loss[59, 39], task-
oriented flow distillation loss [33], Gram matrix loss [58], implicit neural
function [8], occlusion mask [3], anchor points alignment [61] and privileged
distillation [24]. Considering additional information like contextual maps
[49], depth maps [2] and auxiliary visual information from event cameras [67,
20] can also further improve interpolation accuracy. Park et al. employed
symmetric bilateral motion field estimation, and further improved intermediate
motion estimation accuracy through asymmetric bilateral motion field [53]. Lu
et al. [42] leveraged Transformer architecture [68] to model long-term
dependency. Jin et al. [27] proposed a novel bi-directional motion estimator
in a pyramid structure.

In a nutshell, although previous methods have proposed successful designs to
handle complex motion and occlusion, none of them is carefully designed for
human-centric scenes. As discussed in Sec. 1, human-centric VFI is confronted
with various challenges including dynamic pose variation, complex human motion
patterns and occlusion in crowded scenes. Additionally, accurately
synthesizing fine details such as facial expressions and hand gestures can be
challenging. To this end, we propose to consider human-aware priors through
incorporating auxiliary supervision to both human keypoints detection and
panoptic segmentation.

##  3 SportsSloMo Benchmark

A myriad of benchmark datasets for video frame interpolation have been
proposed in recent years, including Middlebury [1], GoPro [48], UCF101 [62],
DAVIS [55], Adobe240fps [63], Vimeo90K [76], SNU-FILM [13] and X4K1000FPS
[61]. But none of these them focuses on human-centric VFI, _e.g_., in sports
scenes. This limits the study of VFI methods targeted for human-centric
applications such as enhanced entertainment experiences, commercial potential,
etc.

Figure 5: Illustration of augmenting a flow-based VFI model with human-aware
auxiliary losses.

To bridge this gap and to foster future research, we propose SportsSloMo, a
challenging dataset consisting of high-resolution ($\geq$ 720p) sports videos
crawled from YouTube under the Common Creative License. Careful curation has
been taken to remove the unwanted content in the videos, including
advertisement, transition frames, changes of shot, flashing lights, and non-
slow-motion content. Specifically, we first conduct human detection on all the
videos by utilizing Yolov3 [60] and remove video frames without detected
humans. Second, we remove frames with flashing lights by setting a threshold
about the brightness change between consecutive frames (large brightness
change indicates the existence of a flashing light). Third, we use RAFT [65]
to measure motion magnitude and set a threshold to discard non-slow-motion
video segments. Finally, we carefully curate the the extracted clips manually
to keep only high-quality self-consistent videos. With such semi-automatic
curation, we end up having a set of short slow-motion clips of 9 frames, where
each frame has a spatial resolution of at least 1280 $\times$ 720\. The first
and last frames are used as input for a VFI method and the 7 intermediate
frames as ground truths for model training and evaluation, approximately
corresponding to converting a video of 30-fps (frames per second) to 240-fps.

In total, we collect 131,464 video clips and 1.2M individual video frames from
259 raw YouTube videos, which covers 22 various sports categories in our
dataset with different content and motion patterns, including hockey,
baseball, skating, basketball, running, volleyball, etc. Fig. 4 shows the
proportion of different sports categories. Visualizations of randomly sampled
video frames are available in Fig. 3. We refer the readers to the
supplementary material for more results.

As shown in Table 1, compared with existing datasets, not only our dataset is
tailored for human-centric scenarios, but also surpasses others in terms of
scale, resolution, and frame rate. Fig. 3 demonstrates the histogram of flow
magnitude of all pixels and mean flow magnitude of all images in our dataset
(calculated using GMFlow [73]), with a comparison with the widely-used SNU-
FILM [13] dataset (which has the same spatial resolution as ours) and the
recently introduced X4k1000FPS [61] dataset. As can be seen, our SportsSloMo
benchmark contains more large-displacement motion compared to the SNU-FILM
dataset. For instance, for the avarege flow magnitude on the image-level (Fig.
3(c)), our dataset has way more images, whose mean flow magnitude is greater
than 20 pixels. And we can also observe that both our SportsSloMo and
X4K1000FPS datasets contain large motions.

We split our proposed SportsSloMo dataset into _train_ and _test_ , containing
115,421 and 16,043 video clips, respectively. For each sports category, videos
are split into _train_ and _test_ _without intersection_ , so that the test
videos are completely unseen during training.

##  4 Human-aware Loss Terms

We first briefly give an overview about the VFI and then introduce our
proposed auxiliary loss terms taking human-aware priors into account.

###  4.1 Overview of the Pipeline

We elaborate the process based on the state-of-the-art method EBME [27], which
consists of two stages: bi-directional flow estimation and intermediate frame
synthesis. The overall pipeline is illustrated in Fig. 5. It is worth noting
that our proposed loss terms are flexible, which can be plugged into other
flow-based or flow-agnostic VFI models, _e.g_., SuperSloMo [26] and AdaCoF
[34], as shown in Section 5.

Given two input frames $I_{0}$ and $I_{1}$, the goal of VFI is to synthesize
the intermediate frame $I_{t}$ at arbitrary time step $t\in(0,1)$.
Specifically, in the first stage, $I_{0}$ and $I_{1}$ are fed into a flow
estimation network to predict bi-directional flow fields $F_{0\rightarrow 1}$
and $F_{1\rightarrow 0}$. After that, there are two available choices to
utilize the flow fields for frame synthesis: forward-warping [49, 50, 27] and
backward-warping [26, 53, 61]. Backward-warping approximates the intermediate
motion fields $F_{t\rightarrow 0}$ and $F_{t\rightarrow 1}$, while forward-
warping works in the opposite direction. Following EBME [27], we adopt the
forward warping and approximate $F_{0\rightarrow t}$ and $F_{1\rightarrow t}$
through a linear combination

$\displaystyle F_{0\rightarrow t}=t\cdot F_{0\rightarrow
1},~{}~{}~{}F_{1\rightarrow t}=(1-t)\cdot F_{1\rightarrow 0}.$ (1)

In the second stage, we forward-warp the input frames $I_{0}$ and $I_{1}$
using $F_{0\rightarrow t}$ and $F_{1\rightarrow t}$, respectively, to the time
step $t$, which are fed into a synthesis network to generate the intermediate
frame $\hat{I}_{t}$.

To supervise the network training, the reconstruction error of the
intermediate frame is used as the loss. In [27], the weighted sum of
Charbonnier loss [7] $L_{char}$ and census loss [44] $L_{cen}$ between the
ground truth intermediate frame $I_{t}$ and predicted frame $\hat{I}_{t}$ are
adopted

$\displaystyle L_{basic}=L_{char}(I_{t}-\hat{I}_{t})+\lambda_{cen}\cdot
L_{cen}(I_{t},\hat{I}_{t}),$ (2)

where $L_{char}(x)=(x^{2}+\epsilon^{2})^{\alpha}$. Following [27], we set the
loss weight $\lambda_{cen}=0.1$, $\alpha=0.5,\epsilon=10^{-6}$.

Optionally, we can add supervision to the bi-directional optical flow
$F_{0\rightarrow 1}$ and $F_{1\rightarrow 0}$. We discuss its effectiveness in
Section 5.5.

Figure 6: Qualitative result with auxiliary loss terms.

###  4.2 Panoptic Segmentation Auxiliary Loss

In human-cenrtic scenarios, accurate estimation of human body boundaries is
crucial. Inaccuracies in the estimated body boundaries may lead to visible
artifacts in the synthesized video frame, especially in regions where the body
movement is complex or there are occlusions. Human segmentation masks directly
tell where human body boundaries are, helping to reduce ghost effects around
the motion boundaries. Though instance and panoptic segmentation could both
serve for this purpose. We empirically find that pre-trained segmentation
models tend to yield messy results for instance segmentation. With this in
mind, we propose an auxiliary panoptic segmentation loss.

Specifically, we adopt state-of-the-art Mask2Former [9] trained on COCO
panoptic dataset [31] with Swin-L backbone [40] from Detectron2 [71] to
generate panoptic segmentation pseudo ground truths from the ground-truth
intermediate frame $I_{t}$. During training, the panoptic segmentation results
of the synthesized intermediate frame $\hat{I_{t}}$ obtained using the same
Mask2Former model is compared with the segmentation results of $I_{t}$, which
enhances the consistency of the human body boundaries between $I_{t}$ and
$\hat{I_{t}}$. We follow the loss function of Mask2Former and use the binary
cross-entropy loss and the dice loss [47] for our auxiliary panoptic
segmentation loss

$\displaystyle L_{aux}^{seg}=\lambda_{ce}\cdot L_{ce}+\lambda_{dice}\cdot
L_{dice}.$ (3)

where we set $\lambda_{ce}=5.0$ and $\lambda_{dice}=5.0$. The panoptic
segmentation model is frozen during training and gradient of this loss will be
backpropagated to the VFI model in a way similar to the perceptual loss [29],
enforcing the model to produce high-quality synthesis results so that the
panoptic segmentation results are close to the ones obtained from the true
intermediate frame.

Figure 7: Qualitative comparisons on SportsSloMo dataset. With the enhancement
of the auxiliary human keypoint and panoptic segmentaiton losses, EBME
achieves better interpolation results.

###  4.3 Human Keypoint Auxiliary Loss

In addition to segmentation, we also consider the human keypoints estimation
as an extra supervision. Accurate estimation of human keypoints is also
crucial for human-centric video frame interpolation, which are indicative
about the position of each body part, providing additional cues for motion
estimation. Hence, by tracking human keypoints, we can better preserve human
motions by enforcing coherent motion trajectories and guide the interpolation
process to generate more plausible intermediate frames. Therefore, we propose
an auxiliary human keypoint auxiliary loss to improve VFI accuracy on human-
centric scenes.

Specifically, we adopt the state-of-the-art human pose estimation model
ViTPose [75] trained on COCO [38], with the ViT-L backbone [14] initialized
with MAE [19] pre-trained weights. We first use an off-the-shelf human
detector YOLOv8 [28] to detect person instances in the ground-truth
intermediate frame $I_{t}$. We then employ ViTPose to estimate the keypoints
of the detected pseudo keypoints ground truths. Similar to the auxiliary
panoptic segmentation loss, the predicted intermediate frame $\hat{I_{t}}$ is
fedd into the pre-trained ViTPose to get the predicted keypoints results,
which are compared with the results obtained from $I_{t}$ as supervision. We
follow the loss function of ViTPose and use the joints MSE loss over heatmaps
for our auxiliary keypoint loss $L_{aux}^{kpt}$. Again, the keypoint
estimation model is frozen during training and the gradient of this loss will
be backpropagated to the VFI model to encourage high-quality intermediate
frame synthesis.

##  5 Experiments

###  5.1 Experimental Setup

Baselines. We benchmark existing VFI approaches by re-training several state-
of-the-art methods on our SportsSloMo dataset using their publicly available
code, including SuperSlomo [26], AdaCoF [34], M2MVFI [22], RIFE [24] and EBME
[27]. All the baseline models are trained from scratch on SportsSloMo dataset.
We select such five VFI models for their high accuracy on other datasets111We
also test CAIN [13] but are unable to get stable training results..

Table 2: Quantitative results of different approaches on the SportsSloMo dataset. Method | Venue |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  IE$\downarrow$
---|---|---|---|---
SuperSlomo [26] | CVPR 2018 | 29.77 | 0.910 | 9.14
SuperSloMo w/ aux. | - | 30.24 | 0.917 | 8.46
AdaCoF [34] | CVPR 2020 | 28.79 | 0.926 | 5.84
AdaCoF w/ aux. | - | 28.94 | 0.926 | 5.72
M2MVFI [22] | CVPR 2022 | 29.03 | 0.935 | 5.16
M2MVFI w/ aux. | - | 29.29 | 0.936 | 5.07
RIFE [24] | ECCV 2022 | 29.69 | 0.931 | 5.25
RIFE w/ aux. | - | 29.87 | 0.933 | 5.16
EBME [27] | WACV 2023 | 30.15 | 0.941 | 4.66
EBME w/ aux. | - | 30.48 | 0.944 | 4.40

Furthermore, to study the effectiveness of our plug-in human-aware auxiliary
losses, we further conduct experiments on top-perform VFI methods on our
benchmark: SuperSlomo [26], AdaCoF [34], M2MVFI [22], RIFE [23], and EBME
[27]. They form strong baseline models for future development of human-centric
VFI.

Evaluation metrics. Following previous VFI methods [26, 23], we adopt the
widely-used signal-to-noise ratio (PSNR), structural similarity (SSIM) [69],
and interpolation error (IE) [1] to evaluate the interpolation results. For
PSNR and SSIM, higher indicates better performance. And for IE, the lower the
better.

Implementation details. For the auxiliary panoptic segmentation and human
keypoint loss terms, to avoid the influence of Batch Normalization (BN) layers
on the training phase, we replaced the BN layers with frozen Batch
Normalization (frozen BN) layers. Training and evaluation are both conducted
on 8 NVIDIA RTX A6000 GPUs.

###  5.2 Benchmarking Existing Methods

During training, we randomly sample a frame between the first and the ninth
(last) frame for VFI methods supporting arbitrary time frame interpolation
like SuperSlomo [26], M2MVFI [22], RIFE [24], and EBME [27]. For VFI methods
that can only synthesize the middle frame like AdaCoF [34], we randomly
generate triplets within the nine frames such that the target frame is always
in the middle. During evaluation, arbitrary-time-supported VFI methods takes
the time step as input and interpolate every intermediate frame to compute the
metrics, while single-frame methods recursively generate each intermediate
frame.

Table 2 shows a quantitative comparison between exisiting VFI methods on our
proposed SportsSloMo dataset. The performance for each baseline model decrease
greatly compared with results on popular datasets. For instance, the best
performing approach EBME [27] yields a PSNR score of 30.15, compared with
36.19 on the Vimeo90K [76] benchmark. This demonstrates the challenge of our
proposed human-centric dataset.

###  5.3 Strong Baseline Models

Table 2 also demonstrates the effectiveness of our proposed human-aware
auxiliary losses. Specifically, we combine keypoint auxiliary loss and
segmentation auxiliary loss to fine-tune EBME [27], M2MVFI [22], RIFE [23],
AdaCoF [34], and SuperSloMo [26] on our dataset, leading to a increase of
0.33dB, 0.26dB, 0.18dB, 0.15dB, and 0.37dB in terms of PSNR, respectively.
They form strong baseline models on our SportsSloMo benchmark to facilitate
future development and evaluation. Fig. 7 shows a qualitative comparison
between different methods. We can see that better interpolation results can be
obtained for the highly deformable arms of the athletes and the background
under occlusions.

###  5.4 Ablation Study

In this section, we present ablation study on EBME [27] to analyze the
different design choices of auxiliary keypoint loss and auxiliary segmentation
loss. The results are demonstrated in Table. 3. As can be seen, both losses
can successfully improve the performance of EBME. This justifies our
motivation to leverage human-aware priors to improve the human-centric video
frame interpolation. Finally, we combine these two auxiliary losses on EBME,
which leads to largest performance gain. We provide an qualitative example in
Fig. 6 to demonstrate the effectiveness of our proposed keypoint and
segmentation auxiliary losses. We can clearly see that better interpolation
results can be obtained around the elbow and hand.

Table 3: Effectiveness of different auxiliary loss terms. Model | segmentation | keypoint | PSNR$\uparrow$ | SSIM$\uparrow$ | IE$\downarrow$
---|---|---|---|---|---
loss | loss
SuperSloMo | ✗ | ✗ | 29.77 | 0.910 | 9.14
✓ | ✗ | 30.19 | 0.916 | 8.56
✗ | ✓ | 30.22 | 0.916 | 8.49
✓ | ✓ | 30.24 | 0.917 | 8.46
EBME | ✗ | ✗ | 30.15 | 0.941 | 4.66
✓ | ✗ | 30.26 | 0.942 | 4.61
✗ | ✓ | 30.34 | 0.943 | 4.46
✓ | ✓ | 30.48 | 0.944 | 4.40

###  5.5 Limitations and Discussions

While we have introduced human-aware loss terms to handle challenging human-
centric scenarios in our proposed dataset, there still remains noteworthy
limitations for future exploration.

First of all, for human-centric scenes, the presence of large, complex, and
highly deformable human motions and occlusion by crowds make finding
correspondences of humans (_i.e_., optical flow) between input frames a
challenging task. Recent studies [24, 33] have demonstrated the effectiveness
of knowledge distillation [21] to improve the intermediate flow estimation.
Similar to the segmetnation and keypoints auxiliar loss terms, we explore the
optical flow auxiliary loss. Specifically, we use the optical flow results
obtained from state-of-the-art model GMFlow [73], which is pre-trained on the
optical flow benchmarks [15, 43, 5], as the teacher network to perform flow
distillation. We found, however, mixed results on our SportsSloMo benchmark as
reported in Table 4. It successfully improves the performance for SuperSlomo
[26] but results in a decline in performance for EBME [27]. We also fine-tune
the GMFlow model on the human flow dataset [57], consisting of synthetic human
figures overlaied on top of real world images as background. However, we
observed significant domain gap between the synthetic training images and
real-world test images, which lead to poor optical flow estimation results. To
this end, how to improve human-centric optical flow, especially in the context
of VFI, is still a challenging problem.

Second, we only utilize 2D cues of human bodies in our proposed loss terms to
enhance VFI. Recently, there has been significant advance made in 3D human
body reconstruction from videos, for instance, [17, 77, 56, 32], considering
both the spatial and temporal information. By lifting 2D videos into the 3D
space, occlusions may be better solved. More advanced loss terms involving 3D
human reconstruction is worth exploring. We leave this as future work to
explore.

Finally, in the SportsSloMo dataset, various sports involve fast-moving balls
and sports equipment. In this paper, we only consider human boundary and body
parts for motion estimation. Fast-moving objects, however, in these scenarios
are also challenging for VFI. More effort is needed for such a problem.

Table 4: Effectiveness of optical flow auxiliary loss. Model |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  IE$\downarrow$
---|---|---|---
SuperSlomo w/ flow loss |  30.18 (+0.31) |  0.917 (+0.007) |  8.47 (-0.67)
EBME w/ flow loss |  29.85 (-0.30) |  0.940 (-0.001) |  4.69 (+0.03)

##  6 Conclusion

In this paper, we introduced a new benchmark, SportsSloMo, focusing on human-
centric video frame interpolation. Our benchmark contains 130K video clips and
more than 1M video frames obtained from high-resolution ($\geq$ 720p) slow-
motion sports videos crawled from YouTube with careful curation. Due to the
complex, highly deformation human motion and frequent occlusion, this
benchmark imposes significant challenges to existing VFI models, where their
accruacy decreases compared with other existing datasets. To address such
challenges, we introduce human-aware auxiliary loss terms to enhance existing
methods. Our loss terms are model agnostic and have been successfully applied
to five existing VFI methods, establishing strong baseline models on our
benchmark. Our benchmark and the baseline models is publicly released to
foster future research in this new direction. Moreover, we believe our
proposed dataset is also useful for tasks such as video super-resolution,
group activity recognition. and dynamic view synthesis, etc.

## Acknowledgement

We thank Thuy-Tien Bui and Devroop Kar for their help on data collection. This
work was partially supported by the National Science Foundation under Award
IIS-2310254.

## References

  * [1] Simon Baker, Daniel Scharstein, JP Lewis, Stefan Roth, Michael J Black, and Richard Szeliski.  A database and evaluation methodology for optical flow.  International journal of computer vision, 92(1):1–31, 2011.
  * [2] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.  Depth-aware video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3703–3712, 2019.
  * [3] Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang.  Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement.  IEEE transactions on pattern analysis and machine intelligence, 43(3):933–948, 2019.
  * [4] Karlis Martins Briedis, Abdelaziz Djelouah, Mark Meyer, Ian McGonigal, Markus H. Gross, and Christopher Schroers.  Neural frame interpolation for rendered content.  ACM Trans. Graph., 40(6):239:1–239:13, 2021.
  * [5] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black.  A naturalistic open source movie for optical flow evaluation.  2012\.
  * [6] Kelvin C.K. Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy.  Investigating tradeoffs in real-world video super-resolution.  In CVPR, 2022.
  * [7] Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud.  Two deterministic half-quadratic regularization algorithms for computed imaging.  In Proceedings of 1st international conference on image processing, volume 2, pages 168–172. IEEE, 1994.
  * [8] Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang.  Videoinr: Learning video implicit neural representation for continuous space-time super-resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2047–2057, 2022.
  * [9] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290–1299, 2022.
  * [10] Xianhang Cheng and Zhenzhong Chen.  Video frame interpolation via deformable separable convolution.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10607–10614, 2020.
  * [11] Xianhang Cheng and Zhenzhong Chen.  Multiple video frame interpolation via enhanced deformable separable convolution.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7029–7045, 2021.
  * [12] Zhixiang Chi, Rasoul Mohammadi Nasiri, Zheng Liu, Juwei Lu, Jin Tang, and Konstantinos N Plataniotis.  All at once: Temporally adaptive multi-frame interpolation with advanced motion modeling.  In European Conference on Computer Vision, pages 107–123. Springer, 2020.
  * [13] Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee.  Channel attention is all you need for video frame interpolation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10663–10671, 2020.
  * [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [15] Alexey Dosovitskiy, Philipp Fischery, Eddy Ilg, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, Thomas Brox, et al.  FlowNet: Learning optical flow with convolutional networks.  2015\.
  * [16] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic view synthesis from dynamic monocular video.  In ICCV, 2021.
  * [17] Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa*, and Jitendra Malik*.  Humans in 4D: Reconstructing and tracking humans with transformers.  In International Conference on Computer Vision (ICCV), 2023.
  * [18] Shurui Gui, Chaoyue Wang, Qihua Chen, and Dacheng Tao.  Featureflow: Robust video interpolation via structure-to-texture generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14004–14013, 2020.
  * [19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [20] Weihua He, Kaichao You, Zhendong Qiao, Xu Jia, Ziyang Zhang, Wenhui Wang, Huchuan Lu, Yaoyuan Wang, and Jianxing Liao.  Timereplayer: Unlocking the potential of event cameras for video interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17804–17813, 2022.
  * [21] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [22] Ping Hu, Simon Niklaus, Stan Sclaroff, and Kate Saenko.  Many-to-many splatting for efficient video frame interpolation.  In CVPR, 2022.
  * [23] Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou.  Rife: Real-time intermediate flow estimation for video frame interpolation.  arXiv preprint arXiv:2011.06294, 2020.
  * [24] Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou.  Real-time intermediate flow estimation for video frame interpolation.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIV, pages 624–642. Springer, 2022.
  * [25] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  Advances in neural information processing systems, 28, 2015.
  * [26] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik G. Learned-Miller, and Jan Kautz.  Super slomo: High quality estimation of multiple intermediate frames for video interpolation.  In CVPR, 2018.
  * [27] Xin Jin, Longhai Wu, Guotao Shen, Youxin Chen, Jie Chen, Jayoon Koo, and Cheul-hee Hahm.  Enhanced bi-directional motion estimation for video frame interpolation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5049–5057, 2023.
  * [28] Glenn Jocher, Ayush Chaurasia, and Jing Qiu.  YOLO by Ultralytics, 1 2023.
  * [29] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, 2016.
  * [30] Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran.  Flavr: Flow-agnostic video representations for fast frame interpolation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2071–2082, 2023.
  * [31] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár.  Panoptic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9404–9413, 2019.
  * [32] Muhammed Kocabas, Chun-Hao P. Huang, Otmar Hilliges, and Michael J. Black.  PARE: Part attention regressor for 3D human body estimation.  In ICCV, 2021.
  * [33] Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang, and Jie Yang.  Ifrnet: Intermediate feature refine network for efficient frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1969–1978, 2022.
  * [34] Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Daehyun Pak, Yuseok Ban, and Sangyoun Lee.  Adacof: Adaptive collaboration of flows for video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5316–5325, 2020.
  * [35] Siyao Li, Shiyu Zhao, Weijiang Yu, Wenxiu Sun, Dimitris N. Metaxas, Chen Change Loy, and Ziwei Liu.  Deep animation video interpolation in the wild.  In CVPR, 2021.
  * [36] Yinxiao Li, Pengchong Jin, Feng Yang, Ce Liu, Ming-Hsuan Yang, and Peyman Milanfar.  Comisr: Compression-informed video super-resolution.  In ICCV, 2021.
  * [37] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In CVPR, 2021.
  * [38] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [39] Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, and Yung-Yu Chuang.  Deep video frame interpolation using cyclic frame generation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8794–8802, 2019.
  * [40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [41] Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala.  Video frame synthesis using deep voxel flow.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4463–4471, 2017.
  * [42] Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia.  Video frame interpolation with transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3532–3542, 2022.
  * [43] Nikolaus Mayer, Eddy Ilg, Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox.  A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation.  In CVPR, 2016.
  * [44] Simon Meister, Junhwa Hur, and Stefan Roth.  Unflow: Unsupervised learning of optical flow with a bidirectional census loss.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [45] Simone Meyer, Abdelaziz Djelouah, Brian McWilliams, Alexander Sorkine-Hornung, Markus Gross, and Christopher Schroers.  Phasenet for video frame interpolation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 498–507, 2018.
  * [46] Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, and Alexander Sorkine-Hornung.  Phase-based frame interpolation for video.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1410–1418, 2015.
  * [47] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In 2016 fourth international conference on 3D vision (3DV), pages 565–571. Ieee, 2016.
  * [48] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee.  Deep multi-scale convolutional neural network for dynamic scene deblurring.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3883–3891, 2017.
  * [49] Simon Niklaus and Feng Liu.  Context-aware synthesis for video frame interpolation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1701–1710, 2018.
  * [50] Simon Niklaus and Feng Liu.  Softmax splatting for video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5437–5446, 2020.
  * [51] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive convolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 670–679, 2017.
  * [52] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive separable convolution.  In Proceedings of the IEEE international conference on computer vision, pages 261–270, 2017.
  * [53] Junheum Park, Chul Lee, and Chang-Su Kim.  Asymmetric bilateral motion estimation for video frame interpolation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14539–14548, 2021.
  * [54] Tomer Peleg, Pablo Szekely, Doron Sabo, and Omry Sendik.  Im-net for high resolution video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2398–2407, 2019.
  * [55] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724–732, 2016.
  * [56] Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, Christoph Feichtenhofer, and Jitendra Malik.  On the benefits of 3d pose and tracking for human action recognition.  In CVPR, 2023.
  * [57] Anurag Ranjan, David T. Hoffmann, Dimitrios Tzionas, Siyu Tang, Javier Romero, and Michael J. Black.  Learning multi-human optical flow.  Int. J. Comput. Vis., 128(4):873–890, 2020.
  * [58] Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless.  Film: Frame interpolation for large motion.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII, pages 250–266. Springer, 2022.
  * [59] Fitsum A Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin J Shih, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  Unsupervised video interpolation using cycle consistency.  In Proceedings of the IEEE/CVF international conference on computer Vision, pages 892–900, 2019.
  * [60] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [61] Hyeonjun Sim, Jihyong Oh, and Munchurl Kim.  Xvfi: Extreme video frame interpolation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14489–14498, 2021.
  * [62] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [63] Shuochen Su, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang.  Deep video deblurring for hand-held cameras.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1279–1288, 2017.
  * [64] Masato Tamura, Rahul Vishwakarma, and Ravigopal Vennelakanti.  Hunting group clues with transformers for social group activity recognition.  In ECCV, 2022.
  * [65] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.
  * [66] Stepan Tulyakov, Alfredo Bochicchio, Daniel Gehrig, Stamatios Georgoulis, Yuanyou Li, and Davide Scaramuzza.  Time lens++: Event-based frame interpolation with parametric non-linear flow and multi-scale fusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17755–17764, 2022.
  * [67] Stepan Tulyakov, Daniel Gehrig, Stamatios Georgoulis, Julius Erbach, Mathias Gehrig, Yuanyou Li, and Davide Scaramuzza.  Time lens: Event-based video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16155–16164, 2021.
  * [68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [69] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [70] Chao-Yuan Wu, Nayan Singhal, and Philipp Krähenbühl.  Video compression through image interpolation.  In ECCV, 2018.
  * [71] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [72] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim.  Space-time neural irradiance fields for free-viewpoint video.  In CVPR, 2021.
  * [73] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao.  Gmflow: Learning optical flow via global matching.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8121–8130, 2022.
  * [74] Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang.  Quadratic video interpolation.  Advances in Neural Information Processing Systems, 32, 2019.
  * [75] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao.  Vitpose: Simple vision transformer baselines for human pose estimation.  arXiv preprint arXiv:2204.12484, 2022.
  * [76] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman.  Video enhancement with task-oriented flow.  International Journal of Computer Vision, 127(8):1106–1125, 2019\.
  * [77] Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa.  Decoupling human and camera motion from videos in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
  * [78] Hangjie Yuan, Dong Ni, and Mang Wang.  Spatio-temporal dynamic inference network for group activity recognition.  In ICCV, 2021.
  * [79] Honglu Zhou, Asim Kadav, Aviv Shamsian, Shijie Geng, Farley Lai, Long Zhao, Ting Liu, Mubbasir Kapadia, and Hans Peter Graf.  Composer: Compositional reasoning of group activity in videos with keypoint-only modality.  In ECCV, 2022.