# Rolling Shutter Inversion: Bring Rolling Shutter Images to High Framerate
Global Shutter Video

Bin Fan, Yuchao Dai, and Hongdong Li, Bin Fan and Yuchao Dai are with School
of Electronics and Information, Northwestern Polytechnical University, Xi’an,
710129, China. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Yuchao
Dai is the corresponding author. Hongdong Li is with School of Computing, the
Australian National University, Canberra, Australia. E-mail:
<EMAIL_ADDRESS>
###### Abstract

A single rolling-shutter (RS) image may be viewed as a row-wise combination of
a sequence of global-shutter (GS) images captured by a (virtual) moving GS
camera within the exposure duration. Although rolling-shutter cameras are
widely used, the RS effect causes obvious image distortion especially in the
presence of fast camera motion, hindering downstream computer vision tasks. In
this paper, we propose to invert the rolling-shutter image capture mechanism,
i.e., recovering a continuous high framerate global-shutter video from two
time-consecutive RS frames. We call this task the RS temporal super-resolution
(RSSR) problem. The RSSR is a very challenging task, and to our knowledge, no
practical solution exists to date. This paper presents a novel deep-learning
based solution. By leveraging the multi-view geometry relationship of the RS
imaging process, our learning based framework successfully achieves high
framerate GS generation. Specifically, three novel contributions can be
identified: (i) novel formulations for bidirectional RS undistortion flows
under constant velocity as well as constant acceleration motion model. (ii) a
simple linear scaling operation, which bridges the RS undistortion flow and
regular optical flow. (iii) a new mutual conversion scheme between varying RS
undistortion flows that correspond to different scanlines. Our method also
exploits the underlying spatial-temporal geometric relationships within a deep
learning framework, where no additional supervision is required beyond the
necessary middle-scanline GS image. Building upon these contributions, this
paper represents the very first rolling-shutter temporal super-resolution
deep-network that is able to recover high framerate global-shutter videos from
just two RS frames. Extensive experimental results on both synthetic and real
data show that our proposed method can produce high-quality GS image sequences
with rich details, outperforming the state-of-the-art methods.

###### Index Terms:

Rolling-shutter, image correction, temporal super-resolution, end-to-end
learning, geometric vision.

##  1 Introduction

Most consumer-grade cameras are built upon CMOS sensors using a rolling-
shutter (RS) mechanism. Unlike its global-shutter (GS) counterpart, the
rolling-shutter camera captures an image frame in a sequential row-by-row
manner, which invariably leads to the so-called RS effect (_e.g._ ,
stretching, wobbling) in the obtained images and videos when the camera moves
relative to the scene. This RS effect is becoming a nuisance in photography
and in downstream computer vision tasks. Simply ignoring the RS effect often
results in performance degradation or even failure for many real-world
applications [1, 2, 3, 4, 5].

Fortunately, we made an important observation in this paper. Namely, we
recognize that a single RS image actually captures richer temporal dynamics
information about the underlying relative motion between the scene and the
camera. This dynamic information, hidden in the RS scanlines, has not been
properly utilized by almost all previous works on RS cameras. By exploiting
this hidden-and-previously overlooked information, one could possibly remove
the RS artifact, and this is the key insight of this paper.

Recovering the above dynamic information from a single RS image, while doable
in theory, is extremely challenging in practice due to the severe ill-
posedness of the task (_e.g._ , [6, 7, 8, 9]). On the other hand, using
multiple RS frames (_e.g._ , [10, 11, 12, 13]) would make the task more
tractable, but at the cost of increased computation. In this paper, we take a
middle ground, using a minimum of two consecutive rolling-shutter frames for
the task, in order to strike a balance between recovery performance and
efficiency (see also [14, 15, 16, 17]).

Intuitively, given a pair of consecutive rolling-shutter frames, humans seem
to be able to infer a plausible explanation for the underlying geometry and
temporal dynamics (_i.e._ , camera motion and scene structure). Hence,
recovering the underlying geometry from consecutive rolling-shutter images
seems to be possible. Towards this goal, this paper advocates recovering a
continuous temporal sequence of latent global-shutter frames from two
consecutive RS images, _i.e._ , this task requires solving and subsequently
inverting the underlying rolling-shutter geometry.

Figure 1: A rolling-shutter image may be generated by continuously extracting
scanlines from continuous global-shutter images row by row and one at a time.
In contrast, our RSSR pipeline essentially inverts this process, _i.e._ ,
reconstructing the latent GS image sequence, frame by frame, from two
consecutive RS images. The recovered GS video (shown in the bottom right
position of the figure) can be viewed in the arXiv version of [18].

To the best of our knowledge, this task (which we call RSSR, stands for
Rolling-Shutter temporal Super-Resolution) is a novel computer vision task,
and has not been studied before. Moreover, existing RS correction methods
(_e.g._ , [15]) are unable to solve this task, because of the need to estimate
small and subtle intra-frame motions, and the need for non-trivial camera
calibration and sophisticated optimization procedures. So far, we have not
seen any deep-learning based solution for this task either.

In this paper, we propose a novel deep-learning based method to invert the
rolling-shutter image capture mechanism, _i.e._ , which recovers a continuous
global-shutter video sequence from two consecutive rolling-shutter images. The
concept of our method is illustrated in Fig. 1. This inversion process
resembles the aforementioned human ability of inferring temporal dynamics from
consecutive RS frames.

In the literature, the following two classes of existing works are mostly
related to our method: (i) GS frame interpolation (_e.g._ , [19, 20, 21]) and
(ii) two-image RS correction (_e.g._ , [14, 15, 16]). However, we would like
to point out that our new method is significantly different from both classes
of methods. For example, our method achieves RS-aware pixel displacement by
scaling the corresponding optical flow vector under the constant velocity
motion model, whereas the GS frame interpolation methods often rely on small
and controllable pixel displacement and hence fall short when dealing with the
RS effect. For another example, our method is free from the unsmoothness or
the ghosting artifact which often hinders the traditional RS correction
methods. Although existing deep-learning based RS correction methods have
achieved impressive performance, by using them only one single GS frame can be
recovered (_e.g._ , DeepUnrollNet [14], or See Subsection 6.10 for more
examples). In contrast, our method produces a visually fluid GS video
sequence.

Our method does more than RS correction. Besides eliminating geometric
distortion, we also recover a set of high framerate GS images in order, for
example, to recover 960 GS images from two 480-height RS images to get a GS
video with 480$\times$ frame rate up-conversion. This is particularly
challenging as one has to ensure the temporal smoothness of the recovered
video sequence.

We formulate the bidirectional RS undistortion flows to characterize the
pixel-wise RS-aware pixel displacement, and further advance two calculation
methods for the mutual conversion between varying RS undistortion flows
corresponding to different scanlines, _i.e._ , constant velocity propagation
and constant acceleration propagation. To this end, we prove that the scaling
factor is in the interval of $(-1,1)$ when correcting an RS image to its
middle-scanline GS image. As a result, we propose a data-driven solution for
RSSR with good interpretability, which intrinsically encapsulates the
underlying RS geometry that more sophisticated methods (_e.g._ , [15, 16])
struggle to learn. Also, we propose to estimate the camera acceleration to
improve the accuracy and adaptation of GS video sequence recovery.

Our proposed geometry-aware RSSR pipeline employs a cascaded architecture to
extract a latent high framerate GS video sequence from two consecutive RS
images. Firstly, we estimate the bidirectional optical flows by using the
classic PWC-Net [22]. Secondly, we design an encoder-decoder UNet network to
learn the scaling factor of each pixel (_i.e._ , the middle-scanline
correlation map in Subsection 4.3) such that the middle-scanline RS
undistortion flows can be inferred. Meanwhile, the RS undistortion flows for
any scanline can be associated and propagated explicitly. Finally, the softmax
splatting [20] is used to produce the high framerate GS video frames at
arbitrary scanlines. Our vanilla RSSR network can be trained end-to-end and
only the middle-scanline GS images are needed for supervision. Additionally,
the first-scanline GS images can be leveraged as a supervisor to estimate the
camera acceleration, thereby obtaining better RS inversion results. Since none
of the learned network parameters are time-dependent, it can synthesize as
many GS frames as needed, _i.e._ , yielding a GS video with an arbitrary
framerate. It is also worth mentioning that our approach has a very high
efficiency of RS inversion due to the geometry-aware propagation. Extensive
experiment results on benchmark datasets demonstrate that our approach not
only outperforms state-of-the-art methods in both RS effect removal and
inference efficiency, but also produces a smooth and coherent video sequence
surpassing all previous methods.

A preliminary version of this paper appeared in [18]. In this extended
version, 1) we extend the formulation to a more realistic constant
acceleration motion model and then propose an alternative but promising
Acceleration-Net to further recover higher quality GS video frames; 2) we
provide more experimental analyses on DeepUnrollNet [14]; 3) we demonstrate
the advantages of our approach over the state-of-the-art video frame
interpolation methods and the two-stage methods; 4) we provide a more
comprehensive generalization and robustness analysis; 5) we conduct a more in-
depth comparison with the analytic solution in [15]; 6) we confirm the
effectiveness of our approach for 3D reconstruction. Note that the extension
of our constant acceleration propagation is also geometrically driven.

To summarize, our main contributions are as follows:

  * •

We give a detailed proof to the scanline-dependent bidirectional RS
undistortion flows.

  * •

We propose the first learning-based RSSR solution for latent global-shutter
video sequence recovery.

  * •

We develop an effective Acceleration-Net for constant acceleration
propagation, enhancing the generality and accuracy of constant velocity
propagation in reverting rolling-shutter cameras.

##  2 Related Work

Video frame interpolation (VFI). After extensive research in recent years, VFI
can be roughly classified into phase-based [23, 24], kernel-based [25, 26] and
flow-based [27, 19, 20] approaches. Notably, with advances in optical flow
estimation [22, 28], flow-based VFI methods have been actively adopted to
explicitly exploit motion information. Not only limited to linear flow
interpolation [19, 29, 20], many variants are dedicated to better intermediate
flow estimation, such as quadratic [30], rectified quadratic [31], and cubic
[32] approximations. Moreover, the initial flow field is enhanced in DAIN [21]
via a depth-aware flow projection layer. The symmetric motion field is
constructed in BMBC [27] to generate the intermediate flow directly. On the
other side, more attention is paid to detail refinement and fusion, including
contextual warping [29, 21], occlusion inference [33, 19], cycle constraints
[34, 35], joint deblurring and frame rate up-conversion [36], and softmax
splatting [20] for more efficient forward warping, _etc._ Unfortunately, these
VFI methods are tailored to GS cameras, which cannot correctly synthesize in-
between frames for RS inputs. In contrast, our method can not only efficiently
interpolate scanline-arbitrary intermediate frames, but also can successfully
remove the RS artifacts.

Traditional non-learning RS correction. Over the last decade, several works
have revisited the RS geometric model to remove the RS effect [2, 8, 37, 12,
5, 38, 39, 40, 41, 42]. Grundmann _et al._ [43] employed a homography mixture
to achieve joint RS removal and video stabilization. The occlusion-aware
undistortion method [10] removed the depth-dependent RS distortions from a
specific setting of $\geq 3$ RS images, assuming a piece-wise planar 3D scene.
Zhuang _et al._ [15] proposed estimating the full camera motion by a
differential formulation to remove RS distortions in two consecutive RS
images. Such a model has achieved considerable success, but further
improvements have appeared challenging, due to the difficulties of making it
robust and efficient to various situations (_e.g._ , relying too much on the
initial optical flow estimation [16]). Subsequently, Zhuang and Tran [16] put
forward a differential RS homography to model the scanline-varying RS camera
poses, which can be used to perform RS-aware image stitching and correction.
Fan _et al._ [13] extended [15] and presented an RS-stereo-aware differential
epipolar constraint to correct the RS stereo image pairs. Albl _et al._ [44]
explored a simple two-camera rig with different shutter directions to
undistort the RS images acquired by a smartphone. Wu _et al._ [45] dealt with
video stabilization and RS removal jointly.

Deep-learning based RS correction. Recently, convolutional neural networks
(CNN) have been used to achieve more flexible and efficient RS correction.
Rengarajan _et al._ [6] proposed the first CNN to correct a single RS image by
assuming a simple affine model. Zhuang _et al._ [7] extended [6] to learn the
RS geometry from a single frame through two independent networks, yielding
geometrically more correct results. Zhong _et al._ [11] simultaneously handled
RS correction and deblurring. Liu _et al._ [14] and Fan _et al._ [17] used two
consecutive RS images as input and designed specialized CNNs to predict one GS
image. To the best of our knowledge, our RSSR model is the first that is
developed to learn the mapping from two consecutive RS frames to high
framerate GS video frames corresponding to any scanline.

##  3 Differential Forward RS Geometry

GS-aware forward warping. Assuming that the GS camera experiences constant
linear velocity $\mathbf{v}=[v_{1},v_{2},v_{3}]^{\text{T}}$ and angular
velocity $\bm{\omega}=[\omega_{1},\omega_{2},\omega_{3}]^{\text{T}}$ over two
consecutive frames 1 and 2, a 3D point $\mathbf{X}$ with depth $Z$ is observed
by the camera to move with the 3D velocity as
$-{\mathbf{v}}-{\bm{\omega}}\times{\mathbf{X}}$ [46]. Projecting this 3D
velocity into the 2D image plane yields the image motion field ${\mathbf{f}}$,
which is usually approximated by the optical flow vector
$({\mathbf{f}}_{u},{\mathbf{f}}_{v})^{\text{T}}$ under the brightness
constancy assumption, at pixel ${\mathbf{x}}=(x,y)$ as [46]:

${\mathbf{f}}=\frac{{\mathbf{A}}{\mathbf{v}}}{Z}+{\mathbf{B}}{\bm{\omega}}\buildrel\Delta\over{=}\pi({\mathbf{v}},{\bm{\omega}},{\mathbf{x}},Z,f),$
(1)

where

$\begin{split}{\mathbf{A}}&=\left[\begin{array}[]{ccc}-{f}&0&x\\\
0&-{f}&y\end{array}\right],\\\
{\mathbf{B}}&=\left[\begin{array}[]{ccc}\frac{xy}{{f}}&-\left(f+\frac{x^{2}}{{f}}\right)&y\\\
\left(f+\frac{y^{2}}{{f}}\right)&-\frac{xy}{{f}}&-x\end{array}\right].\end{split}$
(2)

Here, $(x,y)$ is the normalized image coordinate and $f$ denotes the focal
length.

RS forward motion parameterization. Since all RS scanlines are successively
and instantaneously exposed one by one at different times, each scanline
possesses a different optical center. To account for the scanline-varying
camera poses, we make full use of the small motion hypothesis. Given the
inter-frame camera velocities $({\mathbf{v}},{\bm{\omega}})$ between the two
first scanlines of two consecutive RS images, the subtle intra-frame relative
motion could be obtained by interpolation [47, 38, 15, 16]. Specifically, [15]
derived the linear interpolation scheme under the assumption of constant
velocity motion and the quadratic interpolation scheme under the assumption of
constant acceleration motion. Formally, the camera position and rotation
$({\mathbf{p}}_{1}^{s_{1}},{\mathbf{r}}_{1}^{s_{1}})$ (resp.
$({\mathbf{p}}_{2}^{s_{2}},{\mathbf{r}}_{2}^{s_{2}})$) of the $s_{1}$-th
(resp. $s_{2}$-th) scanline in frame 1 (resp. 2) _w.r.t._ the first scanline
of frame 1 can be expressed as:

$\begin{array}[]{ll}\mathbf{p}_{1}^{s_{1}}=\lambda_{1}^{s_{1}}\mathbf{v},&\mathbf{r}_{1}^{s_{1}}=\lambda_{1}^{s_{1}}\bm{\omega},\\\
\mathbf{p}_{2}^{s_{2}}=\lambda_{2}^{s_{2}}\mathbf{v},&\mathbf{r}_{2}^{s_{2}}=\lambda_{2}^{s_{2}}\bm{\omega},\end{array}$
(3)

where

$\left\\{\begin{aligned} \lambda_{1}^{s_{1}}&=\frac{\gamma s_{1}}{h}\\\
\lambda_{2}^{s_{2}}&=1+\frac{\gamma s_{2}}{h}\end{aligned}\right.,$ (4)

are for the constant velocity motion assumption, or

$\left\\{\begin{aligned} \lambda_{1}^{s_{1}}&=\frac{2}{k+2}\left(\frac{\gamma
s_{1}}{h}+\frac{k}{2}\left(\frac{\gamma s_{1}}{h}\right)^{2}\right)\\\
\lambda_{2}^{s_{2}}&=\frac{2}{k+2}\left(1+\frac{\gamma
s_{2}}{h}+\frac{k}{2}\left(1+\frac{\gamma
s_{2}}{h}\right)^{2}\right)\end{aligned}\right.,$ (5)

are for the constant acceleration motion assumption. Note that here,
$\gamma={h\tau_{r}}/{{\tau}}$ represents the readout time ratio [15], _i.e._ ,
the ratio between the total readout time and the total time $\tau$ in an RS
frame, which can be calibrated by [48, 49]. $\tau_{r}$ is the readout time of
a single scanline and $h$ is the total number of scanlines in an image. $k$
represents an extra unknown motion parameter describing the acceleration,
denoted as the acceleration factor ($k>0$ for acceleration and $k<0$ for
deceleration). It is easy to verify that Eq. (5) will reduce to Eq. (4) when
the acceleration factor vanishes, _i.e._ , $k=0$. Consequently, the relative
motion between $s_{1}$-th and $s_{2}$-th scanlines satisfies:

$\displaystyle\mathbf{v}_{s_{1}s_{2}}$
$\displaystyle=\mathbf{p}_{2}^{s_{2}}-\mathbf{p}_{1}^{s_{1}}=\left(\lambda_{2}^{s_{2}}-\lambda_{1}^{s_{1}}\right)\mathbf{v},$
(6) $\displaystyle\bm{\omega}_{s_{1}s_{2}}$
$\displaystyle=\mathbf{r}_{2}^{s_{2}}-\mathbf{r}_{1}^{s_{1}}=\left(\lambda_{2}^{s_{2}}-\lambda_{1}^{s_{1}}\right)\bm{\omega}.$

Further details can be found in [15].

RS-aware forward warping. As ${\mathbf{f}}_{v}=s_{2}-s_{1}$, plugging Eqs. (4)
and (6) into Eq. (1), we can relate the forward optical flow of pixel
${\mathbf{x}}$ in frame 1 in terms of the RS camera velocity and the 3D point
as:

$\left[{\begin{array}[]{*{20}{c}}{\mathbf{f}}_{u}\\\
{\mathbf{f}}_{v}\end{array}}\right]=\alpha\left[{\begin{array}[]{*{20}{c}}\pi_{u}({\mathbf{v}},{\bm{\omega}},{\mathbf{x}},Z,f)\\\
\pi_{v}({\mathbf{v}},{\bm{\omega}},{\mathbf{x}},Z,f)\end{array}}\right],$ (7)

where

$\alpha=1+\frac{\gamma{\mathbf{f}}_{v}}{h}$ (8)

represents the RS-aware forward interpolation factor under the constant
velocity motion model, which depends on the scanline involved in the optical
flow. $\pi_{u}(\cdot)$ and $\pi_{v}(\cdot)$ denotes the first and second
entries of $\pi(\cdot)$, respectively. Hence, the geometric inaccuracies
induced by the RS effect could be compensated by simply scaling the optical
flow vector of each pixel ${\mathbf{x}}$. Note that this paper does not
investigate the RS-aware optical flow modeling under the assumption of
constant acceleration motion due to its complexity.

##  4 RS Undistortion Flow vs. Optical Flow

###  4.1 RS imaging mechanism

As shown in Fig. 1, the RS camera exposes each scanline in sequence, which
results in a different local frame for each scanline. The RS image
$\mathbf{I}_{r}$ can therefore be regarded as the result of successive row-by-
row combinations of virtual GS image sequences over the period of camera
readout time, _i.e._ , formulating the RS imaging model as:

$\left\lfloor\mathbf{I}_{r}(\mathbf{x})\right\rfloor_{s}=\left\lfloor\mathbf{I}^{s}_{g}(\mathbf{x})\right\rfloor_{s},\,\,\,0\leq
s\leq h-1,$ (9)

where $\mathbf{I}^{s}_{g}$ is the virtual GS image captured at time
$s\tau_{r}$. $\lfloor\cdot\rfloor_{s}$ indicates extracting the $s$-th
scanline. On the contrary, the RSSR task aims to reverse the above RS imaging
formulation, _i.e._ , estimating the displacement vectors ${\mathbf{u}}_{r\to
s}$ of pixel ${\mathbf{x}}$ from the RS image to the virtual GS image at
$s$-th scanline such that

$\mathbf{I}_{r}(\mathbf{x})=\mathbf{I}^{s}_{g}(\mathbf{x}+\mathbf{u}_{r\to
s}),\,\,\,0\leq s\leq h-1.$ (10)

Specifically, it is to estimate the dense _RS undistortion flow_
${\mathbf{U}}_{r\to s}$ that corresponds to scanline $s$ by stacking
${\mathbf{u}}_{r\to s}$ for all pixels in matrix form. Note that similar to
[14], ${\mathbf{U}}_{r\to s}$ is a forward warping operation, _e.g._ , softmax
splatting [20].

Figure 2: Generality of scanline pose interpolation given the more realistic
constant acceleration motion assumption. Here, similar to [16], we assume
$\gamma=1$, $h=480$, a unit displacement length between two first scanlines,
and different values of acceleration factor $k$.

###  4.2 Bidirectional RS undistortion flows

To deliver each RS pixel ${\mathbf{x}}$ on the $\eta$-th scanline in frame 1
to its GS canvas defined by the pose corresponding to $s$-th scanline of frame
1, the RS-aware forward warping displacement vector of pixel ${\mathbf{x}}$
can be formulated as:

$\left[{\begin{array}[]{*{20}{c}}{\mathbf{u}}_{u}\\\
{\mathbf{u}}_{v}\end{array}}\right]=\beta\left[{\begin{array}[]{*{20}{c}}\pi_{u}({\mathbf{v}},{\bm{\omega}},{\mathbf{x}},Z,f)\\\
\pi_{v}({\mathbf{v}},{\bm{\omega}},{\mathbf{x}},Z,f)\end{array}}\right],$ (11)

where

$\beta=\frac{\gamma(s-\eta)}{h}$ (12)

represents the RS-aware forward undistortion factor under the constant
velocity motion model, which depends on the scanline offset between the target
scanline $s$ and the current scanline $\eta$. Note that $0\leq s,\eta\leq h-1$
and the pixels on the $s$-th scanline of the RS image remain unchanged.
Stacking the forward warping displacement vectors of all pixels in frame 1, we
can obtain the pixel-wise forward RS undistortion flow $\mathbf{U}_{1\to s}$
of frame 1, which could be used to restore the latent GS image corresponding
to any scanline $s\in[0,h-1]$ of frame 1.

Unfortunately, Eqs. (7) and (11) describe merely the forward optical flow
(_i.e._ , from frame 1 to frame 2) and the forward RS undistortion flow
(_i.e._ , from frame 1 to its scanline $s$), respectively. Given two input
consecutive RS images, one cannot remove the geometric RS distortion in frame
2 if only using the RS-aware forward warping alone. Therefore, we further
propose an _RS-aware backward warping model_ accounting for frame 2, as shown
in Proposition 1.

###### Proposition 1

The RS-aware backward warping, _i.e._ , the backward optical flow from frame 2
to frame 1 and the backward RS undistortion flow from frame 2 to its scanline
$s$, can be modeled by simply taking a negative readout time ratio under the
constant velocity motion assumption.

###### Proof 1

To formulate the RS-aware backward warping accounting for frame 2, the
backward inter-frame camera velocities
$({\mathbf{v}}^{\prime},{\bm{\omega}}^{\prime})$ should obey:
${\mathbf{v}}^{\prime}=-{\mathbf{v}}$ and
${\bm{\omega}}^{\prime}=-{\bm{\omega}}$. Let $Z^{\prime}$ denote the depth of
each pixel ${\mathbf{x}}^{\prime}$ in frame 2 and
$({\mathbf{f}}^{\prime}_{u},{\mathbf{f}}^{\prime}_{v})$ the backward optical
flow vector from frame 2 to frame 1. In complete analogy with the RS forward
motion parameterizations in Section 3, we again derive the relative motion
between the scanline $s_{1}$ of frame 1 and the scanline $s_{2}$ of frame 2
as:

$\displaystyle\mathbf{v}_{s_{2}s_{1}}$
$\displaystyle=\left(\lambda_{2}^{s_{2}}-\lambda_{1}^{s_{1}}\right)\mathbf{v}^{\prime},$
(13) $\displaystyle\bm{\omega}_{s_{2}s_{1}}$
$\displaystyle=\left(\lambda_{2}^{s_{2}}-\lambda_{1}^{s_{1}}\right)\bm{\omega}^{\prime},$

where $\lambda_{1}^{s_{1}}$ and $\lambda_{1}^{s_{2}}$ satisfy Eq. (4) as well.
Note that ${\mathbf{f}}^{\prime}_{v}=s_{1}-s_{2}$. In the same way, we can
obtain RS-aware backward warping model for the backward optical flow at pixel
${\mathbf{x}}^{\prime}$ as:

$\left[{\begin{array}[]{*{20}{c}}{\mathbf{f}}^{\prime}_{u}\\\
{\mathbf{f}}^{\prime}_{v}\end{array}}\right]=\alpha^{\prime}\left[{\begin{array}[]{*{20}{c}}\pi_{u}({\mathbf{v}}^{\prime},{\bm{\omega}}^{\prime},{\mathbf{x}}^{\prime},Z^{\prime},f)\\\
\pi_{v}({\mathbf{v}}^{\prime},{\bm{\omega}}^{\prime},{\mathbf{x}}^{\prime},Z^{\prime},f)\end{array}}\right],$
(14)

where

$\alpha^{\prime}=1-\frac{\gamma{\mathbf{f}}^{\prime}_{v}}{h}$ (15)

represents the RS-aware backward interpolation factor under the constant
velocity motion model.

Furthermore, we derive the RS-aware backward warping displacement vector,
which transforms each RS pixel ${\mathbf{x}}^{\prime}$ on $\eta$-th scanline
of frame 2 to obtain a distortion-free frame defined by the pose of the $s$-th
scanline of frame 2 by

$\left[{\begin{array}[]{*{20}{c}}{\mathbf{u}}^{\prime}_{u}\\\
{\mathbf{u}}^{\prime}_{v}\end{array}}\right]=\beta^{\prime}\left[{\begin{array}[]{*{20}{c}}\pi_{u}({\mathbf{v}}^{\prime},{\bm{\omega}}^{\prime},{\mathbf{x}}^{\prime},Z^{\prime},f)\\\
\pi_{v}({\mathbf{v}}^{\prime},{\bm{\omega}}^{\prime},{\mathbf{x}}^{\prime},Z^{\prime},f)\end{array}}\right],$
(16)

where

$\beta^{\prime}=-\frac{\gamma(s-\eta)}{h}$ (17)

represents the RS-aware backward undistortion factor under the constant
velocity motion model. Similarly, the GS image corresponding to any scanline
$s\in[0,h-1]$ of frame 2 can be recovered.

Heretofore, we have obtained bidirectional RS undistortion flows
$\mathbf{U}_{1\to s}$ and $\mathbf{U}_{2\to s}$ under the assumption of
constant velocity motion, which can warp two consecutive RS images to their GS
counterparts corresponding to their respective scanlines
$s\in\left[0,h-1\right]$, _i.e._ , activating the rolling-shutter temporal
super-resolution. However, despite its simplicity in compensating for inter-
frame motion, as shown in Fig. 2, a more realistic constant acceleration
motion assumption might enhance the generality of the RSSR task in practice.
To this end, for the constant acceleration motion model, we utilize Eqs. (3)
and (5) and rewrite the bidirectional RS-aware undistortion factor in Eq. (12)
as:

$\beta=\frac{\gamma(s-\eta)}{h}\cdot\frac{2h+k\gamma(s-\eta)}{h(k+2)}.$ (18)

Note that due to the concise and well-defined characteristics of constant
velocity modeling, we will mainly focus on it for further analysis. Note also
that the constant acceleration formulation in this paper is mainly used to
further advance the accuracy and compatibility of mutual conversion between
varying RS undistortion flows.

Figure 3: RS undistortion flow versus regular optical flow. Here, forward
flows are visualized according to [50]. Compared to the isotropically smooth
optical flow map, the RS undistortion flow map exhibits a more significant
scanline dependence. On the one hand, the RS undistortion flows near the
target scanline appear as lighter colors (_i.e._ , smaller warping
displacement values). On the other hand, the RS undistortion flows
corresponding to pixels smaller than and larger than the target scanline show
different colors (_i.e._ , different warping displacement directions).

###  4.3 Connection between RS undistortion flow and optical flow

Note that the optical flow in Eq. (7) exhibits the pixel displacement over two
consecutive RS frames, while the RS undistortion flow in Eq. (11) models the
pixel displacement between the RS frame 1 (or frame 2) and the GS frame at
scanline $s$. Without loss of generality, taking the forward warping as an
example (note that the backward warping is similar, except that $\gamma$ is
negative). First, eliminating ${\mathbf{f}}_{v}$ on the right hand side of Eq.
(7), the optical flow of pixel ${\mathbf{x}}$ can be rewritten as:

$\left[{\begin{array}[]{*{20}{c}}{\mathbf{f}}_{u}\\\
{\mathbf{f}}_{v}\end{array}}\right]=\frac{h}{h-\gamma\pi_{v}}\left[{\begin{array}[]{*{20}{c}}\pi_{u}\\\
\pi_{v}\end{array}}\right],$ (19)

where $\pi_{u}$ and $\pi_{v}$ are abbreviations, determined by camera
parameters, camera motions and 3D depths. Then, through Eqs. (11), (12) and
(19), we establish the connection between the forward/backward RS undistortion
flow $({\mathbf{u}}_{u},{\mathbf{u}}_{v})^{\text{T}}$ and the forward/backward
optical flow $({\mathbf{f}}_{u},{\mathbf{f}}_{v})^{\text{T}}$ at pixel
$\mathbf{x}$ as:

$\left[{\begin{array}[]{*{20}{c}}{\mathbf{u}}_{u}\\\
{\mathbf{u}}_{v}\end{array}}\right]=c\left[{\begin{array}[]{*{20}{c}}{\mathbf{f}}_{u}\\\
{\mathbf{f}}_{v}\end{array}}\right],$ (20)

where

$c=\frac{\gamma(s-\eta)(h-\gamma\pi_{v})}{h^{2}}$ (21)

denotes the forward/backward correlation factor between bidirectional RS
undistortion flow and optical flow, which are distinguished by the sign of
$\gamma$. Stacking $c$ for all pixels yields the forward and backward
correlation maps ${\mathbf{C}}_{1\to s}$ and ${\mathbf{C}}_{2\to s}$,
respectively. The correlation map thus indicates the underlying RS geometry,
_i.e._ , camera motion and scene structure. Note that the connection under the
constant acceleration motion is not discussed in this paper.

Consequently, after obtaining the bidirectional optical flows between two
consecutive RS frames (_e.g._ via PWC-Net [22] or RAFT [28]), we can scale
them to obtain the bidirectional RS undistortion flows to further recover the
latent GS images corresponding to specific scanline $s\in[0,h-1]$. Note that
the size and sign of the scaling factor (_i.e._ , correlation factor $c$) rely
on the underlying RS geometry. As illustrated in Fig. 3, we can intuitively
observe that, compared to the isotropically smooth optical flow map, the RS
undistortion flow is more typically scanline-dependent. The closer the pixel
to scanline $s$ (assuming to correction to $s$-th scanline, _e.g._ , the first
scanline), the smaller the value of the RS undistortion flow is generally.
Moreover, we point out that Eq. (20) is essentially the core of RS correction
in [15], but it requires sophisticated processing (such as RANSAC [51], non-
convex optimization, _etc._) to estimate accurate camera motion and scene
depth based on the optical flow map. Nevertheless, potential gross errors in
optical flow estimates could lead to severe artifacts in the texture-less or
non-overlapping regions [16], as elaborated in Subsection 6.4.

###### Proposition 2

Given bidirectional optical flows ${\mathbf{F}}_{1\to 2}$ and
${\mathbf{F}}_{2\to 1}$, to recover the GS image corresponding to the camera
pose of the middle scanline, the forward correlation map ${\mathbf{C}}_{1\to
m}$ (resp. backward correlation map ${\mathbf{C}}_{2\to m}$) for the forward
RS undistortion flow ${\mathbf{U}}_{1\to m}$ (resp. backward RS undistortion
flow ${\mathbf{U}}_{2\to m}$) in Eq. (20) satisfies:

$\left\lfloor{\mathbf{C}}_{1\to
m}\right\rfloor_{i,j}\in\left\\{\begin{array}[]{ll}(0,1)&\textrm{if
$i<\frac{h}{2}$}\\\ 0&\textrm{if $i=\frac{h}{2}$}\\\ (-1,0)&\textrm{if
$i>\frac{h}{2}$}\\\ \end{array}\right.,$ (22)

resp.

$\left\lfloor{\mathbf{C}}_{2\to
m}\right\rfloor_{i,j}\in\left\\{\begin{array}[]{ll}(-1,0)&\textrm{if
$i<\frac{h}{2}$}\\\ 0&\textrm{if $i=\frac{h}{2}$}\\\ (0,1)&\textrm{if
$i>\frac{h}{2}$}\\\ \end{array}\right.,$ (23)

where $\lfloor\cdot\rfloor_{i,j}$ is the entry in $i$-th row and $j$-th
column.

###### Proof 2

We first decompose $c_{m}$ defined in Eq. (21) into

$c_{m}=\frac{s-\eta}{h}\cdot\frac{\gamma(h-\gamma\pi_{v})}{h}\buildrel\Delta\over{=}c_{m}^{a}\cdot
c_{m}^{b}.$ (24)

Taking the case of $i<\frac{h}{2}$ in Eq. (22) as an example, _i.e._ ,
$0\leq\eta<\frac{h}{2}$. Since the GS image corresponding to the middle
scanline is to be restored, $s=\frac{h}{2}$. Thus,
$0<c_{m}^{a}\leq\frac{1}{2}$. In practice, $\pi_{v}$ indicates the prediction
of the inter-frame vertical optical flow value (see Eq. (1)), which is usually
much smaller than the total number of image scanlines $h$. Also, the readout
time ratio $\gamma$ is less than or equal to 1 for a normal RS camera [15, 5,
52], so $\gamma\in(0,1]$ when modeling the RS-aware forward warping. We thus
arrive at $0<c_{m}^{b}<2$. To sum up, $c_{m}\in\left(0,1\right)$, _i.e._ ,
$\left\lfloor{\mathbf{C}}_{1\to m}\right\rfloor_{i,j}\in\left(0,1\right)$
holds when $i<\frac{h}{2}$. Other cases can be proved similarly. Note that
$\gamma\in[-1,0)$ when modeling the RS-aware backward warping.

###  4.4 Interconversion between varying RS undistortion flows

Eq. (11) defines the RS undistortion flow that warps the RS frame to its GS
counterpart corresponding to scanline $s$. Assuming that two GS images
corresponding to the $s_{1}$-th scanline and the $s_{2}$-th scanline are to be
restored, it can be seen from Eqs. (11) and (12) that the difference in the
warping displacement vector of the same pixel ${\mathbf{x}}$ is only
$(s_{1}-\eta)$ and $(s_{2}-\eta)$ under the constant velocity motion model.
Note that ${\mathbf{x}}$ lies on scanline $\eta$. As a consequence, we can
simply extend the RS undistortion flow of pixel ${\mathbf{x}}$ facing scanline
$s_{1}$ to that for scanline $s_{2}$ by

$\left[{\begin{array}[]{*{20}{c}}{\mathbf{u}}_{u}^{s_{2}}\\\
{\mathbf{u}}_{v}^{s_{2}}\end{array}}\right]=\frac{s_{2}-\eta}{s_{1}-\eta}\left[{\begin{array}[]{*{20}{c}}{\mathbf{u}}_{u}^{s_{1}}\\\
{\mathbf{u}}_{v}^{s_{1}}\end{array}}\right].$ (25)

Analogously for constant acceleration motion model, based on Eqs. (11) and
(18), we can obtain:

$\left[{\begin{array}[]{*{20}{c}}{\mathbf{u}}_{u}^{s_{2}}\\\
{\mathbf{u}}_{v}^{s_{2}}\end{array}}\right]=\frac{(s_{2}-\eta)(2h+\varphi(s_{2}-\eta))}{(s_{1}-\eta)(2h+\varphi(s_{1}-\eta))}\left[{\begin{array}[]{*{20}{c}}{\mathbf{u}}_{u}^{s_{1}}\\\
{\mathbf{u}}_{v}^{s_{1}}\end{array}}\right],$ (26)

where $\varphi\buildrel\Delta\over{=}k\gamma$ denotes an auxiliary variable
that needs to be additionally estimated. Note that the constant velocity
propagation by Eq. (25) is a special case of the constant acceleration
propagation by Eq. (26) at $\varphi=0$ (_i.e._ , $k=0$). At this point, it can
be propagated explicitly by either Eq. (25) or Eq. (26), which is
differentiable, to obtain the RS undistortion flows corresponding to any
scanline.

In other words, the resulting RS undistortion flows corresponding to different
scanlines can be converted to each other, which is also the key to achieve our
RSSR solution. An example is shown in Fig. 3. One can find that the difference
between varying RS undistortion flows is not only reflected in the
distribution of the warping displacement size, but also the warping
displacement direction is closely related to the scanline.

Figure 4: Overview of our RSSR network architecture. Given two consecutive RS
frames, we first estimate the bidirectional optical flows. Then, we utilize an
encoder-decoder UNet architecture with skip connections, which is associated
with Eq. (27), to resolve the correlation maps. Next, the middle-scanline RS
undistortion flows can be explicitly calculated by Eq. (28), while being
certifiable. Finally, we adopt softmax splatting to generate the target
middle-scanline GS frames. Note that our main network is devised to predict
the latent GS images corresponding to the middle scanline during training. In
particular, in the test phase, the RS undistortion flows for any scanline
$s\in[0,h-1]$ can be propagated through Eq. (25) or Eq. (26) (_cf._ , dashed
arrow), followed by the recovery of the global-shutter image corresponding to
scanline $s$.

##  5 RS Temporal Super-Resolution

The objective of RS temporal super-resolution (RSSR) is to invert the rolling-
shutter imaging mechanism, _i.e._ , recovering a high framerate GS video from
two consecutive RS images. In this section, we will utilize the aforementioned
theoretical formulations, which reveal the intrinsic geometric properties of
the RS inversion task, to elaborate a concise and effective end-to-end CNN to
exploit the inherent regularity of data. Inspired by the cascaded architecture
for video frame interpolation in [19, 20, 21], we design a dedicated RSSR
network to encapsulate the complete underlying RS geometry, as illustrated in
Fig. 4.

Our network takes two consecutive RS images ${\mathbf{I}}_{r}^{1}$ and
${\mathbf{I}}_{r}^{2}$ as input and predicts the GS image corresponding to any
scanline (_i.e._ , generating a high framerate GS video). Our proposed vanilla
RSSR pipeline can be distilled down to two main submodules: the optical flow
estimation network $\mathcal{F}$ and the middle-scanline RS undistortion flow
estimation network $\mathcal{U}$. We first utilize $\mathcal{F}$ to obtain
bidirectional optical flows, and then encode the relation between optical
flows and middle-scanline RS undistortion flows by the middle-scanline
correlation maps over $\mathcal{U}$. Finally, we compute the middle-scanline
RS undistortion flows to produce two target middle-scanline GS frames by the
softmax splatting [20]. Below we describe each sub-network in detail.

Optical flow estimator. We employ the classical PWC-Net [22] as our optical
flow estimation network $\mathcal{F}$. As learning optical flow without ground
truth (GT) supervision is extremely difficult, we fine-tune $\mathcal{F}$ on
the RS benchmarks from the pre-trained model of PWC-Net in a self-supervised
manner [53, 54].

Figure 5: Architecture of our Acceleration-Net $\mathcal{A}$, which
corresponds to Eq. (26), _i.e._ , formulating the dashed arrow in Fig. 4. To
improve the generality and accuracy of constant velocity propagation,
$\mathcal{A}$ learns to predict the parameters of constant acceleration
propagation ($\varphi_{1},\varphi_{2}$ in particular).

Middle-scanline RS undistortion flow estimator. We adopt an encoder-decoder
UNet architecture [55, 19] with skip connections as our backbone network
$\mathcal{U}$ to predict the correlation maps, modifying the last layer to
output 6-channel predictions. To comply with Proposition 2, we fetch the first
two channels in the output of network $\mathcal{U}$, followed by a _Sigmoid_
operation to map to the interval of $(0,1)$, resulting in
$\hat{\mathbf{C}}_{1\to m}$ and $\hat{\mathbf{C}}_{2\to m}$. Afterward,
inspired by [14], we multiply them with their respective normalized scanline
offsets (_i.e._ , $\mathbf{T}_{1\to m}$ and $\mathbf{T}_{2\to m}$,
$\left\lfloor{\mathbf{T}}\right\rfloor_{i,j}\in[-1,1]$), which is defined as
the normalized scanline offset between the captured pixel and that of the
middle scanline. Note that ${\mathbf{T}}_{1\to m}=-{\mathbf{T}}_{2\to m}$, and
their values in middle-scanline are set to be small enough (not to be zero)
for subsequent flow propagation. The final correlation maps
${\mathbf{C}}_{1\to m}$ and ${\mathbf{C}}_{2\to m}$ thus can be recovered as:

$\begin{array}[]{ll}\mathbf{C}_{1\to m}={\mathbf{T}}_{1\to
m}\odot\hat{\mathbf{C}}_{1\to m},\\\ \mathbf{C}_{2\to m}={\mathbf{T}}_{2\to
m}\odot\hat{\mathbf{C}}_{2\to m},\end{array}$ (27)

where $\odot$ is an element-wise multiplier. It is easy to verify that they
all naturally fit for the theoretical bounds of Proposition 2. Furthermore, we
use the last four channels of network $\mathcal{U}$ to estimate optical flow
residuals $\Delta\mathbf{F}_{1\to 2}$ and $\Delta\mathbf{F}_{2\to 1}$ to
enhance the alignment of edge details and the generality of the proposed
model. Finally, according to Eq. (20), the bidirectional RS undistortion flows
$\mathbf{U}_{1\to m}$ and $\mathbf{U}_{2\to m}$ can be obtained as:

$\begin{array}[]{ll}\mathbf{U}_{1\to m}={\mathbf{C}}_{1\to
m}\odot\left(\mathbf{F}_{1\to 2}+\Delta\mathbf{F}_{1\to 2}\right),\\\
\mathbf{U}_{2\to m}={\mathbf{C}}_{2\to m}\odot\left(\mathbf{F}_{2\to
1}+\Delta\mathbf{F}_{2\to 1}\right).\end{array}$ (28)

Finally, the softmax splatting [20] is used to warp the RS image
$\mathbf{I}_{r}^{1}$ (resp. $\mathbf{I}_{r}^{2}$) to the target middle-
scanline GS image $\mathbf{I}_{g}^{1}$ (resp. $\mathbf{I}_{g}^{2}$) through
$\mathbf{U}_{1\to m}$ (resp. $\mathbf{U}_{2\to m}$).

Extension to GS images under arbitrary scanlines. As the resulting RS
undistortion flows corresponding to different scanlines can be converted to
each other, based on Eq. (25) directly or on Eq. (26) with additional
estimation of $k$, we can propagate the middle-scanline RS undistortion flows
obtained above to reach the RS undistortion flows $\mathbf{U}_{1\to s}$ and
$\mathbf{U}_{2\to s}$ for any scanline $s\in[0,h-1]$. Then, the corresponding
high framerate GS images can be warped from the original two RS images, in the
same vein, by using softmax splatting, _i.e._ , achieving the rolling-shutter
temporal super-resolution in a temporally coherent manner.

As an alternative, we further devise a compact and lightweight Acceleration-
Net module to efficiently estimate the variable $\varphi$ in order to make Eq.
(26) work for maintaining more realistic and accurate temporal coherence
across corrected GS video frames. Acceleration-Net $\mathcal{A}$ is
constructed and executed after the trained optical flow estimation network
$\mathcal{F}$ and middle-scanline RS undistortion flow estimation network
$\mathcal{U}$. Subsequently, as shown in Fig. 5, the generated maps are
cascaded and fed into three $3\times 3$ convolutional layers (each followed by
a LeakyReLU activation layer) for extracting features, then a 3D adaptive
average pooling layer together with four fully-connected layers is used to
predict $\varphi_{1}$ and $\varphi_{2}$ in forward and backward warpings,
respectively. At this time, we freeze the parameters of $\mathcal{F}$ and
$\mathcal{U}$, and train $\mathcal{A}$ separately by using the GT first-
scanline GS images for supervision. Note that our vanilla RSSR pipeline can
work well with a constant velocity propagation, where the GT first-scanline GS
image is not required, _i.e._ , our entire pipeline is flexible and tractable.

Loss function. Given a pair of consecutive RS images ${\mathbf{I}}_{r}^{1}$
and ${\mathbf{I}}_{r}^{2}$, our vanilla RSSR network predicts the
bidirectional optical flows ${\mathbf{F}}_{1\to 2}$ and ${\mathbf{F}}_{2\to
1}$, the bidirectional RS undistortion flows ${\mathbf{U}}_{1\to m}$ and
${\mathbf{U}}_{2\to m}$, and the target middle-scanline GS images
${\mathbf{I}}_{g}^{1}$ and ${\mathbf{I}}_{g}^{2}$. Note that when training
Acceleration-Net separately, we continue to use ${\mathbf{I}}_{g}^{1}$ and
${\mathbf{I}}_{g}^{2}$ to refer specifically to the target first-scanline GS
images for brevity. Meanwhile, ${\mathbf{I}}_{gt}^{1}$ and
${\mathbf{I}}_{gt}^{2}$ denote their corresponding GT middle-scanline or
first-scanline GS images. Our overall loss function $\mathcal{L}$ is a linear
combination of the reconstruction loss $\mathcal{L}_{r}$, perceptual loss
$\mathcal{L}_{p}$ [56], warping loss $\mathcal{L}_{w}$, and smoothness loss
$\mathcal{L}_{s}$:

$\vspace{-0.4mm}\mathcal{L}=\mu_{r}\mathcal{L}_{r}+\mu_{p}\mathcal{L}_{p}+\mu_{w}\mathcal{L}_{w}+\mu_{s}\mathcal{L}_{s},\vspace{-0.4mm}$
(29)

where $\mu_{r}$, $\mu_{p}$, $\mu_{w}$ and $\mu_{s}$ are hyper-parameters to
balance different losses. Note that when training the vanilla RSSR network, we
set $\mu_{r}=10$, $\mu_{p}=1$, $\mu_{w}=10$, and $\mu_{s}=0.1$; when training
Acceleration-Net $\mathcal{A}$ based on the pre-trained $\mathcal{F}$ and
$\mathcal{U}$, we remove the warping loss $\mathcal{L}_{w}$ and the smoothness
loss $\mathcal{L}_{s}$ while keeping the rest the same, _i.e._ ,
$\mu_{w}=\mu_{s}=0$. In the following, we describe in detail how to supervise
the training of the vanilla RSSR network.

_Reconstruction loss $\mathcal{L}_{r}$_: We measure the pixel-wise
reconstruction qualities of the corrected middle-scanline GS images as:

$\vspace{-0.4mm}\mathcal{L}_{r}=\sum_{i=1}^{2}\left\|{\mathbf{I}}_{g}^{i}-{\mathbf{I}}_{gt}^{i}\right\|_{1}.\vspace{-0.4mm}$
(30)

_Perceptual loss $\mathcal{L}_{p}$_: To mitigate the blur in the corrected
middle-scanline GS images, we use a perceptual loss $\mathcal{L}_{p}$ [56] to
preserve details of the predictions and make estimated GS images sharper.
Similar to [14], $\mathcal{L}_{p}$ is defined as:

$\vspace{-0.4mm}\mathcal{L}_{p}=\sum_{i=1}^{2}\left\|\phi\left({\mathbf{I}}_{g}^{i}\right)-\phi\left({\mathbf{I}}_{gt}^{i}\right)\right\|_{1},\vspace{-0.4mm}$
(31)

where $\phi$ represents the $conv3\\_3$ feature extractor of the VGG19 model
[57].

_Warping loss $\mathcal{L}_{w}$_: Besides supervising the GS image
predictions, we also introduce the warping loss $\mathcal{L}_{w}$ to maintain
the qualities of final bidirectional optical flows, defined as:

$\small\mathcal{L}_{w}=\left\|{\mathbf{I}}_{r}^{1}-g({\mathbf{I}}_{r}^{2},\hat{\mathbf{F}}_{1\to
2})\right\|_{1}+\left\|{\mathbf{I}}_{r}^{2}-g({\mathbf{I}}_{r}^{1},\hat{\mathbf{F}}_{2\to
1})\right\|_{1},$ (32)

where $\hat{\mathbf{F}}_{1\to 2}={\mathbf{F}}_{1\to 2}+\Delta\mathbf{F}_{1\to
2}$, $\hat{\mathbf{F}}_{2\to 1}={\mathbf{F}}_{2\to 1}+\Delta\mathbf{F}_{2\to
1}$, and $g(\cdot,\cdot)$ is the backward warping function.

_Smoothness loss $\mathcal{L}_{s}$_: At last, a smoothness term [26] is
employed to enforce the smoothness of the bidirectional optical flows and
bidirectional RS undistortion flows as:

$\mathcal{L}_{s}=\sum_{i=1}^{2}\sum_{j=1,j\not=i}^{2}\left\|\nabla\hat{\mathbf{F}}_{i\to
j}\right\|_{2}+\left\|\nabla{\mathbf{U}}_{i\to m}\right\|_{2}.$ (33)

##  6 Experiments

###  6.1 Implementation Details

Dataset. We train and evaluate our method on the Carla-RS and Fastec-RS
datasets [14] that provide ground truth middle-scanline and first-scanline GS
supervisory signals. The Carla-RS dataset is generated from a virtual 3D
environment based on the Carla simulator [58], involving general six degrees
of freedom motions. The Fastec-RS dataset contains real-world RS images
synthesized by a professional high-speed GS camera. The image is normalized to
$640\times 448$ resolution to fit the basic input requirements of the backbone
network PWC-Net. Following [14], the Carla-RS dataset is divided into a
training set of 210 sequences and a test set of 40 sequences, and the Fastec-
RS dataset has 56 sequences for training and the remaining 20 ones for
testing. There are no overlapping scenes between the training and test data.
We train our vanilla RSSR network on both two benchmarks to predict the
middle-scanline GS images (_i.e._ , $s=h/2$). At the test time, our model is
able to be extended to generate arbitrary GS video frames for any scanline
$s\in[0,h-1]$.

Training strategy. Our pipeline111Code can be available at
https://github.com/GitCVfb/RSSR. is implemented in PyTorch. We use Adam [59]
as the optimizer. The learning rate is initially set to $10^{-4}$ and
decreases by a factor of 0.8 every 50 epochs. The optical flow estimation
network $\mathcal{F}$ is first fine-tuned for 100 epochs from the pre-trained
PWC-Net model [22], and then the entire model is jointly trained for another
200 epochs. Acceleration-Net is subsequently trained for 30 epochs with a
fixed learning rate of $10^{-4}$. The training batch size is set to 6. We use
a uniform random crop at a horizontal resolution of 256 pixels for data
augmentation. Note that we do not change the longitudinal resolution to
warrant the scanline dependence of RS images.

###  6.2 Evaluation Protocols

We compute the average Peak Signal-to-Noise Ratio (PSNR), Structural
Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS)
[60] between estimated and GT GS images. Higher PSNR/SSIM or lower LPIPS
scores indicate better performance. Note that _unless otherwise stated, all
competing methods refer to the GS image corresponding to the first scanline of
the second RS frame_ for consistent comparisons. Moreover, _the constant
velocity propagation method is preferred,_i.e._ , we mainly evaluate our
vanilla RSSR method_ based on the constant velocity model due to its
simplicity. Subsection 6.10 provides additional instructions on DeepUnrollNet
[14]. Since the Carla-RS dataset provides the GT occlusion masks, for better
evaluation, we conduct quantitative experiments including: the Carla-RS
dataset with occlusion mask (_CRM_), the Carla-RS dataset without occlusion
mask (_CR_), and the Fastec-RS dataset (_FR_).

Figure 6: Effectiveness of optical flow residual estimation layer. It can effectively alleviate the artifacts and holes at the boundaries caused by optical flow misalignment, leading to higher quality results.  TABLE I: Ablation study on the selection and training strategy of network $\mathcal{F}$. We employ different optical flow estimation baselines (RAFT [28] and PWC-Net [22]), while testing the effect of freezing their parameters during training. RAFT | PWC-Net | Freeze |  PSNR$\uparrow$ (dB) |  |  SSIM$\uparrow$
---|---|---|---|---|---
CRM | FR |  | CR | FR
$\checkmark$ |  | $\checkmark$ | 29.81 | 20.63 |  | 0.87 | 0.77
$\checkmark$ |  | $\times$ | 26.75 | 20.38 |  | 0.84 | 0.71
| $\checkmark$ | $\checkmark$ | 29.44 | 20.64 |  | 0.86 | 0.77
| $\checkmark$ | $\times$ | 30.17 | 21.26 |  | 0.87 | 0.78

###  6.3 Ablation Studies

To demonstrate the effectiveness of each component in our proposed network, we
evaluate the controlled comparisons over network $\mathcal{F}$, network
$\mathcal{U}$, and loss function $\mathcal{L}$, respectively. We train these
variations using the same strategy as aforementioned in Subsection 6.1. We
will explain each of them in detail below.

####  6.3.1 Ablation on the selection and training strategy of network
$\mathcal{F}$

We first replace PWC-Net [22] with the state-of-the-art (SOTA) optical flow
estimation baseline RAFT [28]. Then, we analyze the influence of different
training strategies of network $\mathcal{F}$, including parameter freezing and
model initialization during training, _i.e._ ,

  * •

_$\mathcal{F}$ -Scra_: We initialize $\mathcal{F}$ from scratch and optimize
it with the whole model.

  * •

_$\mathcal{F}$ -Pret_: We initialize $\mathcal{F}$ from the pre-trained model
provided by [22] and optimize it with the whole model.

The quantitative results are summarized in Table I and Table II. We can
observe that RAFT contributes slightly worse than PWC-Net to the high
framerate GS video extraction in our implementation. Freezing the network
parameters when using RAFT can significantly improve performance, but it has
the opposite benefit when combined with PWC-Net. When initializing from the
pre-trained model, especially with the addition of fine-tuning, our method
shows a substantial performance improvement. After jointly optimizing the
whole network together with PWC-Net, the overall performance is further
improved, which is better capable of exploiting the concealed motion between
scanlines as well as the scene structure.

####  6.3.2 Ablation on the design of network $\mathcal{U}$

We further investigate the contribution of each component in network
$\mathcal{U}$ as follows:

  * •

w/o _$\mathbf{T}$_ : We remove the normalized scanline offset in Eq. (27), and
replace the _Sigmoid_ function with the _Tanh_ function in network
$\mathcal{U}$ to uniformly map the correlation factor prediction of each pixel
to the interval of $(-1,1)$.

  * •

w/o P2: We remove the underlying constraint defined in Proposition 2, _i.e._ ,
use the normalized scanline offset and remove the _Sigmoid_ function, such
that the network $\mathcal{U}$ output the correlation maps $\mathbf{C}_{1\to
m},\mathbf{C}_{2\to m}\in\mathbb{R}^{h\times w}$, _i.e._ , learn a generalized
middle-scanline undistortion flow.

  * •

w/o _$\Delta\mathbf{F}$_ : We remove the optical flow residual estimation
layer in network $\mathcal{U}$, _i.e._ , $\Delta\mathbf{F}_{1\to
2}=\Delta\mathbf{F}_{2\to 1}=\mathbf{0}$ in Eq. (28).

We report the results in Table II and Fig. 6. One can see that the explicit
constraint of the normalized scanline offset benefits the learning of the
scanline-dependent nature of the RS undistortion flow, which is consistent
with the observation in [14]. Moreover, imposing the intrinsic geometric
constraint in Proposition 2 promotes better convergence of our vanilla network
and decisively improves _CRM_ by 4.22 dB. And adding the optical flow residual
estimation layer is effective to facilitate the edge alignment and improve the
robustness of the proposed model in the extreme case, thereby recovering more
complete and accurate image details.

####  6.3.3 Ablation on the loss function

We show the results of training our models under different loss function
settings in Table II. We remove each loss term from the overall loss function
$\mathcal{L}$ respectively. Without $\mathcal{L}_{w}$ indicates freezing the
parameters of PWC-Net in Table I. Forcing the smoothness of the estimated
flows has a particularly positive effect on improving the performance. Our
loss function is effective as the performance of adopting all loss terms is
the best.

TABLE II: Effectiveness of different components of our vanilla RSSR method on the Carla-RS dataset. |  PSNR$\uparrow$ (dB) |  |  SSIM$\uparrow$ |  |  LPIPS$\downarrow$
---|---|---|---|---|---
CRM | CR |  | CR |  | CR
_$\mathcal{F}$ -Scra_ | 27.37 | 24.22 |  | 0.80 |  | 0.0804
_$\mathcal{F}$ -Pret_ | 29.89 | 24.61 |  | 0.86 |  | 0.0697
w/o $\mathbf{T}$ | 25.43 | 22.55 |  | 0.82 |  | 0.1116
w/o P2 | 25.95 | 22.98 |  | 0.78 |  | 0.0812
w/o $\Delta\mathbf{F}$ | 29.12 | 24.29 |  | 0.85 |  | 0.0725
w/o $\mathcal{L}_{r}$ | 29.44 | 24.35 |  | 0.86 |  | 0.0713
w/o $\mathcal{L}_{p}$ | 29.82 | 24.61 |  | 0.86 |  | 0.0706
w/o $\mathcal{L}_{s}$ | 29.28 | 24.44 |  | 0.86 |  | 0.0725
full model | 30.17 | 24.78 |  | 0.87 |  | 0.0695
Figure 7: Visual comparisons on the Fastec-RS testing set. We zoom in the correction results according to the blue boxes. Our method fully explores the underlying RS geometry and generates a set of high-quality GS results, in spite of the road sign that is subject to large RS effects. Figure 8: Visual examples of RS undistortion flow obtained by DiffSfM and our method, respectively. Thanks to end-to-end learning, our method can generate more accurate RS undistortion flows (see green circles), and thus synthesize higher quality GS images.  TABLE III: Quantitative comparisons on recovering GS images corresponding to the first scanline of the second RS frame. Bold and underlined numbers represent the best and second-best performance. Our approach performs favorably against existing methods. Method |  PSNR$\uparrow$ (dB) |  |  SSIM$\uparrow$ |  |  LPIPS$\downarrow$
---|---|---|---|---|---
CRM | CR | FR |  | CR | FR |  | CR | FR
DeepUnrollNet [14] | 26.90 | 26.46 | 26.52 |  | 0.81 | 0.79 |  | 0.0703 | 0.1222
DiffSfM-_PWCNet_ [15] | 19.53 | 18.62 | 18.59 |  | 0.69 | 0.63 |  | 0.2042 | 0.2416
DiffSfM-_RAFT_ [15] | 24.20 | 21.28 | 20.14 |  | 0.78 | 0.70 |  | 0.1322 | 0.1789
RSSR (Ours) | 30.17 | 24.78 | 21.26 |  | 0.87 | 0.78 |  | 0.0695 | 0.1424

###  6.4 Comparisons with SOTA RS Correction Methods

We evaluate the proposed RSSR method against the following RS correction
algorithms:

  * -

DeepUnrollNet [14]: This is the SOTA learning-based two-image RS correction
method. It has no ability to produce high framerate GS video sequences. Note
that the pre-trained model released by the authors is used to estimate the GS
image of the middle scanline of the second RS frame (See Subsection 6.10 for
more details). For a fair comparison, we retrain it to synthesize the GS image
corresponding to the first scanline of the second RS frame.

  * -

DiffSfM [15] and DiffHomo [16]: They are traditional RS correction methods. We
implement two versions of DiffSfM using optical flow inputs obtained by PWC-
Net and RAFT, respectively. Since DiffHomo is not open source, we asked the
authors to test some examples. Note that these two methods are suitable for
recovering the GS image at the first scanline of the first RS frame. To ensure
a fair comparison, for the input of three consecutive RS images, we run our
method using the first two RS frames as input, and perform DiffSfM and
DiffHomo using the last two RS images as input. In this way, we can generate
and align to the GS image corresponding to the first scanline of the second RS
frame.

We report the quantitative results in Table III. The better optical flow
obtained by RAFT can improve the estimation accuracy of RS geometry, thus
promoting the performance of RS correction in DiffSfM, which also indicates
the importance of mining the underlying RS geometry. Our model performs
favorably against all the compared methods in the Carla-RS dataset and
competes on par with DeepUnrollNet in the Fastec-RS dataset, since the Carla-
RS dataset is more consistent with the constant motion assumption. Note that,
compared with DeepUnrollNet, the black holes in our corrected GS images will
reduce the PSNR score of our method without using masks. Very importantly, our
method can generate high framerate and visually pleasing GS video sequences.
We provide the visual correction results with noticeable RS distortions in
Fig. 7, where DiffHomo and DiffSfM fail to correct the geometric RS
distortions and DeepUnrollNet causes loss of local image details. Compared
with the off-the-shelf RS correction algorithms, our pipeline effectively
restores higher-quality GS images.

Furthermore, considering the existence of an analytic solution for the
correction map ${\mathbf{C}}$ in Eq. (21) (which can be estimated by DiffSfM
[15]), we visualize and compare the intermediate process. Since RS
undistortion flow is essentially the core of GS image recovery, we illustrate
it estimated by our method and DiffSfM in Fig. 8. In addition to the complex
computation, DiffSfM is powerless to correct the initial incorrect optical
flow and thus produces local artifacts, as shown by the red circles.
Furthermore, when DiffSfM estimates erroneous RS geometry, such as biased
camera motion, the RS effect will not be truly removed. In contrast, since our
approach supports end-to-end learning, the inaccuracy of the initial optical
flow can be automatically adjusted to yield a more accurate task-oriented RS
undistortion flow, thereby recovering a more reliable GS image.

Figure 9: Examples of GS video frame interpolation by our vanilla RSSR method. We zoom in the intermediate results according to the orange boxes. For example, “${\mathbf{I}}_{1\to 0.3h}$” denotes the corrected global-shutter image corresponding to $0.3h$-th scanline of RS frame 1. Our method not only preserves the temporal smoothness but also corrects the rolling-shutter artifacts.  Figure 10: Qualitative image correction results on real data with obvious RS distortion provided by [15] and [37], respectively. The left three columns are from [15] and the right three columns are from [37]. Our pipeline can effectively remove RS distortion as a whole. In particular, our more general constant acceleration model produces more trustworthy images on unseen data, as indicated by the red circles.  Figure 11: Robustness evaluation of the readout time ratio. (PSNR _vs._ Readout time ratio and SSIM _vs._ Readout time ratio)  TABLE IV: Quantitative comparisons on recovering GS images corresponding to the first scanline of the second RS frame under the constant velocity propagation and the constant acceleration propagation, respectively. Our Model |  PSNR$\uparrow$ (dB) |  |  SSIM$\uparrow$ |  |  LPIPS$\downarrow$
---|---|---|---|---|---
CRM | CR | FR |  | CR | FR |  | CR | FR
RSSR (_Constant velocity_) | 30.17 | 24.78 | 21.26 |  | 0.867 | 0.784 |  | 0.0695 | 0.1424
RSSR (_Constant acceleration_) | 30.35 | 24.83 | 21.43 |  | 0.870 | 0.790 |  | 0.0688 | 0.1379

###  6.5 Generating Multiple GS Video Frames

We generate multiple GS video frames corresponding to different scanlines, as
shown in Fig. 9. Moreover, we attach a _supplementary video_ to dynamically
show the reversed GS video clips, involving a cropping operation. In
principle, we can produce videos with arbitrary frame rates. Our RSSR network
learns to solve the complex RS geometry embedded in the consecutive RS frames,
so it can robustly and accurately recover photorealistic time-continuous GS
images. Overall, our method not only has the advantage of RS correction at a
specific scanline time, but also has the superior ability to restore GS images
at any scanline time.

Input RS 2 | Ours (Constant velocity) | Ours (Constant acceleration) | Ground truth
---|---|---|---
|  |  |
(23.95/0.480/0.0927) | (30.47/0.884/0.0453) | (30.55/0.891/0.0449) | (inf./1.000/0.0000)
|  |  |
(15.50/0.487/0.2317) | (22.59/0.815/0.0823) | (22.87/0.833/0.0817) | (inf./1.000/0.0000)
|  |  |
(21.55/0.515/0.1991) | (20.78/0.664/0.1983) | (21.12/0.683/0.1913) | (inf./1.000/0.0000)
|  |  |
(13.93/0.516/0.2545) | (19.34/0.749/0.1977) | (20.16/0.787/0.1796) | (inf./1.000/0.0000)
Figure 12: Metrics performance (_i.e._ ,
PSNR$\uparrow$/SSIM$\uparrow$/LPIPS$\downarrow$) on estimating the global-
shutter image of the first scanline under the constant velocity model and the
constant acceleration model, respectively. The overall quality of the
recovered global-shutter image is further upgraded by applying a more general
constant acceleration propagation.

(a) Carla-RS dataset

(b) Fastec-RS dataset

Figure 13: Frequency distribution histograms of the estimated backward
acceleration factors by our Acceleration-Net, where the Carla-RS dataset
exhibits much smaller acceleration factor values compared to the subtle intra-
frame motion, which is in line with the expectation.

###  6.6 Generalization and Robustness Evaluation

Our learning-based model is trained on the Carla-RS dataset, in which the RS
artifacts are mainly caused by uniform camera motion. We apply our method to
real RS images provided by [37] and [15]. The example results are shown in
Fig. 10. The attached _video_ provides more results. Note that the real data
was recorded in [37] by a rotation-dominated hand-held RS camera and [15]
collected 720p RS images at 30 fps. The results reveal that our method owns
good generalization ability and can recover visually compelling GS images, due
to the learned RS geometry.

Since the readout time ratio $\gamma$ is set to 1 in the Carla-RS and Fastec-
RS datasets, the models obtained based on them may be biased towards such
training data. Therefore, to verify the robustness to $\gamma$, we use the
Carla-RS dataset to simulate continuous RS image pairs with $\gamma$ ranging
from 0.6 to 1. For example, to obtain the test data with $\gamma=0.8$, we
first discard the pixels in the last $20\%$ scanlines of each RS image and
then bilinearly interpolate to the original resolution. We report the
evaluation results in Fig. 11. It can be seen that the RS correction
performance gradually decreases as $\gamma$ deviates from the training dataset
of $\gamma=1$, due to the fact that our network $\mathcal{U}$ essentially
encapsulates the underlying RS geometry with $\gamma=1$ (_i.e._ , the
correction map defined in Eq. (21)). Actually, $\gamma$ is crucial for two-
view RS geometry estimation and needs to be calibrated in advance, which has
also been verified in the case of depth estimation [5] and RS correction [15,
16, 10]. Our method allows for a slight perturbation of $\gamma$ and thus
generalizes well to the real RS data provided by [15] and [37], where $\gamma$
is equal to 0.96 and 0.92, respectively. Note that the model exported by
DeepUnrollNet is also quite dependent on the training dataset. Thus,
developing a method that is robust to readout time ratios will be a future
research direction.

###  6.7 An Alternative: Constant Acceleration Propagation

Although our vanilla RSSR method has achieved impressive results in generating
high framerate GS video frames under the constant velocity model as mentioned
before, an alternative propagation scheme based on Acceleration-Net and Eq.
(26) under the constant acceleration model achieves tangible improvements over
the constant velocity propagation method, as shown in Table IV and Figs. 12
and 10. In terms of how the RS correction dataset itself is generated in [14],
the Carla-RS dataset follows the constant velocity motion model more closely,
on which the improvement of performance metrics is therefore not significant.
This is also verified in Fig. 13. The introduction of the constant
acceleration model on the Fastec-RS dataset to extend to the first scanline
(or any scanline) is more successful in compensating for the RS effect, as
would be expected given more realistic motion assumptions. The higher quality
first-scanline GS image recovery is illustrated in Figs. 12 and 10, which
demonstrates that the constant acceleration propagation effectively improves
the performance and makes the inversion results more faithful to the latent
global-shutter images.

Figure 14: Comparison with VFI methods (BMBC [27] and DAIN [21]) and two-stage methods (DeepUnrollNet + BMBC and DeepUnrollNet + DAIN).  TABLE V: Superiority of our method over varying two-stage methods on the Carla-RS dataset. We report the quantitative results corresponding to the first and middle scanlines of the RS frame, respectively. Method |  PSNR$\uparrow$ (dB) |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---
Corresponding to the first scanline
DeepUnrollNet + BMBC | 27.29 | 0.83 | 0.0980
DeepUnrollNet + DAIN | 27.48 | 0.87 | 0.0821
RSSR (Ours) | 30.17 | 0.87 | 0.0695
Corresponding to the middle scanline
SUNet + BMBC | 28.51 | 0.85 | 0.1033
SUNet + DAIN | 28.63 | 0.85 | 0.0919
RSSR (Ours) | 29.36 | 0.90 | 0.0553

###  6.8 Versus SOTA Video Frame Interpolation Methods

The current video frame interpolation algorithms, _e.g._ , BMBC [27] and DAIN
[21], have a common implicit assumption that the camera employs a global-
shutter mechanism, where the pixel displacement is controllable and located in
the corresponding optical flow. Specifically, linearly scaling the optical
flow between 0 and 1 to approximate the required intermediate pixel
displacement in order to warp input images. In contrast, to correct the RS
image, as shown in Eq. (20), the pixel displacement is neither a linear
function of scanline time (including complex RS geometry) nor within the
corresponding optical flow (_i.e._ , the length of the RS undistortion flow
may be larger than that of the optical flow, or its direction may be opposite
to the optical flow), involving intrinsic non-local operations. Therefore,
because of inherent flaws in the network architectures, the existing video
frame interpolation algorithms are incapable of eliminating the RS effect
effectively. We validate this argument in Fig. 14, which also highlights the
superiority of our method in recovering high-quality distortion-free GS video
frames.

Furthermore, a naive approach to the RS inversion task at hand would be to
cascade RS correction and interpolation methods. We thus conduct experiments
to compare with this so-called two-stage approach. Since DeepUnrollNet [14]
and SUNet [17] are fundamentally designed to recover the GS image
corresponding to the middle and first scanlines of the second RS frame,
respectively, we implement four two-stage methods including four cascades as
follows:

  * -

DeepUnrollNet + BMBC and DeepUnrollNet + DAIN: Given three consecutive RS
images, we first obtain two middle-scanline GS images in sequence using
DeepUnrollNet, and then interpolate the GS image corresponding to the first
scanline of the third RS image using BMBC and DAIN, respectively.

  * -

SUNet + BMBC and SUNet + DAIN: Given three consecutive RS images, we first
obtain two first-scanline GS images in sequence using SUNet, and then
interpolate the GS image corresponding to the middle scanline of the second RS
image using BMBC and DAIN, respectively. For consistent comparison, we do not
visualize them.

As outlined in Fig. 14, the two-stage approach is prone to error accumulation
and therefore suffers from serious blurring artifacts and local inaccuracies.
Predictably, further recursive interpolation of other time steps will
exacerbate these image degradations. The quantitative results on the Carla-RS
dataset are shown in Table V. Moreover, it is computationally inefficient. For
instance, to recover 960 GS video frames, the two-stage approach takes about 5
minutes (at least 0.3 seconds to generate a frame with DAIN), while our
vanilla RSSR method takes a total of 1.8 seconds. Since our method
simultaneously solves the problems of RS correction and temporal super-
resolution in an end-to-end manner, it achieves remarkable performance in
terms of both quality and efficiency.

Figure 15: Visual comparison of GS images corrected to the middle scanline of
the second RS frame. At this time, the corresponding plausible global-shutter
image can be reconstructed by DeepUnrollNet [14] and our method. Since we have
not developed a modulated image decoder as in DeepUnrollNet, our method cannot
yet fill the occluded regions. Our RSSR method, however, is able to recover
global-shutter video images at any scanline, which is far beyond the reach of
DeepUnrollNet.  Figure 16: 3D reconstruction results using the original RS
images and our corrected GS images, respectively. Our method removes RS
distortion and thus facilitates the recovery of more accurate 3D geometry.

###  6.9 3D Reconstruction Evaluation

We run an SfM pipeline (_i.e._ , Colmap [61]) to process the original RS
images and our corrected GS images, respectively. Fig. 16 demonstrates that
our method can remove the geometric inaccuracies and reconstruct a more
reliable 3D scene structure. Note that significant geometric distortion
appears in the 3D model obtained from the original RS images. Overall, our
method is beneficial to downstream applications with its superior ability to
remove RS artifacts.

###  6.10 Additional Comparison with DeepUnrollNet

In our evaluation, in order to be consistent with [7, 16, 15] (_i.e._ ,
recovering the GS image corresponding to the first scanline), all competing
results relate to the first scanline of the second frame. We thus retrain
DeepUnrollNet [14] to adapt to this task for fair comparison. However, the
original publication of [14] is designed to restore a GS image corresponding
to the middle scanline of the second frame, so we additionally add
quantitative and qualitative results under the middle scanline of the second
frame, as shown in Table VI and Fig. 15. One can see that, except for some
occluded black edges (_e.g._ , CRM score), our method is comparable or
superior to DeepUnrollNet in returning the GS image corresponding to the
middle scanline of the second frame, which verifies the validity of
Proposition 2. Moreover, combined with the experiments in Table II for
learning a generalized middle-scanline undistortion flow (_i.e._ , the
ablation “w/o P2”), this is why we follow the middle-scanline undistortion
flow generated by scaling the corresponding optical flow vector under the
constant velocity model for our subsequent constant acceleration propagation,
_i.e._ , the initial middle-scanline undistortion flow is able to yield a
reliable and accurate prediction of the warping displacement direction.

Note that our method can restore the GS image corresponding to any scanline
_without demanding access to the supervision of the corresponding GT GS
images_. However, DeepUnrollNet and JCD [11] are limited to hallucinate a
single middle-scanline GS frame. Although DeepUnrollNet can also be trained to
learn the mapping between the RS image and the first-scanline GS image when
the costly first-scanline GT is provided, recovering the GS image
corresponding to the first scanline of the second frame is not impressive.
Since acquiring the GT GS images is expensive and laborious, especially for
arbitrary scanlines, it poses significant challenges for network design and
model training. Our method successfully circumvents this problem and is
satisfactory in restoring GS images of the first scanlines, middle scanlines,
and even arbitrary scanlines.

TABLE VI: Quantitative comparisons of recovering GS images corresponding to
the middle scanline of the second RS frame.

Method |  PSNR$\uparrow$ (dB) |  |  SSIM$\uparrow$
---|---|---|---
CRM | CR | FR |  | CR | FR
JCD† [11] | - | - | 24.84 |  | - | 0.78
DeepUnrollNet [14] | 27.86 | 27.54 | 27.02 |  | 0.83 | 0.83
RSSR (Ours) | 29.36 | 26.57 | 25.01 |  | 0.90 | 0.83

  * •

${\dagger}$: _results copied from[11]_

###  6.11 Inference Times

Our method can simultaneously predict two middle-scanline GS images with VGA
resolution ($640\times 480$ pixels) in near real-time on an NVIDIA GeForce RTX
2080Ti GPU (average 0.12 seconds), which is faster than the 0.34 seconds of
[14] to restore a single GS image. Further, we can extend to generate a GS
image corresponding to any certain scanline via propagation, using an average
runtime of 1.75 milliseconds, because only simple explicit matrix operations
are required. Therefore, our vanilla RSSR method can efficiently _produce 960
GS video frames in about 1.80 seconds_ , achieving temporal upsampling from 30
fps to 14400 (480$\times$30) fps. In other words, quickly and extremely
enlarging the frame rate of reversed GS video will be easy to reach.
Optionally, the lightweight Acceleration-Net will take an additional 1.17
milliseconds under the constant acceleration motion model, which can be
negligible. DiffSfM [15], however, takes about 467.26 seconds to recover a
single GS image on an Intel Core i7-7700K CPU, which is a disadvantage for
time-constrained tasks, such as real-time robotic visual localization.

###  6.12 Limitations

The main limitation of our approach is that it is not robust to heavy
occlusions and moving objects. Occlusion increases the difficulty of
estimating the RS geometry, which leads to black holes in the synthesized
images, degrading the visual experience. This problem might be alleviated by
performing context aggregation in feature space (_e.g._ [17, 29]) or image
space (_e.g._ [30, 19]). Moreover, moving objects have always been a great
challenge for model-based RS correction methods, _e.g._ , [2, 15, 16, 7, 6].
In image regions specific to moving objects, the RS undistortion flow will not
be well estimated due to violation of camera motion assumptions, resulting in
ambiguous motion artifacts.

##  7 Conclusion

In this paper, we tackle the challenging task of RS image inversion, _i.e._ ,
converting consecutive RS images to high framerate GS videos. To this end, we
have constructed and discussed the inherent interaction between bidirectional
RS undistortion flow and regular optical flow. We present the first novel and
intuitive RS temporal super-resolution framework that extracts a latent GS
image sequence from two consecutive RS images, which is guided by the
underlying geometric properties of the problem itself. Our pipeline owns good
interpretability and generalization ability due to RS geometry-aware learning.
Furthermore, we generalize and develop a constant acceleration propagation
method, which contributes to further improving the accuracy and robustness of
the RSSR task. We have demonstrated that our approach not only reconstructs
geometrically and temporally consistent video sequences efficiently but also
removes RS artifacts. In the future, we plan to extend our method to handling
continuous rolling-shutter videos directly, and making better use of the scene
context information. Meanwhile, we hope that our work will help incorporate
traditional computer vision knowledge into deep learning systems, so as to
harness decades of valuable research and contribute to the development of a
more principled framework.

## Acknowledgments

This research was supported in part by National Natural Science Foundation of
China (62271410, 62001394, and 61901387) and National Key Research and
Development Program of China (2018AAA0102803).

## References

  * [1] Y. Dai, H. Li, and L. Kneip, “Rolling shutter camera relative pose: generalized epipolar geometry,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 4132–4140.
  * [2] Y. Lao and O. Ait-Aider, “Rolling shutter homography and its applications,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 8, pp. 2780–2793, 2021.
  * [3] C. Albl, Z. Kukelova, V. Larsson, and T. Pajdla, “Rolling shutter camera absolute pose,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 6, pp. 1439–1452, 2019.
  * [4] K. Wang, C. Liu, K. Wang, and S. Shen, “Depth estimation under motion with single pair rolling shutter stereo images,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 3160–3167, 2021.
  * [5] S. Im, H. Ha, G. Choe, H.-G. Jeon, K. Joo, and I. S. Kweon, “Accurate 3d reconstruction from small motion clip for rolling shutter cameras,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 4, pp. 775–787, 2018.
  * [6] V. Rengarajan, Y. Balaji, and A. Rajagopalan, “Unrolling the shutter: cnn to correct motion distortions,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2291–2299.
  * [7] B. Zhuang, Q.-H. Tran, P. Ji, L.-F. Cheong, and M. Chandraker, “Learning structure-and-motion-aware rolling shutter correction,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4551–4560.
  * [8] V. Rengarajan, A. N. Rajagopalan, and R. Aravind, “From bows to arrows: rolling shutter rectification of urban scenes,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2773–2781.
  * [9] Y. Lao and O. Ait-Aider, “A robust method for strong rolling shutter effects correction using lines with automatic feature selection,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4795–4803.
  * [10] S. Vasu, M. M. Mohan, and A. Rajagopalan, “Occlusion-aware rolling shutter rectification of 3d scenes,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 636–645.
  * [11] Z. Zhong, Y. Zheng, and I. Sato, “Towards rolling shutter correction and deblurring in dynamic scenes,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9219–9228.
  * [12] J. Hedborg, P.-E. Forssén, M. Felsberg, and E. Ringaby, “Rolling shutter bundle adjustment,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 1434–1441.
  * [13] B. Fan, Y. Dai, and K. Wang, “Rolling-shutter-stereo-aware motion estimation and image correction,” _Computer Vision and Image Understanding_ , vol. 213, p. 103296, 2021.
  * [14] P. Liu, Z. Cui, V. Larsson, and M. Pollefeys, “Deep shutter unrolling network,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5941–5949.
  * [15] B. Zhuang, L.-F. Cheong, and G. Hee Lee, “Rolling-shutter-aware differential sfm and image rectification,” in _Proceedings of IEEE International Conference on Computer Vision_ , 2017, pp. 948–956.
  * [16] B. Zhuang and Q.-H. Tran, “Image stitching and rectification for hand-held cameras,” in _Proceedings of European Conference on Computer Vision_ , 2020, pp. 243–260.
  * [17] B. Fan, Y. Dai, and M. He, “Sunet: symmetric undistortion network for rolling shutter correction,” in _Proceedings of IEEE International Conference on Computer Vision_ , 2021, pp. 4541–4550.
  * [18] B. Fan and Y. Dai, “Inverting a rolling shutter camera: bring rolling shutter images to high framerate global shutter video,” in _Proceedings of IEEE International Conference on Computer Vision_ , 2021, pp. 4228–4237.
  * [19] H. Jiang, D. Sun, V. Jampani, M.-H. Yang, E. Learned-Miller, and J. Kautz, “Super slomo: high quality estimation of multiple intermediate frames for video interpolation,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9000–9008.
  * [20] S. Niklaus and F. Liu, “Softmax splatting for video frame interpolation,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5437–5446.
  * [21] W. Bao, W.-S. Lai, C. Ma, X. Zhang, Z. Gao, and M.-H. Yang, “Depth-aware video frame interpolation,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3703–3712.
  * [22] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “Pwc-net: cnns for optical flow using pyramid, warping, and cost volume,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8934–8943.
  * [23] S. Meyer, O. Wang, H. Zimmer, M. Grosse, and A. Sorkine-Hornung, “Phase-based frame interpolation for video,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 1410–1418.
  * [24] S. Meyer, A. Djelouah, B. McWilliams, A. Sorkine-Hornung, M. Gross, and C. Schroers, “Phasenet for video frame interpolation,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 498–507.
  * [25] M. Choi, H. Kim, B. Han, N. Xu, and K. M. Lee, “Channel attention is all you need for video frame interpolation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 10 663–10 671.
  * [26] Z. Liu, R. A. Yeh, X. Tang, Y. Liu, and A. Agarwala, “Video frame synthesis using deep voxel flow,” in _Proceedings of IEEE International Conference on Computer Vision_ , 2017, pp. 4463–4471.
  * [27] J. Park, K. Ko, C. Lee, and C.-S. Kim, “Bmbc: bilateral motion estimation with bilateral cost volume for video interpolation,” in _Proceedings of European Conference on Computer Vision_ , 2020, pp. 109–125.
  * [28] Z. Teed and J. Deng, “Raft: recurrent all-pairs field transforms for optical flow,” in _Proceedings of European Conference on Computer Vision_ , 2020, pp. 402–419.
  * [29] S. Niklaus and F. Liu, “Context-aware synthesis for video frame interpolation,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1701–1710.
  * [30] X. Xu, L. Siyao, W. Sun, Q. Yin, and M.-H. Yang, “Quadratic video interpolation,” in _Proceedings of Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [31] Y. Liu, L. Xie, L. Siyao, W. Sun, Y. Qiao, and C. Dong, “Enhanced quadratic video interpolation,” in _Proceedings of European Conference on Computer Vision_ , 2020, pp. 41–56.
  * [32] Z. Chi, R. Mohammadi Nasiri, Z. Liu, J. Lu, J. Tang, and K. N. Plataniotis, “All at once: temporally adaptive multi-frame interpolation with advanced motion modeling,” in _Proceedings of European Conference on Computer Vision_ , 2020, pp. 107–123.
  * [33] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,” _International Journal of Computer Vision_ , vol. 127, no. 8, pp. 1106–1125, 2019.
  * [34] F. A. Reda, D. Sun, A. Dundar, M. Shoeybi, G. Liu, K. J. Shih, A. Tao, J. Kautz, and B. Catanzaro, “Unsupervised video interpolation using cycle consistency,” in _Proceedings of IEEE International Conference on Computer Vision_ , 2019, pp. 892–900.
  * [35] Y.-L. Liu, Y.-T. Liao, Y.-Y. Lin, and Y.-Y. Chuang, “Deep video frame interpolation using cyclic frame generation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 8794–8802.
  * [36] W. Shen, W. Bao, G. Zhai, L. Chen, X. Min, and Z. Gao, “Blurry video frame interpolation,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5114–5123.
  * [37] P.-E. Forssén and E. Ringaby, “Rectifying rolling shutter video from hand-held devices,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2010, pp. 507–514.
  * [38] O. Saurer, K. Koser, J.-Y. Bouguet, and M. Pollefeys, “Rolling shutter stereo,” in _Proceedings of IEEE International Conference on Computer Vision_ , 2013, pp. 465–472.
  * [39] K. Wang, B. Fan, and Y. Dai, “Relative pose estimation for stereo rolling shutter cameras,” in _Proceedings of IEEE International Conference on Image Processing_ , 2020, pp. 463–467.
  * [40] B. Fan, K. Wang, Y. Dai, and M. He, “Rs-dpsnet: deep plane sweep network for rolling shutter stereo images,” _IEEE Signal Processing Letters_ , vol. 28, pp. 1550–1554, 2021.
  * [41] P. Purkait, C. Zach, and A. Leonardis, “Rolling shutter correction in Manhattan world,” in _Proceedings of IEEE International Conference on Computer Vision_ , 2017, pp. 882–890.
  * [42] Y. Lao, O. Ait-Aider, and A. Bartoli, “Solving rolling shutter 3d vision problems using analogies with non-rigidity,” _International Journal of Computer Vision_ , vol. 129, no. 1, pp. 100–122, 2021.
  * [43] M. Grundmann, V. Kwatra, D. Castro, and I. Essa, “Calibration-free rolling shutter removal,” in _Proceedings of IEEE International Conference on Computational Photography_ , 2012, pp. 1–8.
  * [44] C. Albl, Z. Kukelova, V. Larsson, M. Polic, T. Pajdla, and K. Schindler, “From two rolling shutters to one global shutter,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2505–2513.
  * [45] H. Wu, L. Xiao, and Z. Wei, “Simultaneous video stabilization and rolling shutter removal,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 4637–4652, 2021.
  * [46] H. C. Longuet-Higgins and K. Prazdny, “The interpretation of a moving retinal image,” _Proceedings of the Royal Society of London. Series B. Biological Sciences_ , vol. 208, no. 1173, pp. 385–397, 1980.
  * [47] A. Patron-Perez, S. Lovegrove, and G. Sibley, “A spline-based trajectory representation for sensor fusion and rolling shutter cameras,” _International Journal of Computer Vision_ , vol. 113, no. 3, pp. 208–219, 2015.
  * [48] M. Meingast, C. Geyer, and S. Sastry, “Geometric models of rolling-shutter cameras,” _arXiv preprint arXiv:cs/0503076_ , 2005.
  * [49] L. Oth, P. Furgale, L. Kneip, and R. Siegwart, “Rolling shutter camera calibration,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 1360–1367.
  * [50] D. Sun, S. Roth, and M. J. Black, “Secrets of optical flow estimation and their principles,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2010, pp. 2432–2439.
  * [51] M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” _Communications of the ACM_ , vol. 24, no. 6, pp. 381–395, 1981.
  * [52] E. Ringaby and P.-E. Forssén, “Efficient video rectification and stabilisation for cell-phones,” _International Journal of Computer Vision_ , vol. 96, no. 3, pp. 335–352, 2012.
  * [53] Y. Wang, Y. Yang, Z. Yang, L. Zhao, P. Wang, and W. Xu, “Occlusion aware unsupervised learning of optical flow,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4884–4893.
  * [54] P. Liu, J. Janai, M. Pollefeys, T. Sattler, and A. Geiger, “Self-supervised linear motion deblurring,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 2475–2482, 2020.
  * [55] O. Ronneberger, P. Fischer, and T. Brox, “U-net: convolutional networks for biomedical image segmentation,” in _Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention_ , 2015, pp. 234–241.
  * [56] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _Proceedings of European Conference on Computer Vision_ , 2016, pp. 694–711.
  * [57] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _Proceedings of the International Conference on Learning Representations_ , 2015.
  * [58] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: an open urban driving simulator,” in _Proceedings of the 1st Annual Conference on Robot Learning_ , 2017, pp. 1–16.
  * [59] D. P. Kingma and J. Ba, “Adam: a method for stochastic optimization,” in _Proceedings of the International Conference on Learning Representations_ , 2015.
  * [60] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 586–595.
  * [61] J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in _Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 4104–4113.

|  Bin Fan is currently a PhD student with School of Electronics and
Information, Northwestern Polytechnical University (NPU), Xi’an, China. He
received the B.S. degree in Statistics and the M.E. degree in Control Science
and Engineering from NPU, in 2016 and 2019, respectively. He was selected to
CVPR 2022 Doctoral Consortium (the only one among Chinese universities). He
co-organized the ACCV 2022 tutorial on the topic of rolling shutter cameras.
His research interests include computer vision, computational photography, 3D
reconstruction, and deep learning.
---|---
|  Yuchao Dai is currently a Professor with School of Electronics and
Information at the Northwestern Polytechnical University (NPU). He received
the B.E. degree, M.E degree and Ph.D. degree all in signal and information
processing from Northwestern Polytechnical University, Xi’an, China, in 2005,
2008 and 2012, respectively. He was an ARC DECRA Fellow with the Research
School of Engineering at the Australian National University, Canberra,
Australia. His research interests include structure from motion, multi-view
geometry, low-level computer vision, deep learning, compressive sensing and
optimization. He won the Best Paper Award in IEEE CVPR 2012, the DSTO Best
Fundamental Contribution to Image Processing Paper Prize at DICTA 2014, the
Best Algorithm Prize in NRSFM Challenge at CVPR 2017, the Best Student Paper
Prize at DICTA 2017, the Best Deep/Machine Learning Paper Prize at APSIPA ASC
2017, the Best Paper Award Nominee at IEEE CVPR 2020. He served as Area Chair
in CVPR, ICCV, ACM MM, ACCV, WACV, etc. He serves as Publicity Chair in ACCV
2022.
---|---
|  Hongdong Li is a Chief Investigator of the Australian Centre of Excellence
for Robotic Vision, professor of Computer Science with the Australian National
University. He joined the RSISE of ANU from 2004. He was a Visiting Professor
with the Robotics Institutes, Carnegie Mellon University, doing a sabbatical
in 2017-2018. His research interests include 3D computer vision, machine
learning, autonomous driving, virtual and augmented reality, and mathematical
optimization. He is an Associate Editor for IEEE Transactions on PAMI, and
IVC, and served as Area Chair for recent CVPR, ICCV and ECCV. He was the
winner of the 2012 CVPR Best Paper Award, the 2017 Marr Prize (Honorable
Mention), a finalist for the CVPR 2020 best paper award, ICPR Best student
paper award, ICIP Best student paper award, DSTO Fundamental Contribution to
Image Processing Prize at DICTA 2014, Best algorithm award in CVPR NRSFM
Challenge 2017, and a Best Practice Paper (honourable mention) award at WACV
2020. He is a Co Program Chair for ACCV 2018 and co general chair for ACCV
2022, Co-Publication Chair for IEEE ICCV 2019.
---|---