11institutetext:  The University of Tokyo
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>11email:
<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>11email:
<EMAIL_ADDRESS>
# RS-NeRF: Neural Radiance Fields from Rolling Shutter Images

Muyao Niu  Tong Chen  Yifan Zhan  Zhuoxiao Li
Xiang Ji  Yinqiang Zheng $\star$$\star$

###### Abstract

Neural Radiance Fields (NeRFs) have become increasingly popular because of
their impressive ability for novel view synthesis. However, their
effectiveness is hindered by the Rolling Shutter (RS) effects commonly found
in most camera systems. To solve this, we present RS-NeRF, a method designed
to synthesize normal images from novel views using input with RS distortions.
This involves a physical model that replicates the image formation process
under RS conditions and jointly optimizes NeRF parameters and camera extrinsic
for each image row. We further address the inherent shortcomings of the basic
RS-NeRF model by delving into the RS characteristics and developing algorithms
to enhance its functionality. First, we impose a smoothness regularization to
better estimate trajectories and improve the synthesis quality, in line with
the camera movement prior. We also identify and address a fundamental flaw in
the vanilla RS model by introducing a multi-sampling algorithm. This new
approach improves the model’s performance by comprehensively exploiting the
RGB data across different rows for each intermediate camera pose. Through
rigorous experimentation, we demonstrate that RS-NeRF surpasses previous
methods in both synthetic and real-world scenarios, proving its ability to
correct RS-related distortions effectively. Codes and data available:
https://github.com/MyNiuuu/RS-NeRF

11footnotetext: Corresponding author

##  1 Introduction

CMOS imaging sensors are commonly employed in a wide range of consumer and
industrial products. Many of these sensors utilize a Rolling Shutter (RS)
mechanism for image capture because of its cost-effectiveness and portability.
In contrast to Global Shutter (GS) cameras, which capture all pixels
simultaneously, rolling shutter cameras capture image pixels row by row in a
sequential manner. Consequently, if the camera is in motion during image
capture, visible distortions can occur. It is widely acknowledged that rolling
shutter distortions pose challenges for various downstream computer vision
tasks, as indicated by previous research [1, 14, 15, 18, 29, 30].
Consequently, the correction of rolling shutter effects has attracted
significant attention in the past [28, 34, 42, 22, 40, 41, 39, 11].

In the emerging field of novel view synthesis, technologies such as NeRF [25]
have become increasingly proficient in generating highly realistic renderings.
NeRF utilizes a continuous volumetric function that is parameterized by a
multilayer perceptron (MLP) to associate 3D locations and 2D view directions
with color and density information. It relies on volume rendering techniques
to enable differentiable rendering.

Although traditional NeRF models deliver exceptional results when applied to
well-calibrated and well-captured images, they encounter difficulties when
dealing with RS effects. For example, in scenarios where the camera is in
motion during image capture, these RS effects can significantly impact the
performance of the model. Given the widespread use of RS cameras in various
industrial applications, it is imperative to take into account this phenomenon
when reconstructing 3D representations using NeRFs.

One straightforward approach involves a two-stage baseline solution, where the
first step corrects the rolling shutter effect in 2D image space, and then the
NeRF is trained using these corrected images. While these baseline methods
enhance the quality of novel view synthesis in NeRF to some extent by
utilizing 2D rolling shutter correction techniques, they can not fully explore
the 3D scene’s geometry, leading to inaccuracies in correspondences,
especially when there is significant camera motion.

In this paper, we introduce RS-NeRF, a robust framework designed to explicitly
incorporate the physics-based formation of rolling shutter effects into the
rendering process. RS-NeRF has the ability to reconstruct a high-quality NeRF
representation from only input data that contains rolling shutter artifacts.
To achieve this, we explicitly model the camera’s motion trajectory throughout
the exposure duration for each frame and combine the RGB rendering outcomes
from different camera poses to generate the rolling shutter image. Throughout
this procedure, we optimize both the NeRF network and the pose estimation
network simultaneously by minimizing the photometric loss.

Building upon the basic RS-NeRF framework, we delve into its shortcomings and
enhance its functionality through various refinements. Initially, we impose a
camera trajectory smoothness regularization to refine our model’s trajectory
prediction, by leveraging the sequential nature of camera movement to yield
improved results. We further identify a critical shortcoming due to the
vanilla RS camera motion model and introduce a multi-sampling technique that
significantly enhances RS-NeRF’s performance. Specifically, we reveal the
issue of the standard camera pose formulation, which assigns distinct poses
per row, potentially limiting RS-NeRF’s efficiency due to limited training
data for each pose. We then make a key observation that multiple poses may
share the same 2D RGB observation for the same 3D point during the exposure
time, which can drastically increase the available training data. To
capitalize on this insight, we introduce and compute the PP-ratio—a measure of
camera pose movement in 3D space relative to pixel movement in 2D space, and
apply this to broaden our training data by sampling out-of-row pixel points.
This approach significantly bolsters our model’s performance by tapping into
the inherent data augmentation possibilities presented by the RS formation
process.

We collect both synthetic and real data to evaluate our proposed RS-NeRF. The
experimental results reveal that RS-NeRF outperforms the preceding models and
demonstrate the effectiveness of our proposed refinements on the vanilla
model. In summary, our contributions are as follows:

  * •

We propose a framework that can reconstruct NeRF from inputs with Rolling
Shutter (RS) effects by modeling the camera trajectory and aggregating RGB
rendering results of different poses during the exposure time.

  * •

By analyzing the vanilla framework in detail, we proposed a series of
improvements to it, including trajectory smoothness regularization and the
multi-sampling algorithm.

  * •

We contribute synthetic and real datasets to evaluate our framework and future
works.

##  2 Related Work

Figure 1:  Pipeline of vanilla RS-NeRF. Given a series of images with RS
distortion, RS-NeRF learns the underlying normal 3D representations by jointly
estimating pose and RGB values for each row. For each RS image, we first
sample the rows and obtain their camera poses via the pose estimation network.
We then sample points for each row and feed them with the estimated camera
pose to the NeRF network and optimize the photo-metric loss with ground
truths.

2D Rolling Shutter Correction. Rolling Shutter Correction (RSC) aims to
restore normal images given the input with RS distortions. In recent years,
several appealing image-based RSC methods [28, 34, 42, 22, 40, 41, 39] have
been developed. These methods take either single RS image or continuous RS
frames as input and predict their global shutter (GS) counterparts. However,
single-image RSC can be seriously ill-posed and struggle to correctly estimate
the GS counterpart in complex scenarios. To reduce the ill-poseness of single-
image RSC, more attention has been paid to multi-image RSC [22, 9, 40, 6, 10,
11]. However, these multi-image RS correction methods, which are built on
image space operations, fail to exploit the 3D geometry of the scene. As a
result, these methods produce multi-view inconsistency results which largely
impact the reconstruction results of NeRF. In contrast, our proposed method
can jointly reconstruct 3D geometry and correct RS images by effectively
aggregating multi-view information of the scene.

Neural Radiance Fields (NeRF). NeRF [25] is an end-to-end model that
synthesizes novel views of complex 3D scenes by optimizing an implicit
continuous volumetric scene function encoded by an MLP. Due to its simplicity
in training and high-quality rendering results, NeRF-based methods [21, 12,
27, 37, 7, 35, 3, 4] have attracted intensive attention from various computer
vision fields, leading to significant improvements in terms of both accuracy
and efficiency. NeRF-based models have also inspired many subsequent works
that extend their continuous neural representation to different setups [13, 8,
17, 2, 24, 36]. Deblur-NeRF [24] and BAD-NeRF [36] reconstruct sharp images
given the blurry input from different view directions. One concurrent work
[19] appears in ICLR’24, which formulates the RS process with B-cubic models.
This work combines the basic modeling of camera motion with the NeRF network
to consider the RS effect, without further technical improvements. In this
paper, we first bake the RS model into NeRF formulation, then analyze several
inherit limitations of the vanilla model, and propose a series of algorithms
to further improve its performance.

##  3 RS-NeRF

In this section, we present the specifics of RS-NeRF. Initially, we explain
the basic formulation of RS-NeRF in Sec. 3.1. We also note that the vanilla
model possesses specific shortcomings that hinder its effectiveness. To
address these, we suggest several improvements to enhance its capabilities,
which are detailed in Sec. 3.2 and Sec. 3.3.

###  3.1 Vanilla RS-NeRF Model

Rolling shutter camera model. Unlike Global Shutter (GS) cameras, the Rolling
Shutter (RS) camera captures each row at distinct timestamps. Without loss of
generality, we consider the readout direction of the RS camera to be from top
to bottom, assuming an infinitesimal exposure time for each row, following
existing RS modeling techniques [22, 10, 11]. This mechanism can be described
through a mathematical model as follows:

$\displaystyle[\mathbf{I}^{R}(\mathbf{x})]_{i}=[\mathbf{I}^{G}_{i}(\mathbf{x})]_{i},$
(1)

where $[\mathbf{I}(\mathbf{x})]_{i}$ is an operator to extract the $i$-th row
from an image $\mathbf{I}(\mathbf{x})$,
$\mathbf{I}^{R}(\mathbf{x})\in\mathbb{R}^{H\times W\times 3}$ is the rolling
shutter image, and $\mathbf{I}_{i}^{G}(\mathbf{x})\in\mathbb{R}^{H\times
W\times 3}$ is the virtual global shutter image captured at the same pose as
the $i$-th row of $\mathbf{I}^{R}(\mathbf{x})$. As a result, for one RS image
$\mathbf{I}^{R}(\mathbf{x})$, we denote the pose of the $i$-th row as
$\mathbf{T}_{i}$.

Camera motion trajectory modeling. In our rolling shutter camera model, it is
necessary to establish the poses for each row in each image. Given the
typically short exposure time of an RS image, we use a linear model to
approximate the motion of the camera. Specifically, we start by defining two
camera poses: one for the first row, denoted as $\mathbf{T}_{0}$, and one for
the last row, $\mathbf{T}_{H-1}$. For the rows in between, we linearly
interpolate their poses within the Lie algebra of $\mathbf{S}\mathbf{E}(3)$,
as informed by prior research [20, 36, 23]:

$\displaystyle\mathbf{T}_{i}=\mathbf{T}_{0}\cdot(1-\rho)+\mathbf{T}_{H-1}\cdot\rho,$
(2)

where the interpolation coefficient $\rho=\frac{i}{H-1}$ is based on the row
index $i$. Thus, for each RS image, RS-NeRF receives one pose initialization,
then optimizes the poses at both ends and then determines the poses for each
intermediate row through linear interpolation. This interpolation method for
calculating each row’s pose is fully differentiable, enabling the optimization
of pose parameters using a series of rolling shutter images.

Optimizing NeRF from RS observation. For a set of $K$ rolling shutter images,
we select $M$ rows per image and $N$ points per row, resulting in $K\times
M\times N$ distinct camera rays for each iteration. These rays are used to
concurrently optimize the NeRF network and the pose parameters. Specifically,
for each image, we initially use the row indices to access the pose estimation
network. This allows us to determine the camera ray
$\hat{\mathbf{r}}(\theta_{p};u,v))$ for each sampled pixel point $(u,v)$.
Following this, these rays are employed to train the NeRF network by computing
the ensuing photometric loss:

$\displaystyle\mathcal{L}=||\mathbf{C}(\mathbf{r}(u,v))-\hat{\mathbf{C}}(\theta_{n};\hat{\mathbf{r}}(\theta_{p};(u,v))||_{2},$
(3)

where $\theta_{n}$ represents the trainable parameters of the NeRF network,
and $\theta_{p}$ denotes the learnable parameters for camera poses.
$\hat{\mathbf{C}}(\theta_{n};\hat{\mathbf{r}}(\theta_{p};(u,v))$ refers to the
predicted RGB value for the pixel at coordinates $(u,v)$, while
$\mathbf{C}(\mathbf{r}(u,v))$ denotes the corresponding RGB ground truth
value. The workflow of our vanilla RS-NeRF model is depicted in Fig. 1.

figuret

Figure 2:  Smoothness regularization on camera trajectory. For each pair of
adjacent trajectory vectors $\overrightarrow{\mathbf{d}}_{k}$ and
$\overrightarrow{\mathbf{d}}_{k+1}$, we compute the mid-point trajectory
vector $\overrightarrow{\mathbf{d}}_{mid}$ and their unit vectors
$\overrightarrow{\mathbf{n}}_{k}$, $\overrightarrow{\mathbf{n}}_{k+1}$, and
$\overrightarrow{\mathbf{n}}_{mid}$. We then apply the $L_{2}$ regularization
between $\overrightarrow{\mathbf{n}}_{mid}$ and
$\operatorname{mean}(\overrightarrow{\mathbf{n}}_{k},\overrightarrow{\mathbf{n}}_{k+1})$.

###  3.2 Trajectory Smoothness Regularization.

Given the previous description of our model, it is evident that accurately
estimating the camera pose for each row in RS frames is crucial for
effectively reconstructing the normal NeRF representation. We observe that the
basic RS-NeRF struggles with precise camera trajectory estimation, often
resulting in high-frequency predictions, as further analyzed in our
experiments. This issue arises because the network focuses only on the
photometric loss for individual images, neglecting the consistent movement
characteristics of the camera across different RS frames. To address this, we
introduce a camera trajectory smoothness regularization, based on the
expectation of a smooth and continuous camera trajectory in 3D space.
Specifically, for every two adjacent RS frames $\mathbf{I}^{R}_{k}$ and
$\mathbf{I}^{R}_{k+1}$, we aim to minimize the following loss during the
optimization process:

$\displaystyle\mathcal{L}_{s}=||\overrightarrow{\mathbf{n}}_{mid}-\operatorname{mean}(\overrightarrow{\mathbf{n}}_{k}+\overrightarrow{\mathbf{n}}_{k+1})||_{2}.$
(4)

As illustrated in Fig. 2, $\overrightarrow{\mathbf{n}}_{k}$ and
$\overrightarrow{\mathbf{n}}_{k+1}$ represent the unit direction vectors of
the estimated camera trajectories for RS frames $\mathbf{I}^{R}_{k}$ and
$\mathbf{I}^{R}_{k+1}$, respectively. Meanwhile,
$\overrightarrow{\mathbf{n}}_{mid}$ denotes the unit direction vector for the
camera trajectory during the exposure interval between these two RS frames.

$N_{pose}$ | On Traj. / Out Traj.
---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
$100$ | 24.36/25.16 | .7321/.7727 | .0703/.0578
$400$ | 24.06/24.66 | .7245/.7536 | .0697/.0579

Table 1: Quantitative results for different $N_{pose}$. $N_{pose}=100$ yields
better results than $N_{pose}=400$.

Figure 3: Qualitative comparisons for different $N_{pose}$. The artifact
decreases as $N_{pose}$ grows.

###  3.3 Multi-Sampling Algorithm

As discussed in Sec. 3.1, each row in an RS frame is assigned a unique camera
pose. For example, in an RS frame with 400 rows, there are 400 distinct camera
poses. This rigorous approach aligns with the real-world, physics-based
photography process of RS cameras but also presents certain challenges. For
example, under this framework, the data points for each pose are confined to a
single row. This limitation can significantly affect our model’s performance
due to the sparse training data available for each pose.

Figure 4:  Motivation for the multi-sampling algorithm. Up: Three consecutive
RS frames, labeled as (a), (b), and (c), along with the optical flow between
each pair of frames, shown in (d) and (e). Down: For a single point in 3D
space, we observe its 2D projections in the first (marked in yellow) and
second (marked in red) RS frames. These projections exhibit a row displacement
of 8 and a column displacement of 18, as depicted in (f). However, during this
interval, the camera undergoes a shift of at least 392 poses, as indicated in
(g).

A straightforward solution to this problem is to reduce the number of poses
per image. For example, assigning 200 poses to each RS frame means that all
two adjacent rows would share the same pose. This approach can lead to
blurriness since these rows are captured at different poses, but it also
doubles the amount of data available for training each pose. To further
explore this trade-off, we conducted a series of experiments. Fig. 3 shows
qualitative results for different intermediate pose numbers $N_{pose}$,
indicating that the artifacts decrease as $N_{pose}$ increases. Interestingly,
we observed that using 100 poses per frame often yields results comparable to
or even better than using 400 poses. The quantitative results presented in
Tab. 1 also support the superiority of having 100 poses per frame.

This observation seems counter-intuitive at first, as one might assume that
having fewer than 400 poses per frame would lead to poorer results due to
artifacts like blurriness. However, it can be rationally explained. As
demonstrated in Fig. 4, for a given point in 3D space, the 2D projections on
the first and second RS frames show relatively small pixel displacements (8 in
row and 12 in column). Meanwhile, the camera shifts through at least 392 poses
in 3D space. This is based on the assumption that there is no overlap in the
trajectories of two adjacent frames (indicated as ’$\geq$ 0 poses’ in the
figure), which is naturally satisfied given the sparse observation setting and
short exposure times typically involved.

Following our prior analysis, we introduced a multi-sampling algorithm aimed
at explicitly increasing the sample points for each intermediate camera pose.
The fundamental concept involves computing the ratio of camera pose
displacement to pixel displacement (termed the PP-ratio) between two adjacent
frames for the same 3D point. For example, a pixel displacement of 18 combined
with a camera pose shift of 392 results in a PP-ratio of 21. This implies that
at least 21 consecutive camera poses share the same pixel value, which means
that all these 21 camera poses can utilize this RGB value for training
purposes.

For a pixel $\mathbf{x}=(u,v)$ located in the $u$-th column and $v$-th row of
the $k$-th RS frame $\mathbf{I}^{R}_{k}(\mathbf{x})$, we determine both the
forward PP-ratio $\rho_{k\rightarrow{k+1}}(\mathbf{x})$ and the backward PP-
ratio $\rho_{k\rightarrow{k-1}}(\mathbf{x})$. This is because the point
represented by $\mathbf{x}$ can transition from
$\mathbf{I}^{R}_{k}(\mathbf{x})$ to $\mathbf{I}^{R}_{k+1}(\mathbf{x})$ or
$\mathbf{I}^{R}_{k-1}(\mathbf{x})$. Consequently, all camera poses within the
range of
$[v-\rho_{k\rightarrow{k-1}}(\mathbf{x}),v+\rho_{k\rightarrow{k+1}}(\mathbf{x})]$
are capable of sampling the RGB value of $\mathbf{x}$ for training purposes.

To compute the forward PP-ratio $\rho_{k\rightarrow{k+1}}(\mathbf{x})$ for a
pixel $\mathbf{x}$ in the $k$-th RS frame $\mathbf{I}^{R}_{k}(\mathbf{x})$,
the first step is to estimate the forward optical flow
$F_{k\rightarrow{k+1}}(\mathbf{x})$, utilizing RAFT [33]. Consequently, the
row displacement $\Delta^{row}_{k\rightarrow{k+1}}(\mathbf{x})$ and column
displacement $\Delta^{col}_{k\rightarrow{k+1}}(\mathbf{x})$ of $\mathbf{x}$
from $\mathbf{I}^{R}_{k}(\mathbf{x})$ to $\mathbf{I}^{R}_{k+1}(\mathbf{x})$
would be the second and the first element of
$F_{k\rightarrow{k+1}}(\mathbf{x})$:

$\displaystyle
F_{k\rightarrow{k+1}}(\mathbf{x})=\operatorname{RAFT}\left(\mathbf{I}^{R}_{k}(\mathbf{x}),\mathbf{I}^{R}_{k+1}(\mathbf{x})\right),$
(5)
$\displaystyle\Delta^{col}_{k\rightarrow{k+1}}(\mathbf{x}),\Delta^{row}_{k\rightarrow{k+1}}(\mathbf{x})=F_{k\rightarrow{k+1}}(\mathbf{x}),$
(6)

and the calculation are similar for the backward situation.

We then calculate the forward and backward camera pose shift as:

$\displaystyle\Delta^{pose}_{k\rightarrow{k+1}}(\mathbf{x})$
$\displaystyle=v+\Delta^{row}_{k\rightarrow{k+1}}(\mathbf{x})+(H-v),$ (7)
$\displaystyle=H+\Delta^{row}_{k\rightarrow{k+1}}(\mathbf{x}),$ (8)
$\displaystyle\Delta^{pose}_{k\rightarrow{k-1}}(\mathbf{x})$
$\displaystyle=\left(H-(v+\Delta^{row}_{k\rightarrow{k-1}}(\mathbf{x}))\right)+v,$
(9) $\displaystyle=H-\Delta^{row}_{k\rightarrow{k-1}}(\mathbf{x}).$ (10)

Finally, we calculate forward and backward PP-ratio as:

$\displaystyle\rho_{k\rightarrow{k+1}}(\mathbf{x})=\frac{\Delta^{pose}_{k\rightarrow{k+1}}(\mathbf{x})}{\max(\Delta^{col}_{k\rightarrow{k+1}}(\mathbf{x}),\Delta^{row}_{k\rightarrow{k+1}}(\mathbf{x}))},$
(11)
$\displaystyle\rho_{k\rightarrow{k-1}}(\mathbf{x})=\frac{\Delta^{pose}_{k\rightarrow{k-1}}(\mathbf{x})}{\max(\Delta^{col}_{k\rightarrow{k-1}}(\mathbf{x}),\Delta^{row}_{k\rightarrow{k-1}}(\mathbf{x}))}.$
(12)

Therefore, when dealing with a sampled pixel $RGB(\mathbf{x})$ in the RS frame
$\mathbf{I}^{R}_{k}(\mathbf{x})$, where $\mathbf{x}=(u,v)$, rather than
directly using the $v$-th pose $\mathbf{T}_{v}$ of
$\mathbf{I}^{R}_{k}(\mathbf{x})$ for training, we randomly sample
$\mathbf{T}_{\hat{v}}$ where

$\displaystyle\hat{v}\in[v-\rho_{k\rightarrow{k-1}}(\mathbf{x}),v+\rho_{k\rightarrow{k+1}}(\mathbf{x})].$
(13)

figuret

Figure 5:  Comparing the accessible training points in one RS image under
different strategies. The image size is $H\times W$, providing $H$ poses.
Distinct circle hues represent different RGB values. In case C, pixels
transported from other poses are with dash contours, and the number signifying
that it originates from the $i$-th pose.

Why does Multi-Sampling work? Let us consider a basic task: optimizing NeRF
with $H$ known poses and images (non-RS). Case A: Only one row is available
per pose. Case B: all rows are accessible. Clearly, case B would yield a
better NeRF due to more data per pose. Moving to our RS setting as depicted in
Fig.5, case A is the vanilla RS observation for one RS frame ($H$ rows lead to
$H$ poses), and case B is the virtual GS observation, where all rows are
available for each pose. Thus, case B is the upper bound for case A.
Essentially, MS aims to transport all possible pixel values from nearby poses
to fill the missing observation that can be potentially recovered (case C in
Fig.5), _i.e_., partially restore the virtual GS observation from the vanilla
RS observation, by allocating additional poses for each pixel, which ensures
better results than vanilla RS. Also, interpolating two poses does not harm
MS. We optimize two poses for each frame, and each in-between pose becomes a
fixed blend ratio of two poses. The best case is virtual GS (Fig.5(B)), where
every pixel coordinate can be sampled in all ratios
($\frac{0}{H},\frac{1}{H-1},...,\frac{H}{0}$), resulting in a total of
H$\times$W$\times$H data points to optimize the two poses. In vanilla RS
(Fig.5(A)), each coordinate only contributes to one ratio determined by row
indices, producing only 1$\times$W$\times$H data points. With MS (Fig.5(C)),
some pixels are replicated to other poses, allowing multiple ratios for them.
Thus, the number of data points is greater than 1$\times$W$\times$H but less
than H$\times$W$\times$H. This ensures better pose estimation and view
synthesis than vanilla RS.

| $\mathcal{L}_{s}$ | MS | Torii | Wine | Pool | Factory | Tanabata | Average
---|---|---|---|---|---|---|---|---
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
On Traj. |  |  | 23.45 | .7535 | .0448 | 23.53 | .7019 | .0681 | 22.99 | .6111 | .0784 | 19.53 | .5018 | .1072 | 17.86 | .4921 | .1030 | 21.47 | .6121 | .0803
✓ |  | 22.25 | .6729 | .0564 | 25.60 | .7749 | .0712 | 27.22 | .7705 | .0582 | 21.81 | .6128 | .0949 | 23.42 | .7916 | .0679 | 24.06 | .7245 | .0697
| ✓ | 26.65 | .8572 | .0342 | 25.55 | .7637 | .0682 | 25.86 | .7303 | .0555 | 25.45 | .7754 | .0708 | 20.94 | .6802 | .0832 | 24.89 | .7614 | .0624
✓ | ✓ | 30.97 | .9058 | .0395 | 28.36 | .8482 | .0675 | 28.50 | .8137 | .0562 | 25.43 | .7673 | .0867 | 26.40 | .8614 | .0651 | 27.93 | .8393 | .0630
Out Traj. |  |  | 23.26 | .7561 | .0428 | 24.05 | .7169 | .0601 | 23.27 | .6218 | .0738 | 19.98 | .5426 | .0892 | 17.85 | .5065 | .0942 | 21.68 | .6288 | .0720
✓ |  | 22.39 | .6875 | .0501 | 26.62 | .8113 | .0579 | 27.29 | .7783 | .0489 | 22.72 | .6624 | .0815 | 24.27 | .8286 | .0514 | 24.66 | .7536 | .0579
| ✓ | 26.58 | .8611 | .0308 | 25.96 | .7750 | .0612 | 25.49 | .7080 | .0549 | 24.38 | .7446 | .0742 | 20.73 | .6850 | .0751 | 24.63 | .7547 | .0592
✓ | ✓ | 33.07 | .9499 | .0243 | 29.27 | .8703 | .0560 | 28.77 | .8313 | .0458 | 25.11 | .7738 | .0807 | 27.02 | .8881 | .0498 | 28.65 | .8627 | .0513

Table 2: Quantitative results on synthetic scenes for ablation study.
$\mathcal{L}_{s}$ denotes the camera trajectory smoothness regularization, and
‘MS’ denotes the multi-sampling algorithm. We color code each result as best
and second best.

##  4 Experiments

###  4.1 Settings

Datasets. Evaluating RS-aware view synthesis technologies requires synthetic
datasets that have multi-view images with RS distortions and the corresponding
GS ground truth. We synthesize 5 scenes via blender to evaluate our method.
For each scene, we rendered images from various camera poses while moving the
camera gradually around the scene in a forward-facing manner. We then combined
rows from different poses to produce the final RS images. Each scene comprises
34 images with a resolution of $400\times 400$. We employed the pose of the
middle row of each RS image as the initial camera pose for our model. We also
analyze our model’s robustness to pose initialization in the experiment
section. For evaluation, we used 34 on-trajectory views and 16 out-trajectory
views to test RS correction and novel view synthesis results, respectively. To
test our model in real-world scenarios, we also captured real scenes using the
RS camera EO-1312C. These RS images were captured by manually moving the
camera during exposure, in a forward-facing manner, with a resolution of
$400\times 500$. We utilize COLMAP [31, 32] for the initial camera pose in our
model.

Evaluation Metrics. The quality of the rendered image is evaluated with the
commonly used metrics including PSNR, SSIM and LPIPS [38]. To evaluate the
accuracy of the estimated camera trajectory, we calculate the Mean Square
Trajectory Error (MSTE) with the ground-truth camera trajectory.

###  4.2 Implementation Details

In each training iteration, we randomly choose 64 rows from each of the images
and 2 pixels from each of these rows. We employ the Adam optimizer [16] with
$\beta_{1}=0.9$ and $\beta_{2}=0.999$ to optimize our model. The learning rate
is initially set at $5\times 10^{-4}$ and undergoes exponential decay
throughout the optimization process. To avoid potential errors from inaccurate
optical flow estimation, we disable multi-sampling and use the vanilla
sampling strategy in the later part of the training process. This makes the
model use much fewer but absolutely accurate data points in the final training
phase. Each scene is trained for 200,000 iterations. All this training is
conducted on a single NVIDIA 4090 GPU.

Figure 6: Qualitative results on synthetic scenes for ablation study. The
corresponding error maps and zoomed regions are visualized in the bottom. Our
full model has the smaller error to the GT, with less spatial misalignment and
twist caused by the RS distortion.

tablet

Vanilla | Vanilla+$\mathcal{L}_{s}$ | Vanilla+MS | Vanilla+$\mathcal{L}_{s}$+MS
---|---|---|---
0.0509 | 0.0358 | 0.0462 | 0.0319

Table 3:  Average MSTE (the lower the better) results for ablation study.

###  4.3 Ablation Study

To comprehensively evaluate the effectiveness of each component in our RS-NeRF
model, we carried out extensive ablation studies. The quantitative and
qualitative results are presented in Tab. 2, Fig. 6 and Tab. 3, respectively.

Trajectory Smoothness Regularization. The accurate estimation of camera
trajectory is crucial for effectively restoring normal images from RS inputs,
given the inherent formation process of RS effects. As evidenced in Tab. 2,
RS-NeRF enhanced with trajectory smoothness regularization ($\mathcal{L}_{s}$)
shows notable improvements over the vanilla model. Specifically, it achieves a
PSNR increase of $2.59$ for on-trajectory views and $2.98$ for out-trajectory
views. Additionally, removing $\mathcal{L}_{s}$ from our full model leads to a
PSNR decrease of $3.04$ for on-trajectory views and $4.02$ for out-trajectory
views. The qualitative results further underscore the effectiveness of
$\mathcal{L}_{s}$. As illustrated in Fig. 6, the vanilla RS-NeRF, when
complemented with $\mathcal{L}_{s}$, exhibits reduced spatial misalignment and
a smaller synthesis error to the ground truth. Furthermore, Tab. 3 reveals
that the vanilla RS-NeRF tends to provide an inaccurate trajectory estimation,
primarily focusing on the photo-metric loss for each image and overlooking the
pattern of camera movement across different RS frames. In contrast, the
‘Vanilla + $\mathcal{L}_{s}$’ model, with the addition of smoothness
regularization, yields a more accurate estimation of the camera trajectory,
resulting in improved synthesis outcomes.

Multi-Sampling Algorithm. The Multi-sampling algorithm in RS-NeRF
significantly enhances the training data available for each intermediate pose
by calculating the PP-ratio for each pixel, marking a notable advancement over
the vanilla model. According to the results in Tab. 2, RS-NeRF equipped with
the multi-sampling algorithm shows substantial improvements: a PSNR increase
of $3.42$ for on-trajectory views and $2.95$ for out-trajectory views.
Moreover, excluding the multi-sampling algorithm from our full model leads to
a PSNR reduction of $3.87$ for on-trajectory views and $3.99$ for out-
trajectory views. The qualitative results also affirm the effectiveness of it.
As depicted in Fig. 6, our model, with the multi-sampling algorithm, displays
fewer artifacts and a smaller synthesis error to the ground truth.
Interestingly, the multi-sampling algorithm also seems to enhance trajectory
estimation. As shown in Tab. 3, RS-NeRF with only the multi-sampling algorithm
(Vanilla + MS) provides a more accurate trajectory estimation than the vanilla
model, which likely contributes to its significantly improved synthesis
results.

| Methods | Torii | Wine | Pool | Factory | Tanabata | Average
---|---|---|---|---|---|---|---
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
On Traj. | DU | 21.23 | .6409 | .0845 | 19.87 | .4520 | .1329 | 23.08 | .5394 | .1189 | 19.95 | .5128 | .1325 | 18.17 | .5262 | .1213 | 20.46 | .5343 | .1180
CVR | 19.94 | .6148 | .1454 | 18.86 | .4090 | .2052 | 21.51 | .5173 | .1885 | 18.59 | .4851 | .1883 | 17.45 | .5277 | .1848 | 19.27 | .5108 | .1824
JAM | 20.66 | .6311 | .0872 | 19.50 | .4214 | .1595 | 22.43 | .5180 | .1263 | 19.34 | .4907 | .1510 | 17.87 | .5342 | .1292 | 19.96 | .5191 | .1307
RS-NeRF | 30.97 | .9058 | .0395 | 28.36 | .8482 | .0675 | 28.50 | .8137 | .0562 | 25.43 | .7673 | .0867 | 26.40 | .8614 | .0651 | 27.93 | .8393 | .0630
Out Traj. | NeRF | 19.35 | .5976 | .2535 | 19.06 | .3845 | .3550 | 22.02 | .4740 | .3196 | 17.87 | .4075 | .3570 | 17.56 | .4958 | .3019 | 19.17 | .4719 | .3174
DU+NeRF | 20.33 | .6585 | .2528 | 20.85 | .4923 | .3404 | 23.25 | .5403 | .2860 | 19.16 | .5037 | .3219 | 18.14 | .5226 | .3102 | 20.35 | .5435 | .3022
CVR+NeRF | 23.67 | .7779 | .1967 | 22.27 | .6114 | .2791 | 24.56 | .6802 | .2408 | 19.81 | .5619 | .2996 | 18.58 | .6250 | .2736 | 21.78 | .6513 | .2580
JAM+NeRF | 24.13 | .7852 | .1512 | 22.14 | .5801 | .2822 | 25.51 | .6867 | .1855 | 20.46 | .5834 | .2717 | 18.63 | .6357 | .2388 | 22.18 | .6542 | .2259
RS-NeRF | 33.07 | .9499 | .0243 | 29.27 | .8703 | .0560 | 28.77 | .8313 | .0458 | 25.11 | .7738 | .0807 | 27.02 | .8881 | .0498 | 28.65 | .8627 | .0513

Table 4: Quantitative comparison on synthetic scenes of different methods. DU,
CVR, and JAM represent DeepUnroll [22], CVR [10], and JAMNet [11]
respectively. We color code each result as best and second best. Figure 7:
Qualitative comparison results on synthetic scenes. The corresponding error
maps and zoomed regions are visualized in the bottom. DU, CVR, and JAM
represent DeepUnroll [22], CVR [10], and JAMNet [11] respectively.

###  4.4 Comparisons

Figure 8: Qualitative comparison results on real-world scenes. Our method
achieves superior performance on real-world scenes.

To evaluate the effectiveness of RS correction for on-trajectory images, our
method is compared with three state-of-the-art 2D RS correction techniques:
DeepUnroll [22], CVR [10], and JAMNet [11]. To evaluate out-trajectory views,
we selected several baselines for comparison. The most basic approach is to
train NeRF directly using RS images. We further compare our method with two-
stage baselines that initially apply state-of-the-art 2D RS correction methods
(DeepUnroll [22], CVR [10], and JAMNet [11]) to restore GS images, followed by
training NeRF with these restored images. For 2D RS correction methods, we
utilized the official checkpoints for evaluation. Furthermore, we compare RS-
NeRF with DRSC [26] and USB-NeRF [19].

| Methods | Torii | Wine | Pool | Factory | Tanabata | Average
---|---|---|---|---|---|---|---
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
On Traj. | USB-NeRF [19] | 26.32 | .8283 | .0825 | 24.72 | .7105 | .1673 | 26.72 | .7130 | .1592 | 24.32 | .7213 | .1369 | 23.05 | .7359 | .1333 | 25.02 | .7418 | .1359
DRSC [26] | 23.04 | .8008 | .0539 | 21.84 | .6895 | .0880 | 22.85 | .6962 | .0874 | 20.85 | .6901 | .0967 | 21.19 | .7492 | .0731 | 21.95 | .7251 | .0798
RS-NeRF | 30.97 | .9058 | .0395 | 28.36 | .8482 | .0675 | 28.50 | .8137 | .0562 | 25.43 | .7673 | .0867 | 26.40 | .8614 | .0651 | 27.93 | .8393 | .0630
Out Traj. | USB-NeRF [19] | 28.95 | .8932 | .0736 | 25.75 | .7446 | .1619 | 28.13 | .7765 | .1576 | 25.25 | .7676 | .1427 | 24.62 | .7853 | .1204 | 26.54 | .7935 | .1313
DRSC [26]+NeRF | 21.51 | .6790 | .2831 | 20.30 | .5145 | .3001 | 22.66 | .5614 | .3021 | 18.99 | .5295 | .2904 | 18.35 | .5643 | .2962 | 20.36 | .5697 | .2944
RS-NeRF | 33.07 | .9499 | .0243 | 29.27 | .8703 | .0560 | 28.77 | .8313 | .0458 | 25.11 | .7738 | .0807 | 27.02 | .8881 | .0498 | 28.65 | .8627 | .0513

Table 5: Quantitative comparisons on synthetic scenes with DRSC [26]+NeRF and
USB-NeRF [19]. We color code each result as best and second best. Figure 9:
Qualitative comparison results on synthetic and real-world scenes with DRSC
[26]+NeRF and USB-NeRF [19].

Quantitative Results. Quantitative comparison results are shown in Tab. 4 and
Tab. 5. 2D RS correction methods do not yield satisfactory outcomes and
receive obvious lower metrics due to twist artifacts and spatial misalignment.
In contrast, our approach significantly exceeds these methods in all metrics
from both on-trajectory and out-trajectory views by incorporating the RS
formation process into the modeling and utilizing the multi-view information
from continuous RS frames. USB-NeRF [19] integrates basic camera motion
modeling, _i.e_., assigning unique camera poses for different rows to account
for the RS effect, without additional technical improvements. The comparison
shows that this approach fails to deliver satisfactory results compared to RS-
NeRF.

Qualitative Results. The qualitative performance of our method is compared to
others through tests on synthetic and real-world datasets. In synthetic
scenes, as shown in Fig. 7, 2D RS correction methods struggle with twisting
artifacts and spatial misalignment for on-trajectory views. In contrast, our
method takes advantage of multi-view information for RS correction, leading to
significantly better results. For out-trajectory views, while 2D RS correction
methods combined with NeRF moderately enhance the reconstruction results, they
still exhibit noticeable visual flaws, such as blurriness. RS-NeRF excels by
integrating the RS formation process into the model and accurately learning
the underlying 3D representation for joint RS correction and novel view
synthesis. Qualitative results in real world scenes are shown in Fig. 8. We
can observe that baseline methods show obvious artifacts, particularly around
object edges. Also, the performance of 2D RS correction combined with NeRF is
poorer in real-world scenes than in synthetic ones, which further demonstrates
the generalization issues of 2D RS correction methods with real-world data.
RS-NeRF, on the other hand, significantly outperforms these methods with clear
edges and detailed rendering, consistent with results in synthetic scenes.
Fig. 9 also demonstrates the superiority of RS-NeRF over DRSC [26]+NeRF and
USB-NeRF [19] on synthetic and real-world scenes.

tablet

Perturbation | On Traj. | Out Traj. | MSTE $\downarrow$
---|---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$
None | 27.93 | .8393 | .0630 | 28.65 | .8627 | .0513 | .0319
$U[-0.01,0.01]$ | 27.97 | .8403 | .0610 | 28.84 | .8680 | .0485 | .0297
$U[-0.03,0.03]$ | 28.31 | .8479 | .0578 | 29.00 | .8693 | .0470 | .0304
$U[-0.05,0.05]$ | 28.47 | .8530 | .0580 | 29.24 | .8769 | .0466 | .0308
$U[-0.07,0.07]$ | 28.43 | .8504 | .0597 | 29.25 | .8771 | .0471 | .0336

Table 6: Quantitative results on synthetic dataset for different noise
perturbations on pose initialization. The results demonstrate that our model
is robust to the accuracy of pose initialization.

###  4.5 Discussion and Limitations

Robustness to pose initialization. We evaluated the robustness of our method
to the precision of pose initialization on synthetic dataset by introducing
random noise of varying scales to the initial poses in the $\textbf{SE}(3)$
space and using these altered poses for initialization. The results, as
detailed in Tab. 6, indicate that our method maintains superior performance
across different levels of noise perturbation. These results confirm the
robustness of RS-NeRF in the accuracy of pose initialization. However, it is
important to note that our method might fail when completely random poses are
used for initialization. This challenge is also an ongoing issue in the field
of NeRF-without-Poses, as discussed in works like [5, 20]. Addressing this
limitation is a direction that we plan to pursue in future research.

Limitations. Similar to previous works that address NeRF with abnormal inputs
[24, 36], our model is designed to handle degradation (referred to the RS
effects in our context) caused by camera movements, since we primarily model
the camera trajectory. When encountering scenarios where RS effects are caused
by object movements, _i.e_., dynamic scenes, our method may fail and produce
unsatisfying results. We plan to involve the modeling of object movements
based on dynamic NeRFs in future work.

##  5 Conclusion

In this paper, we present RS-NeRF, a method designed to synthesize normal
images from novel views using input with Rolling Shutter (RS) distortions. We
formulate a physical model that replicates the image formation process under
RS conditions and jointly optimizes the NeRF parameters and the camera
extrinsic for each image row. We then identify and address the fundamental
limitations of the standard RS-NeRF model by introducing two innovative
approaches: trajectory smoothness regularization and the multi-sampling
algorithm. Through rigorous experimentation, we demonstrate that RS-NeRF
surpasses previous methods in both synthetic and real-world scenarios,
confirming its efficiency in rectifying distortions caused by RS cameras.

## References

  * [1] Albl, C., Kukelova, Z., Larsson, V., Pajdla, T.: Rolling shutter camera absolute pose. IEEE transactions on pattern analysis and machine intelligence 42(6), 1439–1452 (2019)
  * [2] Attal, B., Laidlaw, E., Gokaslan, A., Kim, C., Richardt, C., Tompkin, J., O’Toole, M.: Törf: Time-of-flight radiance fields for dynamic scene view synthesis. Advances in neural information processing systems 34, 26289–26301 (2021)
  * [3] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5855–5864 (2021)
  * [4] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5470–5479 (2022)
  * [5] Bian, W., Wang, Z., Li, K., Bian, J.W., Prisacariu, V.A.: Nope-nerf: Optimising neural radiance field with no pose prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4160–4169 (2023)
  * [6] Cao, M., Zhong, Z., Wang, J., Zheng, Y., Yang, Y.: Learning adaptive warping for real-world rolling shutter correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17785–17793 (2022)
  * [7] Chen, T., Wang, P., Fan, Z., Wang, Z.: Aug-nerf: Training stronger neural radiance fields with triple-level physically-grounded augmentations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15191–15202 (2022)
  * [8] Dave, A., Zhao, Y., Veeraraghavan, A.: Pandora: Polarization-aided neural decomposition of radiance. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII. pp. 538–556. Springer (2022)
  * [9] Fan, B., Dai, Y., He, M.: Sunet: symmetric undistortion network for rolling shutter correction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4541–4550 (2021)
  * [10] Fan, B., Dai, Y., Zhang, Z., Liu, Q., He, M.: Context-aware video reconstruction for rolling shutter cameras. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17572–17582 (2022)
  * [11] Fan, B., Mao, Y., Dai, Y., Wan, Z., Liu, Q.: Joint appearance and motion learning for efficient rolling shutter correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5671–5681 (2023)
  * [12] Garbin, S.J., Kowalski, M., Johnson, M., Shotton, J., Valentin, J.: Fastnerf: High-fidelity neural rendering at 200fps. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14346–14355 (2021)
  * [13] Guo, Y.C., Kang, D., Bao, L., He, Y., Zhang, S.H.: Nerfren: Neural radiance fields with reflections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18409–18418 (2022)
  * [14] Hedborg, J., Forssén, P.E., Felsberg, M., Ringaby, E.: Rolling shutter bundle adjustment. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 1434–1441. IEEE (2012)
  * [15] Kim, J.H., Latif, Y., Reid, I.: Rrd-slam: Radial-distorted rolling-shutter direct slam. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). pp. 5148–5154. IEEE (2017)
  * [16] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [17] Klenk, S., Koestler, L., Scaramuzza, D., Cremers, D.: E-nerf: Neural radiance fields from a moving event camera. IEEE Robotics and Automation Letters (2023)
  * [18] Klingner, B., Martin, D., Roseborough, J.: Street view motion-from-structure-from-motion. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 953–960 (2013)
  * [19] Li, M., Wang, P., Zhao, L., Liao, B., Liu, P.: USB-neRF: Unrolling shutter bundle adjusted neural radiance fields. In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=igfDXfMvm5
  * [20] Lin, C.H., Ma, W.C., Torralba, A., Lucey, S.: Barf: Bundle-adjusting neural radiance fields. In: IEEE International Conference on Computer Vision (ICCV) (2021)
  * [21] Liu, L., Gu, J., Zaw Lin, K., Chua, T.S., Theobalt, C.: Neural sparse voxel fields. Advances in Neural Information Processing Systems 33, 15651–15663 (2020)
  * [22] Liu, P., Cui, Z., Larsson, V., Pollefeys, M.: Deep shutter unrolling network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5941–5949 (2020)
  * [23] Liu, P., Zuo, X., Larsson, V., Pollefeys, M.: Mba-vo: Motion blur aware visual odometry. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5550–5559 (2021)
  * [24] Ma, L., Li, X., Liao, J., Zhang, Q., Wang, X., Wang, J., Sander, P.V.: Deblur-nerf: Neural radiance fields from blurry images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12861–12870 (2022)
  * [25] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
  * [26] Qu, D., Liao, B., Zhang, H., Ait-Aider, O., Lao, Y.: Fast rolling shutter correction in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
  * [27] Reiser, C., Peng, S., Liao, Y., Geiger, A.: Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14335–14345 (2021)
  * [28] Rengarajan, V., Balaji, Y., Rajagopalan, A.: Unrolling the shutter: Cnn to correct motion distortions. In: Proceedings of the IEEE Conference on computer Vision and Pattern Recognition. pp. 2291–2299 (2017)
  * [29] Saurer, O., Koser, K., Bouguet, J.Y., Pollefeys, M.: Rolling shutter stereo. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 465–472 (2013)
  * [30] Saurer, O., Pollefeys, M., Lee, G.H.: Sparse to dense 3d reconstruction from rolling shutter images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3337–3345 (2016)
  * [31] Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4104–4113 (2016)
  * [32] Schönberger, J.L., Zheng, E., Frahm, J.M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14. pp. 501–518. Springer (2016)
  * [33] Teed, Z., Deng, J.: Raft: Recurrent all-pairs field transforms for optical flow. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. pp. 402–419. Springer (2020)
  * [34] Vasu, S., Rajagopalan, A., et al.: Occlusion-aware rolling shutter rectification of 3d scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 636–645 (2018)
  * [35] Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J.T., Srinivasan, P.P.: Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR (2022)
  * [36] Wang, P., Zhao, L., Ma, R., Liu, P.: Bad-nerf: Bundle adjusted deblur neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4170–4179 (2023)
  * [37] Xu, Q., Xu, Z., Philip, J., Bi, S., Shu, Z., Sunkavalli, K., Neumann, U.: Point-nerf: Point-based neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5438–5448 (2022)
  * [38] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586–595 (2018)
  * [39] Zhong, Z., Cao, M., Sun, X., Wu, Z., Zhou, Z., Zheng, Y., Lin, S., Sato, I.: Bringing rolling shutter images alive with dual reversed distortion. In: European Conference on Computer Vision. pp. 233–249. Springer (2022)
  * [40] Zhong, Z., Zheng, Y., Sato, I.: Towards rolling shutter correction and deblurring in dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9219–9228 (2021)
  * [41] Zhou, X., Duan, P., Ma, Y., Shi, B.: Evunroll: Neuromorphic events based rolling shutter image correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17775–17784 (2022)
  * [42] Zhuang, B., Tran, Q.H., Ji, P., Cheong, L.F., Chandraker, M.: Learning structure-and-motion-aware rolling shutter correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4551–4560 (2019)


# A Multilingual Neural Machine Translation Model for Biomedical Data

Alexandre Bérard  Naver Labs Europe  Zae Myung Kim  Naver Papago  Vassilina
Nikoulina  Naver Labs Europe  Eunjeong L. Park  Naver Papago  Matthias Gallé
Naver Labs Europe

###### Abstract

We release a multilingual neural machine translation model, which can be used
to translate text in the biomedical domain. The model can translate from 5
languages (French, German, Italian, Korean and Spanish) into English. It is
trained with large amounts of generic and biomedical data, using domain tags.
Our benchmarks show that it performs near state-of-the-art both on news
(generic domain) and biomedical test sets, and that it outperforms the
existing publicly released models. We believe that this release will help the
large-scale multilingual analysis of the digital content of the COVID-19
crisis and of its effects on society, economy, and healthcare policies. We
also release a test set of biomedical text for Korean-English. It consists of
758 sentences from official guidelines and recent papers, all about COVID-19.

##  1 Motivation

The 2019–2020 coronavirus pandemic has disrupted lives, societies and
economies across the globe. Its classification as a pandemic highlights its
global impact, touching people of all languages. Digital content of all types
(social media, news articles, videos) have focused for many weeks
predominantly on the sanitary crisis and its effects on infected people, their
families, healthcare workers and the society and economy at large. This calls
not only for a large set of tools to help during the pandemic (as evidenced by
the submissions to this workshop), but also for tools to help digest and
analyze this data after it ends. By analyzing the representation and reaction
across countries with different guidelines or global trends, it might be
possible to inform policies in prevention of and reaction to future epidemics.
Several institutions and groups have already started to take snapshots of the
digital content shared during these weeks (Croquet, 2020; Banda et al., 2020).

However, because of its global scale, all this digital content is accessible
in a variety of different languages, and most existing NLP tools remain
English-centric (Anastasopoulos and Neubig, 2019). In this paper we describe
the release of a multilingual neural machine translation model (MNMT) that can
be used to translate biomedical text. The model is both multi-domain and
multilingual, covering translation from French, German, Spanish, Italian and
Korean to English.

Our contributions consist in the release of:

  * •

An MNMT model, and benchmark results on standard test sets;

  * •

A new biomedical Korean-English test set.

This paper is structured as follows: in Section 2 we overview previous work
upon which we build; Section 3 details the model and data settings, and the
released test set; and Section 4 compares our model to other public models and
to state-of-the-art results in academic competitions.

The model can be downloaded at https://github.com/naver/covid19-nmt: the
repository consists in a model checkpoint that is compatible with Fairseq (Ott
et al., 2019), and a script to preprocess the input text.

##  2 Related Work

In order to serve its purpose, our model should be able to process
multilingual input sentences, and generate tailored translations for
COVID-19-related sentences. As far as NMT models are concerned, both
multilingual and domain-specific sentences are just sequences of plain tokens
that should be distinguished internally and handled in a separate manner
depending on the multiple languages or domains. Due to this commonality in
both fields of MNMT and domain adaptation of NMT models, they can be broadly
categorized into two groups: 1) data-centric and 2) model-centric (Chu and
Wang, 2018).

The former focuses on the preparation of the training data such as handling
and selecting from multi-domain (Kobus et al., 2016; Tars and Fishel, 2018) or
multilingual parallel corpora (Aharoni et al., 2019; Tan et al., 2019a); and
generating synthetic parallel data from monolingual corpora (Sennrich et al.,
2015; Edunov et al., 2018).

The model-centric approaches, on the other hand, center on adjusting the
training objectives (Wang et al., 2017; Tan et al., 2019b); modifying the
model architectures (Vázquez et al., 2018; Dou et al., 2019a); and tweaking
the decoding procedure (Hasler et al., 2018; Dou et al., 2019b).

While the two types of approaches are orthogonal and can be utilized in
tandem, our released model is trained using data-centric approaches. One of
the frequently used data-centric methods for handling sentences of multiple
languages and domains is simply prepending a special token that indicates the
target language or domain that the sentence should be translated into (Kobus
et al., 2016; Aharoni et al., 2019). By feeding the task-specific meta-
information via the reserved tags, we signal the model to treat the following
input tokens accordingly. Recent works show that this method is also
applicable to generating diverse translations (Shu et al., 2019) and
translations in specific styles (Madaan et al., 2020).

In addition, back-translation of target monolingual or domain-specific
sentences is often conducted in order to augment the low-resource data (Edunov
et al., 2018; Hu et al., 2019). Afterward, the back-translated data (and
existing parallel data) can be filtered (Xu et al., 2019) or treated with
varying amount of importance (Wang et al., 2019) using data selection methods.
Back-translated sentences can be tagged to achieve even better results
(Caswell et al., 2019).

While myriads of research works on MNMT and domain adaptation exist, the
number of publicly available pre-trained NMT models is still low. For example,
Fairseq, a popular sequence-to-sequence toolkit maintained by Facebook AI
Research, has released ten uni-directional models for translating English,
French, German, and Russian
sentences.111https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md
For its widespread usage, we trained our model using this toolkit.

A large number of public MT models are available thanks to OPUS-
MT,222https://github.com/Helsinki-NLP/OPUS-MT-train created by the Helsinki-
NLP group. Utilizing the OPUS corpora (Tiedemann, 2012), more than a thousand
MT models are trained and released, including several multilingual models
which we use to compare with our model.

To the best of our knowledge, we release the first public MNMT model that is
capable of producing tailored translations for the biomedical domain.

The COVID-19 pandemic has shown the need for multilingual access to hygiene
and safety guidelines and policies (McCulloch, 2020). As an example of crowd-
sourced translation, we point out “The COVID Translate
Project”333https://covidtranslate.org/ which allowed the translation of 75
pages of guidelines for public agents and healthcare workers, from Korean into
English in a matter of days. Although our model could assist in furthering
such initiatives, we do not recommend relying solely on our model for
translating such guidelines, where quality is of uttermost importance.
However, the huge amount of digital content created in the last months around
the pandemic makes such professional translations of all that content not only
infeasible, but sometimes unnecessary depending on the objective. For
instance, we believe that the release of this model can unlock the possibility
of large-scale translation with the aim of conducting data analysis on the
reaction of the media and society on the matter.

##  3 Model Settings and Training Data

The model uses a variant of the Transformer Big architecture (Vaswani et al.,
2017) with a shallower decoder: 16 attention heads, 6 encoder layers, 3
decoder layers, an embedding size of 1024, and a feed-forward dimension of
8192 in the encoder and 4096 in the decoder.

As all language pairs have English as their target language, no special token
for target language was used (language detection can be performed internally
by the model).

As the model performs many-to-English translation, its encoder should be able
to hold most of the complexity. Thus, we increase the capacity of the encoder
by doubling the default size of the feed-forward layer as in Ng et al. (2019).

On the other hand, previous works (Clinchant et al., 2019; Kasai et al., 2020)
have shown that it is possible to reduce the number of decoder layers without
sacrificing much performance, allowing both faster inference, and smaller
network size.

During training, regularization was done with a dropout of $0.1$ and label
smoothing of $0.1$. For optimization, we used Adam (Kingma and Ba, 2014) with
warm-up, and maximum learning rate of $0.001$. The model was trained for 10
epochs and the best checkpoint was selected based on perplexity on the
validation set.

As training data, we used the standard open-accessible datasets, including
biomedical data whenever available, for example, the “Corona Crisis Corpora”
(TAUS, 2020). Following our past success in domain adaptation (Berard et al.,
2019), we used domain tokens (Kobus et al., 2016) to differentiate between
domains, allowing multi-domain translation with a single model. We initially
experimented with more tags, and combinations of tags (e.g., medical
$\rightarrow$ patent or medical $\rightarrow$ political) to allow for more
fine-grained control of the resulting translation. The results however were
not very conclusive, and often under-performed. An exception worth noting was
the case of transcribed data such as TED talks, and OpenSubtitles, which are
not the main targets of this work. Therefore, for simplicity, we used only two
tags: medical and back-translation. No tag was used with training data that
does not belong to one of these two categories.

In addition to biomedical data, we also used back-translated data, although
only for Korean, the language with the smallest amount of training data (13.8M
sentences). Like Arivazhagan et al. (2019), we used a temperature parameter of
5, to give more chance to Korean. Additionally, the biomedical data was
oversampled by a factor of 2. Table 1 details the amount of training sentences
used for each language and each domain tag.

As for pre-processing, we cleaned the available data by conducting white-space
normalization and NFKC normalization. We filtered noisy sentence pairs based
on length (min. 1 token, max. 200), and automatic language identification with
langid.py (Lui and Baldwin, 2012).

We trained a lower-cased shared BPE model using SentencePiece (Kudo and
Richardson, 2018) by using 6M random lines for each language (including
English). We filtered out single characters with fewer than 20 occurrences
from the vocabulary. This results in a shared vocabulary of size 76k.

We reduced the English vocabulary size to speed up training and inference, by
setting a BPE frequency threshold of 20, which gives a target vocabulary of
size 38k. To get the benefits of a shared vocabulary (i.e., tied source/target
embeddings) we sorted the source Fairseq dictionary to put the 38k English
tokens at the beginning, which lets us easily share the embedding matrix
between the encoder and the decoder.444We modified the released checkpoint for
it to work out-of-the box with vanilla Fairseq.

The BPE segmentation is followed by inline-casing (Berard et al., 2019), where
each token is lower-cased and directly followed by a special token specifying
its case (<T> for title case, <U> for all caps, no token for lower-case).
Word-pieces whose original case is undefined (e.g., “MacDonalds”) are split
again into word-pieces with defined case (“mac” and “donalds”).

Language | Total | General | BT | Biomed.
---|---|---|---|---
French | 128.8 | 125.0 | – | 3.8
Spanish | 92.9 | 90.8 | – | 2.1
German | 87.3 | 84.8 | – | 2.5
Italian | 45.6 | 44.9 | – | 0.7
Korean | 13.8 | 5.7 | 8.0 | 0.1
Total | 368.4 | 351.2 | 8.0 | 9.2
Table 1: Amount of training data in millions of sentences. BT refers to back-
translation from English to Korean.

###  3.1 New Korean-English Test Set

To benchmark the performance on the COVID-19 domain, we built an in-domain
test set for Korean-English, as it is the only language pair that is not
included in the Corona Crisis Corpora.

The test set contains 758 Korean-English sentence pairs, obtained by having
English sentences translated into Korean by four professional Korean
translators. We note that any acronym written without its full form in the
source sentence is kept the same in the translation unless it is very widely
used in general. The English sentences were distributed among the translators
with the same guidelines to get consistent tone and manner.

We gathered English sentences from two sources: 1) The official English
guidelines and reports from Korea Centers for Disease Control and Prevention
(KCDC)555http://ncov.mohw.go.kr/en under Ministry of Health and Welfare of
South Korea (258 sentences); and 2) Abstracts of biomedical papers on SARS-
CoV-2 and COVID-19 from arXiv,666https://arxiv.org
medRxiv777https://www.medrxiv.org and bioRxiv888https://www.biorxiv.org (500
sentences). The sentences were handpicked, focusing on covering diverse
aspects of the pandemic, including safety guidelines, government briefings,
clinical tests, and biomedical experimentation.

##  4 Benchmarks

We benchmarked the released multilingual models against: 1) reported numbers
in the literature, and 2) other publicly released models. We used OPUS-MT, a
large collection (1000+) of pre-trained models released by the NLP group at
University of Helsinki. Note that these models were trained with much smaller
amounts of training data.

We note that the biomedical test sets (Medline) are very small (around 600
lines). We do not report comparison for Spanish-English newstest2013, as the
latest reported numbers are outdated (the best WMT entry achieved $30.4$).

Our single model obtains competitive results on “generic” test sets (News and
IWSLT), on par with the state of the art. We also obtain strong results on the
biomedical test sets. Note the SOTA models were trained to maximize
performance in the very specific Medline domain, for which training data is
provided. While we included this data in our tagged biomedical data, we did
not fine-tune aggressively over it.

Table 2 shows the BLEU scores for the Korean-English COVID-19 test set. The
results greatly outperform existing public Korean-English models, even more so
than with the IWSLT test sets (Table 3).

Model | arXiv | KCDC | Full
---|---|---|---
Ours | 36.5 | 38.3 | 37.2
Ours (medical) | 36.6 | 38.6 | 37.4
OPUS-MT | 18.7 | 19.0 | 18.8
Table 2: Benchmark of the released model on new Korean-English COVID-19 test set. Language | Model | News | Medline | IWSLT
---|---|---|---|---
French | Ours | 41.00 | 36.16 | 41.09
SOTA | 40.224 | 35.564 | –
OPUS-MT | 36.80 | 33.60 | 38.90
German | Ours | 41.28 | 29.76 | 31.55
SOTA | 40.984 | 28.824 | 32.014
OPUS-MT | 39.50 | 28.10 | 30.30
Spanish | Ours | 36.63 | 46.18 | 48.79
SOTA | – | 43.034 | –
OPUS-MT | 30.30 | 43.30 | 46.10
Italian | Ours |  |  | 42.18
SOTA |  |  | –
OPUS-MT |  |  | 39.70
Korean | Ours |  |  | 21.33
SOTA |  |  | –
OPUS-MT |  |  | 17.60
Table 3: Benchmark of the released model against the best reported numbers and
the public OPUS-MT models. Our scores on the medical test sets are obtained
with the medical tag. No tag is used with the other test sets. The SOTA
numbers for _News_ and _IWSLT_ were either obtained by running the
corresponding models (Berard et al., 2019; Ng et al., 2019). For _Medline_ ,
we copied the results from the WMT19 Biomedical Task report (Bawden et al.,
2019). News consists in the _newstest2019_ for German (WMT19 News Task test
set), _newstest2014_ for French, and _newstest2013_ for Spanish. For IWSLT,
the 2017 test set is used for all but Spanish, where the 2016 one is used.
**footnotetext: NLE @ WMT19 Robustness Task (Berard et al.,
2019)††footnotetext: FAIR @ WMT19 News Task (Ng et al., 2019)‡‡footnotetext:
Reported results in the WMT19 Biomedical Task (Bawden et al., 2019)

##  5 Conclusion

We describe the release of a multilingual translation model that supports
translation in both the general and biomedical domains. Our model is trained
on more than 350M sentences, covering French, Spanish, German, Italian and
Korean (into English). Benchmarks on public test sets show its strength across
domains. In particular, we evaluated the model in the biomedical domain, where
it performs near state-of-the-art, but with the advantage of being a single
model that operates on several languages. To address the shortage of Korean-
English data, we also release a dataset of 758 sentence pairs covering recent
biomedical text about COVID-19.

Our aim is to support research studying the international impact that this
crisis is causing, at a societal, economical and healthcare level.

## References

  * Aharoni et al. (2019) Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019.  Massively multilingual neural machine translation.  _arXiv preprint arXiv:1903.00089_.
  * Anastasopoulos and Neubig (2019) Antonios Anastasopoulos and Graham Neubig. 2019.  Should all cross-lingual embeddings speak english?  _arXiv preprint arXiv:1911.03058_.
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019\.  Massively multilingual neural machine translation in the wild: Findings and challenges.  _arXiv preprint arXiv:1907.05019_.
  * Banda et al. (2020) Juan M Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, and Gerardo Chowell. 2020.  A large-scale covid-19 twitter chatter dataset for open scientific research–an international collaboration.  _arXiv preprint arXiv:2004.03688_.
  * Bawden et al. (2019) Rachel Bawden, Kevin Bretonnel Cohen, Cristian Grozea, Antonio Jimeno Yepes, Madeleine Kittner, Martin Krallinger, Nancy Mah, Aurelie Neveol, Mariana Neves, Felipe Soares, Amy Siu, Karin Verspoor, and Maika Vicente Navarro. 2019\.  Findings of the WMT 2019 biomedical translation shared task: Evaluation for MEDLINE abstracts and biomedical terminologies.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)_ , pages 29–53, Florence, Italy. Association for Computational Linguistics.
  * Berard et al. (2019) Alexandre Berard, Ioan Calapodescu, and Claude Roux. 2019.  Naver labs Europe’s systems for the WMT19 machine translation robustness task.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 526–532, Florence, Italy. Association for Computational Linguistics.
  * Caswell et al. (2019) Isaac Caswell, Ciprian Chelba, and David Grangier. 2019.  Tagged back-translation.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_ , pages 53–63, Florence, Italy. Association for Computational Linguistics.
  * Chu and Wang (2018) Chenhui Chu and Rui Wang. 2018.  A survey of domain adaptation for neural machine translation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1304–1319, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Clinchant et al. (2019) Stephane Clinchant, Kweon Woo Jung, and Vassilina Nikoulina. 2019.  On the use of BERT for neural machine translation.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 108–117, Hong Kong. Association for Computational Linguistics.
  * Croquet (2020) Pauline Croquet. 2020.  Comment les archivistes de la BNF sauvegardent la mémoire du confinement sur internet.  Accessed June 2020.
  * Dou et al. (2019a) Zi-Yi Dou, Junjie Hu, Antonios Anastasopoulos, and Graham Neubig. 2019a.  Unsupervised domain adaptation for neural machine translation with domain-aware feature embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1417–1422.
  * Dou et al. (2019b) Zi-Yi Dou, Xinyi Wang, Junjie Hu, and Graham Neubig. 2019b.  Domain differential adaptation for neural machine translation.  _arXiv preprint arXiv:1910.02555_.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
  * Hasler et al. (2018) Eva Hasler, Adrià de Gispert, Gonzalo Iglesias, and Bill Byrne. 2018.  Neural machine translation decoding with terminology constraints.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 506–512, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hu et al. (2019) Junjie Hu, Mengzhou Xia, Graham Neubig, and Jaime Carbonell. 2019.  Domain adaptation of neural machine translation by lexicon induction.  _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Kasai et al. (2020) Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith. 2020.  Deep encoder, shallow decoder: Reevaluating the speed-quality tradeoff in machine translation.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kobus et al. (2016) Catherine Kobus, Josep Crego, and Jean Senellart. 2016.  Domain control for neural machine translation.  _arXiv preprint arXiv:1612.06140_.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _arXiv preprint arXiv:1808.06226_.
  * Lui and Baldwin (2012) Marco Lui and Timothy Baldwin. 2012.  langid.py: An off-the-shelf language identification tool.  In _Proceedings of the ACL 2012 System Demonstrations_ , pages 25–30, Jeju Island, Korea. Association for Computational Linguistics.
  * Madaan et al. (2020) Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabas Poczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. 2020\.  Politeness transfer: A tag and generate approach.
  * McCulloch (2020) Gretchen McCulloch. 2020.  Covid-19 is history’s biggest translation challenge.
  * Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019\.  Facebook FAIR’s WMT19 news translation task submission.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 314–319, Florence, Italy. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  _arXiv preprint arXiv:1904.01038_.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015.  Improving neural machine translation models with monolingual data.  _CoRR_ , abs/1511.06709.
  * Shu et al. (2019) Raphael Shu, Hideki Nakayama, and Kyunghyun Cho. 2019.  Generating diverse translations with sentence codes.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1823–1827, Florence, Italy. Association for Computational Linguistics.
  * Tan et al. (2019a) Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tie-Yan Liu. 2019a.  Multilingual neural machine translation with language clustering.  _arXiv preprint arXiv:1908.09324_.
  * Tan et al. (2019b) Xu Tan, Yi Ren, Di He, Tao Qin, and Tie-Yan Liu. 2019b.  Multilingual neural machine translation with knowledge distillation.  In _International Conference on Learning Representations_.
  * Tars and Fishel (2018) Sander Tars and Mark Fishel. 2018.  Multi-domain neural machine translation.  _arXiv preprint arXiv:1805.02282_.
  * TAUS (2020) TAUS. 2020.  Taus corona crisis corpus.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Vázquez et al. (2018) Raúl Vázquez, Alessandro Raganato, Jörg Tiedemann, and Mathias Creutz. 2018.  Multilingual nmt with a language-independent attention bridge.  _arXiv preprint arXiv:1811.00498_.
  * Wang et al. (2017) Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017.  Instance weighting for neural machine translation domain adaptation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1482–1488, Copenhagen, Denmark. Association for Computational Linguistics.
  * Wang et al. (2019) Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019.  Improving back-translation with uncertainty-based confidence estimation.  _arXiv preprint arXiv:1909.00157_.
  * Xu et al. (2019) Guanghao Xu, Youngjoong Ko, and Jungyun Seo. 2019.  Improving neural machine translation by filtering synthetic parallel data.  _Entropy_ , 21(12):1213.