11institutetext: Tampere University, Finland
11email: {german.torresvanegas<EMAIL_ADDRESS>
https://github.com/germanftv/ParallaxICB

# Depth-Aware Image Compositing Model for Parallax Camera Motion Blur

German F. Torres  Joni Kämäräinen

###### Abstract

Camera motion introduces spatially varying blur due to the depth changes in
the 3D world. This work investigates scene configurations where such blur is
produced under parallax camera motion. We present a simple, yet accurate,
Image Compositing Blur (ICB) model for depth-dependent spatially varying blur.
The (forward) model produces realistic motion blur from a single image, depth
map, and camera trajectory. Furthermore, we utilize the ICB model, combined
with a coordinate-based MLP, to learn a sharp neural representation from the
blurred input. Experimental results are reported for synthetic and real
examples. The results verify that the ICB forward model is computationally
efficient and produces realistic blur, despite the lack of occlusion
information. Additionally, our method for restoring a sharp representation
proves to be a competitive approach for the deblurring task.

###### Keywords:

blur formation, image compositing blur, neural representations, deblurring

##  1 Introduction

---
Figure 1:  Our blur formation model accurately describes parallax motion blur.
By providing the depth and camera trajectory, we can fit a set of motion
kernels $k_{l}$ and alpha-matte terms $\mathcal{A}_{l}$, which are used to
blend the blur from different layers.

Motion blur is a common problem in photography and in certain computer vision
tasks such as feature matching [22] or object detection [16]. In essence,
motion blur occurs when either the camera or the scene objects, or both, are
in motion during exposure. Recovering the edges and textures of the latent
sharp image, _i.e_. deblurring, remains as an open problem since there are
infinite latent sharp sequences consistent with the generated blur.

In conventional deblurring approaches, there is a model that describes the
formation of the blur, coupled with an image prior that regularizes the
solution space of the optimization problem. A major part of the research has
been conducted upon suitable image priors that characterize natural images [3,
14, 15, 41, 24, 20]. However, the applicability in real scenarios also depends
on the accuracy of the assumed blur formation model. Pioneering works assume
that the blur results from the shift-invariant convolution of the sharp image
with an unknown Point Spread Function (PSF) [7, 18, 19, 39]. For this to be
precise, either of the two possible scenarios must hold, apart from the scene
being static: 1) the camera shake only involves in-plane translation while the
scene is either planar or sufficiently far from the camera, 2) the focal
length of the camera is large and there is no in-plane rotation [38].
Otherwise, camera shake generally induces non-uniform (spatially varying)
blur.

Several deblurring algorithms have been proposed to deal with spatially-
varying blur that is produced by more realistic 6D camera motion [30, 8, 10,
38, 41]. Nevertheless, these works fail at modeling the induced blur in 3D
scenes, especially at depth discontinuities. With the advances in deep
learning, several network architectures have been proposed to handle multiple
types of blur by learning from data [23, 17, 32, 42, 44, 4, 34, 5, 43, 35].
They benefit from not requiring an explicit description of the blur formation
process. In such works, a neural network is trained over large-scale datasets
to restore the sharp image. Deep deblurring represents state-of-the-art on
multiple benchmarks, but their performance depends on the type of blur that is
present in the training set.

Due to the parallax effect, objects positioned at different depths from the
camera produce spatially varying blur, when the camera moves during capture.
Following this line, a number of works have incorporated the depth in their
deblurring methods as an extra auxiliary input [25, 27, 21] or by a joint
estimation process [40, 11, 47], but they do not provide a concrete blur
model.

In this work, we study the impact of the depth variation on the motion blur
focusing on parallax camera motion, _i.e_. when the camera moves parallel to
the image plane. By analyzing the geometry of this type of camera motion, we
identify two realistic scene types where depth plays a significant role in the
produced blur: 1) Macro Photography and 2) Trucking Photography. For such
configurations, we propose a tractable Image Compositing Blur (ICB) model for
parallax motion assuming that the depth and camera trajectory are available.
This model accurately approximates the camera blur under parallax motion (Fig.
1). In addition, we provide evidence that our ICB model, in conjunction with
coordinate-based Multi-Layer Perceptron (MLP) models, can be used to extract a
sharp neural representation from a single blurry image.

In summary, the main contributions are: 1) insight analysis about the scene
configurations and capture settings for which depth becomes meaningful in the
blur formation; 2) a simplified, yet accurate enough, Image Compositing Blur
(ICB) model for parallax depth-induced camera motion blur; 3) an alternative
approach to restore sharp images without the need for training over large
datasets; and 4) one synthetic and one real dataset of realistic scenes that
include pairs of blurry and sharp images with depth maps and camera
trajectories.

##  2 Related work

#### Blur formation models.

Blur formation models have been studied in the context of image deblurring.
Arguably, the simplest model assumes uniform behavior over the whole image. In
this case, the blurred image is presumed to be the result of shift-invariant
convolution with a Point Spread Function (PSF) [7]. However, this model only
holds for very limited practical scenarios.

For the more general case of spatially-varying blur, some works are based on
the projective motion path of the camera shake [30]. Gupta  _et al_. [8]
assume that the blur can be accurately modeled by in-plane camera translation
and rotation. White  _et al_. [38] focus on the blur produced by 3D rotations.
Furthermore, Hirsch  _et al_. [10] model the blur as the linear combination of
patch-based blur kernel basis. Nevertheless, none of these models precisely
determine the blur generation in 3D scenes, especially around abrupt changes
in depth

#### Image deblurring.

Conventional methods for image deblurring are optimization frameworks that
tackle the blur produced by the camera motion. To handle the well-known ill-
posed nature, previous works enforced different image priors in their
solutions, such as Total-Variation [3], normalized sparsity prior [15],
$L_{0}$-norm regularization [41], dark channel prior [24], or discriminative
prior [20].

With the advances in deep learning, several Convolutional Neural Network (CNN)
architectures have been proposed. These architectures only take the blurred
image as input and produce the estimated sharp image. Su et al. [29] used an
encoder-decoder architecture for video deblurring. Nah et al. [23]
incorporated the multi-scale processing approach in their deep network.
Following the multi-scale principle, numerous CNN-based methods have been
introduced including components such as Generative Adversarial Networks (GAN)
[16, 17], Long-Short Term Memory (LSTM) [32], scale-iterative upscaling scheme
[42], half instance normalization [4], multi-scale inputs and outputs [5],
blur-aware attention [34], and multi-stage progressive restoration [44]. More
recently, progress on Transformer [37] and MLP models demonstrate the ability
to handle global-local representations for image restoration tasks [43, 35].

The problem with conventional methods is that they do not use depth in
deblurring and they are computationally expensive. On the contrary, deep
deblurring performance strongly depends on the training data which, in the
case of the above works, do not contain spatial blur induced by depth
variations.

#### Depth-aware deblurring.

The involvement of the depth cue in the motion blur, although not widely
studied, it is not new. Xu and Jia [40] proposed the first work on this track.
They used a stereopsis setup to estimate the depth information and
subsequently perform layer-wise deblurring. Optimization-based solutions have
been introduced for the joint estimation of the scene depth and sharp image,
employing either expectation-maximization [11] or energy-minimization [25]
methods. Sheng  _et al_. [27] proposed an algorithm that iteratively refines
the depth and estimates the latent sharp image from an initial depth map and a
blurry image, using belief propagation and Richardson-Lucy algorithm,
respectively. Park and Lee [26] proposed and alternating energy-minimization
algorithm for the joint dense-depth reconstruction, camera pose estimation,
super-resolution, and deblurring. However, their method requires an image
sequence instead of a single image.

On the deep learning side, Zhou  _et al_. [47] proposed a stereo deblurring
network that internally estimates bi-directional disparity maps to convey
information about the spatially-varying blur that is caused by the depth
variation. Moreover, Li  _et al_. [21] introduced a depth-guided network
architecture for single-image deblurring, which both refines an initial depth
map and restores the sharp image.

The above depth-aware deblurring methods properly acknowledge that depth
changes produce spatially-varying blur, but it is not clear in which cases
this holds. The depth is used as an additional cue for deblurring but, on the
other hand, they do not address the spatial blur due to scene depth variation.
In contrast, we first identify practical scenarios where the depth variations
certainly yield to non-uniform blur. We then characterize how depth and camera
motion result in regions with different blur behavior.

##  3 Geometry of camera motion blur

###  3.1 Fundamentals

#### Projective motion path blur model.

For static scenes, image blur comes from the motion of the camera during the
exposure time. More precisely, the captured blurry image $\mathbf{y}$ is the
summation of the transient sharp images $\\{\mathbf{x}_{m}\\}_{m=1}^{M}$ seen
by the camera in the poses $\\{\vartheta_{m}\\}_{m=1}^{M}$ that follow its
trajectory. Assuming there is a linear transformation
$\mathcal{T}_{\vartheta_{m}}$ that warps the latent sharp image $\mathbf{x}$
to any transient image $\mathbf{x}_{m}$, the blurred image $\mathbf{y}$ can be
expressed as:

$\mathbf{y}=\sum_{m=1}^{M}w_{m}\mathcal{T}_{\vartheta_{m}}(\mathbf{x})+\eta\enspace,$
(1)

where the weight $w_{m}$ indicates the time the camera stays at pose
$\vartheta_{m}$ and $\eta$ error in the model. The transformation
$\mathcal{T}_{\vartheta_{m}}$ is induced by a homography $H_{m}$ such that a
pixel $\mathbf{p}$ from the latent image $\mathbf{x}$ is mapped to the pixel
$\mathbf{p}^{\prime}_{m}$ in the transient image $\mathbf{x}_{m}$. In
homogeneous coordinates,
$[\mathbf{p}^{\prime}_{m}]_{h}=H_{m}[\mathbf{p}]_{h}$, where $[\cdot]_{h}$
denotes the conversion from Cartesian to homogeneous coordinates.

For a camera following a 6D motion trajectory, the homography $H_{m}$ that
relates pixels from the latent image $\mathbf{x}$ to the transient image
$\mathbf{x}_{m}$, which are captured from a planar scene at depth $D$, has the
form:

$H_{m}=C(R_{m}+\frac{1}{D}T_{m}[0,0,1])C^{-1}\enspace,$ (2)

where $R_{m}$ and $T_{m}$ stand for the rotation and translation components,
and $C$ is the intrinsic camera matrix. Eq. (2) reveals that there is non-
uniform blur caused by the depth-dependence of the translation component, as
well as when rotations are introduced. Notwithstanding, the homography model
only holds for fronto-parallel scenes since the warping operator would require
an estimation of the occluded areas that become visible, particularly at the
depth discontinuities.

#### Pixel-Wise Blur (PWB) model.

In general, image blur has a spatially-varying nature. To take this into
account, the blurred image $\mathbf{y}$ can be modeled via convolutions with
pixel-wise kernels $\mathbf{k}(\mathbf{p},\mathbf{u})$:

$\mathbf{y}(\mathbf{p})=\mathbf{x}(\mathbf{p})*\mathbf{k}(\mathbf{p},\mathbf{u})+\eta\enspace,$
(3)

where $*$ denotes to the convolution operator, $\mathbf{p}=(i,j)$ are pixel
coordinates and $\mathbf{u}=(u,v)$ the kernel coordinates. One can blur an
image by computing the Empirical Probability Density Function (EPDF) of pixel
displacements
$\Delta\mathbf{p}^{\prime}_{m}=\mathbf{p}^{\prime}_{m}-\mathbf{p}$. This model
is used as a baseline in our experiments, and its limitations against the
proposed blur formation model are demonstrated.

In the remainder of this section, we take a closer look at the influence of
depth in the blur generation for in-plane camera translations. Here, we
provide insights of what are the scenarios, and to what extent, the depth
should be considered in the deblurring problem.

Figure 2: Blur induced by camera translation of length $s$ for two 3D points
$\mathbf{P}^{(1)}$ and $\mathbf{P}^{(2)}$ with their depth difference of
$\Delta D$.

###  3.2 In-plane camera motion

|
---|---
(a) | (b)
Figure 3: Blur variation determined by Eq. 6, at different depths of the
closest point $D(\mathbf{p}^{(1)})$ (in meters): (a) blur variation in pixels
as function of the depth difference (fixed camera displacement baseline of
$s=3[mm]$); (b) the camera displacement as a function of the depth difference
(fixed blur variation $\Delta T=10$ pixels [px]). Camera focal length is
$F$=2.8[mm] and pixel size 4[$\mu$m] which correspond to settings that can be
found in mobile phone cameras (ultrawide lenses).

Let us first consider a pin-hole camera with a uniform in-plane motion in the
horizontal axis of length $s$ during the exposure time, and two trivial 3D
points $\mathbf{P}^{(1)}$ and $\mathbf{P}^{(2)}$ such that the former
represents the closest point to the camera in the depth direction and the
latter is the farthest as depicted in Fig. 2. On the one hand,
$\mathbf{P}^{(1)}$ and $\mathbf{P}^{(2)}$ are respectively mapped to the
points $\mathbf{p}^{(1)}$ and $\mathbf{p}^{(2)}$ in the latent image
$\mathbf{x}$. On the other hand, they are seen, by the camera at pose
$\vartheta_{m}$, on $\mathbf{p}^{\prime(1)}$ and $\mathbf{p}^{\prime(2)}$. In
this case, the induced homography $H_{m}(\mathbf{p})$ is given by

$H_{m}(\mathbf{p})=\begin{bmatrix}1&0&T_{m}(\mathbf{p})\\\ 0&1&0\\\
0&0&1\end{bmatrix}\enspace,$ (4)

where $T_{m}(\mathbf{p})$ is the image plane translation component that is
dependent on the pixel depth $D(\mathbf{p})$ as

$T_{m}(\mathbf{p})=\frac{sF}{D(\mathbf{p})}\enspace,$ (5)

where $F$ denotes the focal length of the camera. Due to the simplicity of the
motion, the blur extent of an arbitrary 3D point $\mathbf{P}$ in the blurry
image $\mathbf{y}$ is given by
$T_{m}(\mathbf{p})=\mathbf{p}^{\prime}_{x}-\mathbf{p}_{x}$, where $x$ denotes
the horizontal component. Noteworthy, this is equivalent to the disparity in
stereo vision.

#### Blur variation.

Since there is a difference in depth $\Delta
D=D(\mathbf{p}^{(2)})-D(\mathbf{p}^{(1)})$, there must be difference in the
blur extent for $\mathbf{P}^{(1)}$ and $\mathbf{P}^{(2)}$, as illustrated in
Fig. 2. Thus, we define the blur variation $\Delta T$ as the difference in
blur extent between two points at different depths. Expressively, $\Delta
T=T_{m}(\mathbf{p}^{(1)})-T_{m}(\mathbf{p}^{(2)})$. $\Delta T$ measures the
non-uniform behavior of the blur caused by the depth and under in-plane camera
movements. By replacing terms, we get

$\Delta
T=\frac{sF}{D(\mathbf{p}^{(1)})\Big{[}\frac{D(\mathbf{p}^{(1)})}{\Delta
D}+1\Big{]}}\enspace.$ (6)

To gain intuition of the blur variation in practical scenarios, we describe
its behavior in Fig.3 by assuming $F$=2.8[mm] and pixel size of 4[$\mu$m].

#### Macro photography scenes.

Fig. 3(a) illustrates the blur variation $\Delta T$ as a function of the depth
difference $\Delta D$, at different depths of the closest point
$D(\mathbf{p}^{(1)})$, while keeping a fixed camera displacement $s$=3[mm] (a
reasonable choice for natural hand shake). It can be seen that whereas the
blur variation is negligible for far-field scenes no matter what is the depth
variation, non-uniform blur becomes significant for near-field macro scenes
(the closest target $\leq$ 0.1m from the camera) even with rather low depth
variation ($\geq$ 0.1m). Although the blur variation increases as the depth
difference gets higher, there is an upper bound that is determined by $\Delta
T<\frac{sF}{D(\mathbf{p}^{(1)})}$. In conclusion, spatially-variant blur is
particularly affected by the proximity of the scene whenever there is any
variation in depth. Consequently, depth plays a significant role for Macro
Photography scenes. In this setting, images suffer from defocus blur due to
the limited depth-of-field of optics, but defocus blur is a separate issue
addressed in other works [2, 46, 1].

#### Trucking photography scenes.

From another perspective, Fig. 3(b) shows the camera displacement $s$ as a
function of the depth difference $\Delta D$, by assuming a constant blur
variation $\Delta T=10$ pixels, for different depths of the closest point
$D(\mathbf{p}^{(1)})$. In other words, this plot tells us how much the camera
should be moved to produce a blur variation of $10$ pixels. In this case, it
is observed that a few millimeters are sufficient to produce such blur
variation for near-field scenes, regardless of the depth difference. In
contrast, in the case of far-field scenes, such a level of blur variation can
only be achieved through a camera displacement that ranges from tens of
centimeters to a few meters, depending on the depth difference. Such intense
movement is unlikely to happen in natural hand shake, but appears in cases
where the camera is placed on a fast-moving object. For example, when
capturing pictures from inside a moving car. We dub this as Trucking
Photography scenes.

##  4 Image Compositing Blur (ICB) model

From Fig. 3(a), we see that there are depth ranges that yield to nearly the
same amount of blur. Hence, pixels in a particular depth range share a common
2D convolutional kernel that characterizes the blur. Inspired by the defocus
blur formation models of Hassinoff  _et al_. [9] and Ikoma  _et al_. [12], we
present a new parallax motion Image Compositing Blur (ICB) model that takes
the depth into account:

$\mathbf{y}=\sum_{l=0}^{L-1}(\mathbf{x}*k_{l})\cdot\mathcal{A}_{l}+\eta\enspace,$
(7)

where $\\{\mathcal{A}_{l}\\}_{l=0}^{L-1}$ and $\\{k_{l}\\}_{l=0}^{L-1}$ are
the set of alpha-matting terms and blur kernels, respectively; and ”$\cdot$”
is pixel-wise multiplication. We define each alpha matte as:

$\mathcal{A}_{l}=\frac{\hat{\mathcal{R}}_{l}\cdot\mathcal{M}_{l}}{C}\enspace,$
(8)

where $C$ is a normalization constant over the $L$ depth layers (_i.e_.
$C:=\sum_{l=0}^{L-1}\hat{\mathcal{R}}_{l}\cdot\mathcal{M}_{l}$).
$\mathcal{M}_{l}$ are the z-buffers from far to near layers:

$\mathcal{M}_{l}=\prod_{l^{\prime}=l+1}^{L-1}(1-\hat{\mathcal{R}}_{l}^{\prime})\enspace.$
(9)

$\hat{\mathcal{R}}_{l}$ is the smooth spatially-extended version of the depth
region $\mathcal{R}_{l}$. $\hat{\mathcal{R}}_{l}$ is defined as
$\hat{\mathcal{R}}_{l}:=(\mathcal{R}_{l}\oplus\operatorname{supp}{k_{l}})*G_{\sigma,\operatorname{supp}{k_{l}}}$,
with $\oplus$ denoting the dilation operator, and
$G_{\sigma,\operatorname{supp}{k_{l}}}$ is a Gaussian smoothing window with
the standard deviation $\sigma$ and a window size of
$\operatorname{supp}{k_{l}}$. $\\{\mathcal{R}_{l}\\}_{l=0}^{L-1}$ comes from
the discretization of the depth map, but dilation and smoothing of
$\hat{\mathcal{R}}_{l}$ are used to approximate the mixed blur around the
depth discontinuities, and therefore allows to omit explicit estimation of the
occluded pixels. Specifically, $\mathcal{R}_{l}$ is determined by the scene
depth as

$\mathcal{R}_{l}=\begin{cases}\mathbf{p}\in\Omega|D(\mathbf{p})\geq
D_{0}&,l=0\\\ \mathbf{p}\in\Omega|D_{l-1}<D(\mathbf{p})\leq
D_{l}&,l=1,\dots,L-1\end{cases}\enspace,$ (10)

where $\Omega$ refers to the pixel domain in the latent image $\mathbf{x}$,
and $\\{D_{l}\\}_{l=0}^{L-1}$ is the sequence of depth values that define the
regions with ”uniform” blur. In particular, $D_{0}$ represents the depth limit
value, the depth values from $D_{0}$ to $\infty$, for which pixels seem not to
move at all.

Next, we derive how to compute the depth sequence $\\{D_{l}\\}_{l=0}^{L-1}$
and the respective kernels $k_{l}$, for the known camera trajectory $s$ and
depth map $D(\mathbf{p})$.

###  4.1 Depth-dependent regions

Figure 4: Spatially-varying blur from depth: (a) full depth map of the latent
image $\mathbf{x}$, (b) in-plane camera trajectory, (c) depth sequences in the
$x$ and $y$ axis that delimit the regions with the same amount of blur (see
Eq. 13) and (d) the region indicators from 0 to 32 that denote the amount of
blur from less than 1 to 16 pixels in both dimensions.

The image regions for which the blur behaves in the same way are completely
defined by the depth sequence $\\{D_{l}\\}_{l=0}^{L-1}$. Without loss of
generality, let us consider a one-dimensional camera movement whose maximum
absolute displacement is denoted by $s_{\max}$. As introduced above, we
consider $D_{0}$ the depth limit where pixels do not move, namely those pixels
whose blur extent is less than one pixel (half a pixel for rounding issues, in
practice). The pixels must satisfy

$\frac{\delta}{2}=\frac{s_{\max}F}{D_{0}}\enspace,$ (11)

where $\delta$ denotes the pixel size. This means that $D_{0}=2\kappa$ with
$\kappa=\frac{s_{\max}F}{\delta}$.

For the rest elements of the sequence, we take into account our definition of
blur variation presented in Sec. 3.2. The next element in the sequence is
characterized as the depth that produces a blur variation of $n$ pixels
111$\sigma$ and $n$ correspond to hyper-parameters in our blur formation
model. Ablation studies on those can be found in the supplementary material..
In other words, the blur extent varies $n$ pixels from $\mathcal{R}_{l-1}$ to
$\mathcal{R}_{l}$. This is expressed as:

$\Delta T=\frac{s_{\max}F}{D_{l}}-\frac{s_{\max}F}{D_{l-1}}=n\delta\enspace.$
(12)

Similarly, by reorganizing the terms, we find an equation for the $l$-th
element of the sequence. It can be proven by induction that

$D_{l}=\frac{2\kappa}{2ln+1}\enspace,\hbox{ where
}\kappa=\frac{s_{\max}F}{\delta}\enspace.$ (13)

To extend this methodology to 2D motion, we simply compute the component-wise
sequences $D_{l(x)}$ and $D_{l(y)}$, which are obtained by replacing different
values of $s_{\max}$ and $\delta$ for the $x$ and $y$ components of the
movement. Then, the complete sequence $D_{l}$ is the sorted vector of the set
union $\\{D_{l(x)}\cup D_{l(y)}\\}$. Fig. 4 exemplifies the discrete regions
automatically obtained using the above procedure for a synthetically generated
image, with $n=1$. Fig. 4(a) and (b) show the full depth and the 2D camera
trajectory, respectively. Fig. 4(c) illustrates the depth sequences $D_{l(x)}$
and $D_{l(y)}$ computed by using the aforementioned procedure. Lastly, the set
of regions $\\{\mathcal{R}_{l}\\}_{l=0}^{L-1}$ whose blur behaves similarly
within each layer is shown in Fig. 4(d). It is worth mentioning that the total
number $L$ of regions is completely adaptive to the scene configuration. One
only needs to compute the sequences until $D_{L-1(x)}$ and $D_{L-1(y)}$ cover
the minimum depth.

###  4.2 Blur kernels synthesis

Having the time-dependent in-plane camera trajectory $s(t)$ and the depth map
$D(\mathbf{p})$, the pixel-wise motion blur kernel is given by

$\mathbf{k}(\mathbf{p})=\hat{f}\Big{(}\Big{\lfloor}\frac{-s(t)F}{\delta
D(\mathbf{p})}\Big{\rfloor}\Big{)}\enspace,$ (14)

where $\lfloor\cdot\rfloor$ denotes the rounding operation and $\hat{f}$ is
the operator that computes the EPDF for the discretized values in the
argument. Instead of computing kernels $\mathbf{k}(\mathbf{p})$ at pixel
level, we compute a smaller set $\\{k_{l}\\}_{l=0}^{L-1}$ where every kernel
is paired with a region $\hat{\mathcal{R}}_{l}$. The pixels in the region
$\hat{\mathcal{R}}_{l}$ share the same motion blur kernel $k_{l}$:

$k_{l}=\hat{f}\Big{(}\Big{\lfloor}\frac{-s(t)F}{\delta
D_{l}^{*}}\Big{\rfloor}\Big{)}\enspace,$ (15)

where $D_{l}^{*}$ is the optimal depth value in the range $[D_{l},D_{l-1}]$
that minimizes the mean-square error in $D(\mathbf{p})$ for
$\mathbf{p}\in\mathcal{R}_{l}$. The depths $D(\mathbf{p})$ in
$[D_{l},D_{l-1}]$ follow a random variable $\zeta$ with PDF $f(\zeta)$ and the
mean-square error is determined by

$\int_{D_{l}}^{D_{l-1}}(\zeta-D_{l}^{*})^{2}f(\zeta)d\zeta\enspace.$ (16)

It can be proven that the mean depth $\bar{D}(\mathbf{p})$ in the range
minimizes (16).

##  5 Neural representations from blur

Advances in implicit neural representations demonstrate that MLPs can learn
the high-frequency details in 2D images [31, 28]. In those works, a
coordinate-based MLP $\Phi_{\theta}$ optimizes its parameters $\theta$ to fit
a sharp image, _i.e_. $\Phi_{\theta}:\mathbf{p}\mapsto\mathbf{x}$. We propose
a different approach where $\Phi_{\theta}$ fits the sharp image $\mathbf{x}$
from its corresponding blurred one $\mathbf{y}$, by embedding a blur function
$b:\mathbf{x}\mapsto\mathbf{y}$ defined by either the PWB model (3) or our ICB
model (7). This provides an alternative solution for deblurring from a single
blurred image. Since $b$ is differentiable, we can use gradient-descent
methods to optimize $\theta$ with the following loss:

$\mathcal{L}=\sum_{\mathbf{p}}\left\lVert
b(\Phi_{\theta}(\mathbf{p}))-\mathbf{y}(\mathbf{p})\right\rVert_{2}^{2}+\lambda\left\lVert\nabla_{\mathbf{p}}\Phi_{\theta}(\mathbf{p})\right\rVert_{1}^{1}\enspace,$
(17)

where $\lambda$ is a hyper-parameter that controls the smoothness of the
gradients. This method is similar to the approach presented by Ulyanov  _et
al_. [36], with the exception that we utilize a coordinate-based MLP rather
than a CNN for fitting $\mathbf{x}$. In practice, we use the SIREN
architecture [28] for its ability to fit derivatives robustly.

##  6 Experiments

###  6.1 Evaluation Datasets

#### Synthetic dataset.

We constructed the Virtual Camera Motion Blur (VirtualCMB) dataset, where the
ground-truth latent images and depth maps are rendered from the 3D scene
models. We utilized the Unity engine [33] for rendering 3D scenes in HD
resolution. The dataset was built using five high-quality scenes available in
the unity asset store. The viewpoints were manually selected to represent a
virtual snapshot camera and motion blur for the three studied cases: 1) Macro
Photography, 2) Trucking Photography and 3) Standard Photography. Table 1
summarizes the number of images captured for each case. Macro and Trucking
represent practical settings where depth contributes to blurring (see Sec.
3.2). Standard Photography is the typical setting where all scene objects are
far from the camera and thus depth-agnostic models work well. In all cases,
the camera was moved through pre-defined trajectories. For the Macro and
Standard cases, we randomly selected six trajectories from the Kohler dataset
[13]. For the Trucking Photography cases, six linear trajectories with a
constant speed in the $xy$ plane were generated. The purpose is to mimic
photography from a moving object (e.g., inside a car). To test our method
beyond motion parallax, also camera motions of pan-tilt rotations and full
6-DoF camera motion were recorded. Overall, 983 blurred images with
corresponding latent sharp images and depth maps were rendered.

Table 1: Summary of captured images: i) parallax motion, ii) with pan-tilt
rotations (w/ $xy$ rotations), and iii) 6-DoF.

Scene | Macro | Trucking | Standard
---|---|---|---
i) | ii) | iii) | i) | ii) | iii) | i) | ii) | iii)
VikingVillage | 26 | 28 | 26 | 23 | 22 | 23 | - | - | 30
IndustrialSet | - | - | - | 60 | 60 | 60 | - | - | 30
ModularCity | - | - | - | 60 | 60 | 60 | - | - | 30
ModernStudio | 58 | 55 | 55 | - | - | - | - | - | 30
LoftOffice | 56 | 50 | 51 | - | - | - | - | - | 30
Total: 983 | 140 | 133 | 132 | 143 | 142 | 143 | - | - | 150

#### Real dataset.

For evaluation with real images, we used the iOS app introduced by Chugunov
_et al_. [6] to capture synchronized RGB, LiDAR depth maps, and camera poses.
These videos match with the Macro photography case, where a static object is
recorded by a hand-held smartphone camera. As preprocessing, RGB frames are
down-scaled to the depth map resolution (256x192), blurry frames are obtained
by temporal average, while the sharp and depth correspondences are taken from
the middle point in the camera trajectory. Accordingly, we built the Real
Camera Motion Blur (RealCMB) dataset, comprised of 58 pairs of blurry and
sharp images, as well as depth and camera motion; from which 48 come from our
own recordings and 10 are available in [6].

###  6.2 Model validation

#### Parallax motion blur.

Our ICB model in Sec. 4 was particularly designed for the parallax motion of a
camera. Thus, we first evaluate the model under in-plane camera motion in the
VirtualCMB dataset. For comparison, we considered the PWB model (3). The
standard image quality metrics: PSNR and SSIM, and the perceptual quality
metric LPIPS [45], are used for performance evaluation.

Parallax results are in the ”Parallax” column of Table. 2. In the terms of
PSNR, SSIM, and LPIPS, the proposed ICB model outperforms the baseline PWB
model in both of the main cases: Macro and Trucking, except for the LPIPS in
the Trucking case. Although the difference between SSIM and LPIPS is marginal
in practice. By nature, PWB cannot properly trace the generated blur over the
depth discontinuities where occluded areas become visible during the motion.
Conversely, our ICB model merges blur from different depth layers more
effectively, resulting in a more realistic blur. Fig. 5(a) and (b) illustrates
this finding for the two types of blur, Macro, and Trucking. The error images
reveal that the proposed model is more precise at the object edges.

#### Out-of-plane rotations and 6-DoF.

Non-uniform blur does not only come from motion parallax but also rotations.
Assuming a large focal length as in [38], pan-tilt rotations can be
approximated by $xy$ translations that are non-depth dependent. Thus, we can
compute a global uniform kernel which is added on top of the kernels in Sec.
4.2. In particular, this approximation works well in narrow-lens devices. This
approach was adopted to handle motion camera blur beyond motion parallax,
neglecting the effect of $z$ translation and roll rotation. The results beyond
parallax motion (xy-rotation and 6-DoF) in Table 2 are similar to the parallax
motion experiment. Consequently, the used approximation works well for the
captured images in the VirtualCMB dataset.

Table 2: Blur formation results in VirtualCMB.

| Parallax | w/ $xy$ rotation | 6 DoF
---|---|---|---
Macro | Trucking | Macro | Trucking | Macro | Trucking | Standard
Ours | PWB | Ours | PWB | Ours | PWB | Ours | PWB | Ours | PWB | Ours | PWB | Ours | PWB
$\uparrow$PSNR | 42.48 | 41.54 | 37.42 | 36.59 | 42.16 | 41.11 | 36.99 | 36.21 | 38.84 | 38.37 | 37.08 | 36.29 | 37.38 | 36.97
$\uparrow$SSIM | 0.993 | 0.992 | 0.985 | 0.984 | 0.990 | 0.989 | 0.984 | 0.983 | 0.984 | 0.982 | 0.984 | 0.983 | 0.978 | 0.978
$\downarrow$LPIPS ($\times 10^{-5}$) | 4.639 | 5.049 | 5.870 | 5.238 | 5.399 | 6.316 | 6.158 | 5.676 | 8.883 | 9.744 | 6.088 | 5.613 | 11.17 | 13.01

Table 3: Blur formation results for RealCMB (avg over 58 test images).

|  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS ($\times 10^{-5}$)
---|---|---|---
PWB | 36.31 | 0.984 | 6.826
Ours | 38.21 | 0.990 | 4.484

Table 4: Results in terms of computational resources.

| RealCMB | VirtualCMB
---|---|---
| Memory [MB] | Run time [s] | Memory [MB] | Run time [s]
PWB | 57.39 | 1.95 | 2799 | 13.41
Ours | 1.74 | 2.78 | 48.97 | 7.27

---
(a)
(b)
(c)
(d)
Figure 5: Examples of the blur formation results. (a) Macro, (b) Trucking (c)
Macro with 6-DoF motion, and (d) Real images.

#### Real images.

Surprisingly, the proposed ICB model performs clearly better on real 6-DoF
motion in the RealCMB dataset (Table 3) than in the previous experiment with
synthetic data. These results indicate that 1) parallax motion can be more
dominating in real data than in our simulated cases and 2) our model is robust
to depth and trajectory noise that appears in real data. Moreover, as the
depth measurements are non-linearly quantized in ICB (see Sec. 4.1), there is
no need for high-resolution depth maps. Fig. 5(d) shows a visual example of
the blur generation in the RealCMB dataset.

#### Computational resources.

Table 4 reports the averaged run time and memory size of our Pytorch
implementations of PWB and ICB. It turns out that the proposed ICB model is
slightly slower in the RealCMB dataset but significantly faster in VirtualCMB.
Most importantly, ICB demonstrates considerably greater efficiency in terms of
memory consumption, with reductions of $\times$32 and $\times$50 in the
RealCMB and VirtualCMB datasets, respectively.

###  6.3 Neural representations from blur

Table 5: Comparison of sharp restoration results.

| VirtualCMB | RealCMB
---|---|---
|  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS ($\times 10^{-4}$) |  $\uparrow$PSNR |  $\uparrow$SSIM |  $\downarrow$LPIPS ($\times 10^{-4}$)
SRN [32] | 29.92 | 0.9135 | 7.423 | 28.26 | 0.9146 | 8.492
SIUN [42] | 29.75 | 0.9114 | 7.235 | 28.33 | 0.9139 | 7.231
HINet [4] | 29.86 | 0.9133 | 8.651 | 28.32 | 0.9133 | 7.627
BANet [34] | 29.77 | 0.9099 | 9.007 | 28.34 | 0.9140 | 8.241
MIMO-UNet++ [5] | 28.79 | 0.8964 | 13.12 | 27.92 | 0.9106 | 10.33
MPRNet [44] | 30.01 | 0.9146 | 8.062 | 28.79 | 0.9178 | 7.769
MAXIM [35] | 30.34 | 0.9186 | 7.418 | 28.89 | 0.9207 | 5.695
Restormer [43] | 30.41 | 0.9174 | 7.318 | 29.56 | 0.9243 | 6.887
PWB + SIREN | 27.08 | 0.7975 | 37.25 | 30.61 | 0.9429 | 5.398
Ours + SIREN | 27.14 | 0.8001 | 36.56 | 31.92 | 0.9546 | 4.032

Table 5 summarizes the results of the sharp implicit representations with ICB
and PWB models. In addition, we evaluated SOTA deep-deblurring methods. For
the task of learning implicit representations from a single blurry image, our
ICB model produces superior reconstruction results compared to the PWB model.
Learned sharp representation using ICB does not match the performance of
state-of-the-art deep deblurring methods on VirtualCMB, but it performs
significantly better than others on RealCMB. The variation in performance
between the two datasets can be attributed to the difference in image
resolution. The SIREN architecture utilized in the experiments may be better
suited to handling low-resolution images, such as those in RealCMB. Visual
restoration examples are in Fig. 6. It is observed that the learned
representation roughly restores the edges, but global noise remains in the
VirtualCMB example. On the contrary, an accurate sharp representation is
obtained in the RealCMB case.

---
(a)
(b)
Figure 6: Examples of the deblurring results in (a) VirtualCMB, (b) RealCMB
datasets.

##  7 Conclusion

This work provides analytical and experimental results about the scene
configurations in which the scene depth affects the camera motion blur. In
particular, we identified two types of scenes that appear in consumer
photography: ”Macro” and ”Trucking”. Primarily, we presented an Image-
Compositing Blur (ICB) model that efficiently and accurately describes the
induced blur in those cases. Experimental validation was performed in our
introduced synthetic and real datasets. Interestingly, we demonstrated the
effectiveness of the ICB model to learn sharp neural representations from a
single blurry image. Our findings and the new datasets help to develop better
deblurring approaches.

#### Limitations.

Although our ICB model is derived for parallax motion, the model was found
accurate enough under certain scene configurations, _e.g_., Macro and Trucking
photography. Besides, the model is computationally efficient and robust
against occlusions due to abrupt depth changes. Regarding the deblurring task,
our results are still far from being practical. In real scenarios, the depth
maps and camera trajectories need to be estimated and that would need a
careful study of the suitability of IMU-based odometry and depth sensors in
the current hand-held devices.

## Acknowledgements

This project was supported by a Huawei Technologies Oy (Finland) project. We
also thank Jussi Kalliola for building the iOS app [6] for data collection.

## References

  * [1] Akpinar, U., Sahin, E., Meem, M., Menon, R., Gotchev, A.: Learning wavefront coding for extended depth of field imaging. IEEE Transactions on Image Processing 30, 3307–3320 (2021)
  * [2] Anwar, S., Hayder, Z., Porikli, F.: Depth estimation and blur removal from a single out-of-focus image. In: BMVC. vol. 1, p. 2 (2017)
  * [3] Chan, T.F., Wong, C.K.: Total variation blind deconvolution. IEEE transactions on Image Processing 7(3), 370–375 (1998)
  * [4] Chen, L., Lu, X., Zhang, J., Chu, X., Chen, C.: Hinet: Half instance normalization network for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 182–192 (2021)
  * [5] Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4641–4650 (2021)
  * [6] Chugunov, I., Zhang, Y., Xia, Z., Zhang, X., Chen, J., Heide, F.: The implicit values of a good hand shake: Handheld multi-frame neural depth refinement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2852–2862 (2022)
  * [7] Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing camera shake from a single photograph. In: ACM SIGGRAPH 2006 Papers. pp. 787–794 (2006)
  * [8] Gupta, A., Joshi, N., Zitnick, C.L., Cohen, M., Curless, B.: Single image deblurring using motion density functions. In: European conference on computer vision. pp. 171–184. Springer (2010)
  * [9] Hasinoff, S.W., Kutulakos, K.N.: A layer-based restoration framework for variable-aperture photography. In: 2007 IEEE 11th International Conference on Computer Vision. pp. 1–8. IEEE (2007)
  * [10] Hirsch, M., Schuler, C.J., Harmeling, S., Schölkopf, B.: Fast removal of non-uniform camera shake. In: 2011 International Conference on Computer Vision. pp. 463–470. IEEE (2011)
  * [11] Hu, Z., Xu, L., Yang, M.H.: Joint depth estimation and camera shake removal from single blurry image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2893–2900 (2014)
  * [12] Ikoma, H., Nguyen, C.M., Metzler, C.A., Peng, Y., Wetzstein, G.: Depth from defocus with learned optics for imaging and occlusion-aware depth estimation. In: 2021 IEEE International Conference on Computational Photography (ICCP). pp. 1–12. IEEE (2021)
  * [13] Köhler, R., Hirsch, M., Mohler, B., Schölkopf, B., Harmeling, S.: Recording and playback of camera shake: Benchmarking blind deconvolution with a real-world database. In: European conference on computer vision. pp. 27–40. Springer (2012)
  * [14] Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-laplacian priors. Advances in neural information processing systems 22, 1033–1041 (2009)
  * [15] Krishnan, D., Tay, T., Fergus, R.: Blind deconvolution using a normalized sparsity measure. In: CVPR 2011. pp. 233–240. IEEE (2011)
  * [16] Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., Matas, J.: Deblurgan: Blind motion deblurring using conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8183–8192 (2018)
  * [17] Kupyn, O., Martyniuk, T., Wu, J., Wang, Z.: Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8878–8887 (2019)
  * [18] Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Understanding and evaluating blind deconvolution algorithms. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 1964–1971. IEEE (2009)
  * [19] Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Efficient marginal likelihood optimization in blind deconvolution. In: CVPR 2011. pp. 2657–2664. IEEE (2011)
  * [20] Li, L., Pan, J., Lai, W.S., Gao, C., Sang, N., Yang, M.H.: Blind image deblurring via deep discriminative priors. International journal of computer vision 127(8), 1025–1043 (2019)
  * [21] Li, L., Pan, J., Lai, W.S., Gao, C., Sang, N., Yang, M.H.: Dynamic scene deblurring by depth guided model. IEEE Transactions on Image Processing 29, 5273–5288 (2020)
  * [22] Mustaniemi, J., Kannala, J., Särkkä, S., Matas, J., Heikkilä, J.: Gyroscope-aided motion deblurring with deep networks. In: IEEE Winter Conference on Applications of Computer Vision (WACV) (January 2019)
  * [23] Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3883–3891 (2017)
  * [24] Pan, J., Sun, D., Pfister, H., Yang, M.H.: Blind image deblurring using dark channel prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1628–1636 (2016)
  * [25] Pan, L., Dai, Y., Liu, M.: Single image deblurring and camera motion estimation with depth map. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 2116–2125. IEEE (2019)
  * [26] Park, H., Mu Lee, K.: Joint estimation of camera pose, depth, deblurring, and super-resolution from a blurred image sequence. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 4613–4621 (2017)
  * [27] Sheng, B., Li, P., Fang, X., Tan, P., Wu, E.: Depth-aware motion deblurring using loopy belief propagation. IEEE Transactions on Circuits and Systems for Video Technology 30(4), 955–969 (2019)
  * [28] Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems 33, 7462–7473 (2020)
  * [29] Su, S., Delbracio, M., Wang, J., Sapiro, G., Heidrich, W., Wang, O.: Deep video deblurring for hand-held cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1279–1288 (2017)
  * [30] Tai, Y.W., Tan, P., Brown, M.S.: Richardson-lucy deblurring for scenes under a projective motion path. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(8), 1603–1618 (2010)
  * [31] Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., Ng, R.: Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems 33, 7537–7547 (2020)
  * [32] Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8174–8182 (2018)
  * [33] Technologies, U.: Unity real-time development platform, https://unity.com/
  * [34] Tsai, F.J., Peng, Y.T., Tsai, C.C., Lin, Y.Y., Lin, C.W.: Banet: A blur-aware attention network for dynamic scene deblurring. IEEE Transactions on Image Processing 31, 6789–6799 (2022)
  * [35] Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., Li, Y.: Maxim: Multi-axis mlp for image processing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5769–5780 (2022)
  * [36] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 9446–9454 (2018)
  * [37] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [38] Whyte, O., Sivic, J., Zisserman, A., Ponce, J.: Non-uniform deblurring for shaken images. International journal of computer vision 98(2), 168–186 (2012)
  * [39] Xu, L., Jia, J.: Two-phase kernel estimation for robust motion deblurring. In: European conference on computer vision. pp. 157–170. Springer (2010)
  * [40] Xu, L., Jia, J.: Depth-aware motion deblurring. In: 2012 IEEE International Conference on Computational Photography (ICCP). pp. 1–8. IEEE (2012)
  * [41] Xu, L., Zheng, S., Jia, J.: Unnatural l0 sparse representation for natural image deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1107–1114 (2013)
  * [42] Ye, M., Lyu, D., Chen, G.: Scale-iterative upscaling network for image deblurring. IEEE Access 8, 18316–18325 (2020)
  * [43] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5728–5739 (2022)
  * [44] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14821–14831 (2021)
  * [45] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586–595 (2018)
  * [46] Zhang, X., Wang, R., Jiang, X., Wang, W., Gao, W.: Spatially variant defocus blur map estimation and deblurring from a single image. Journal of Visual Communication and Image Representation 35, 257–264 (2016)
  * [47] Zhou, S., Zhang, J., Zuo, W., Xie, H., Pan, J., Ren, J.S.: Davanet: Stereo deblurring with view aggregation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10996–11005 (2019)

##  Appendix 0.A Image Compositing Blur (ICB) model details

###  0.A.1 $D_{l}$ sequence

In the main paper, we introduced the $D_{l}$ sequence that is used to
determine the depth-dependent layers with similar blur behavior. Here, we
present a detailed derivation.

Eq. 12 can be reorganized to obtain a recursive expression for $D_{l}$:

$D_{l}=\frac{\kappa D_{l-1}}{nD_{l-1}+\kappa},\hbox{ where
}\kappa=\frac{s_{\max}F}{\delta}\enspace.$ (0.A.1.1)

###### Proposition 1

Given $D_{l}$ as stated in Eq. 0.A.1.1, and $D_{0}=2\kappa$, the explicit
expression for $D_{l}$ is:

$D_{l}=\frac{2\kappa}{2ln+1}$

###### Proof

For $l=1$, we have:

$D_{1}=\frac{\kappa
D_{0}}{nD_{0}+\kappa}=\frac{2\kappa^{2}}{2n\kappa+\kappa}=\frac{2\kappa}{2n+1}$

It is then clear that the explicit expression holds for $l=1$.

Assuming that the statement holds for some $l^{\prime}>0$, _i.e_.
$D_{l^{\prime}}=\frac{2\kappa}{2l^{\prime}n+1}$, we must show that:

$D_{l^{\prime}+1}=\frac{2\kappa}{2(l^{\prime}+1)n+1}$

Using Eq. 0.A.1.1:

$D_{l^{\prime}+1}=\frac{\kappa
D_{l^{\prime}}}{nD_{l^{\prime}}+\kappa}=\frac{\kappa\big{(}\frac{2\kappa}{2l^{\prime}n+1}\big{)}}{n\big{(}\frac{2\kappa}{2l^{\prime}n+1}\big{)}+\kappa}=\frac{2\kappa}{2n+2nl^{\prime}+1}=\frac{2\kappa}{2(l^{\prime}+1)n+1}$

###  0.A.2 Ablation

Our ICB model for parallax motion includes two hyper-parameters: 1) $n$
controls the level of discretization in the depth map, the smaller the values
for $n$ the more depth layers are obtained; and 2) $\sigma$ has an effect on
the smoothness of each alpha matte $\mathcal{A}_{l}$.

Table 0.A.2.1 reports the ablation results for 12 images within the Macro and
Trucking scenes in the VirtualCMB dataset. We can observe that $\sigma$ does
not play a significant role in the performance. In contrast, $n$ balances
accuracy with run time. Smaller $n$ leads to higher accuracy but in slower run
time. In consequence, we set $n=1$ and $\sigma=4.0$ to get higher quality
performance throughout our experimentation.

Table 0.A.2.1: Ablation results.

|  | Macro | Trucking
---|---|---|---
$\sigma$ | $n$ | PSNR | SSIM | LPIPS ($\times 10^{-5}$) | Time [s] | PSNR | SSIM | LPIPS ($\times 10^{-5}$) | Time [s]
0.5 | 1.0 | 40.799 | 0.99142 | 3.4104 | 3.3310 | 36.905 | 0.98416 | 4.7988 | 2.9216
| 2.0 | 40.626 | 0.99097 | 3.2748 | 2.6664 | 36.725 | 0.98337 | 4.8637 | 2.3630
| 3.0 | 40.393 | 0.99053 | 3.3637 | 2.4532 | 36.333 | 0.98190 | 6.6962 | 2.1599
1.0 | 1.0 | 40.868 | 0.99151 | 3.3610 | 3.4100 | 36.971 | 0.98429 | 4.7715 | 2.8925
| 2.0 | 40.691 | 0.99106 | 3.2451 | 2.6764 | 36.786 | 0.98350 | 4.8065 | 2.3663
| 3.0 | 40.463 | 0.99062 | 3.3218 | 2.7093 | 36.381 | 0.98203 | 6.6180 | 2.2219
1.5 | 1.0 | 40.892 | 0.99154 | 3.3590 | 4.0521 | 37.009 | 0.98438 | 4.7872 | 2.8227
| 2.0 | 40.712 | 0.99110 | 3.2493 | 2.5828 | 36.824 | 0.98359 | 4.7764 | 2.3238
| 3.0 | 40.492 | 0.99067 | 3.3150 | 2.4378 | 36.409 | 0.98211 | 6.6034 | 2.1928
2.0 | 1.0 | 40.901 | 0.99156 | 3.3733 | 3.2586 | 37.037 | 0.98444 | 4.8103 | 2.8838
| 2.0 | 40.720 | 0.99112 | 3.2624 | 2.9163 | 36.854 | 0.98367 | 4.7623 | 2.4189
| 3.0 | 40.507 | 0.99070 | 3.3235 | 2.5660 | 36.433 | 0.98218 | 6.6126 | 2.5661
2.5 | 1.0 | 40.903 | 0.99157 | 3.3926 | 3.8899 | 37.062 | 0.98450 | 4.8365 | 2.8771
| 2.0 | 40.723 | 0.99114 | 3.2786 | 2.6463 | 36.881 | 0.98373 | 4.7593 | 2.4833
| 3.0 | 40.515 | 0.99072 | 3.3375 | 2.3812 | 3.6456 | 0.98225 | 6.6283 | 2.2431
3.0 | 1.0 | 40.902 | 0.99157 | 3.4123 | 3.4124 | 37.085 | 0.98455 | 4.8687 | 2.8062
| 2.0 | 40.724 | 0.99115 | 3.2951 | 2.5933 | 36.907 | 0.98378 | 4.7640 | 2.3603
| 3.0 | 40.520 | 0.99073 | 3.3526 | 2.2284 | 36.477 | 0.98231 | 6.6487 | 2.0337
3.5 | 1.0 | 40.900 | 0.99158 | 3.4297 | 3.7939 | 37.105 | 0.98459 | 4.9037 | 2.8090
| 2.0 | 40.722 | 0.99116 | 3.3101 | 2.6797 | 36.929 | 0.98382 | 4.7729 | 2.3857
| 3.0 | 40.523 | 0.99074 | 3.3669 | 2.4121 | 36.495 | 0.98235 | 6.6714 | 2.2891
4.0 | 1.0 | 40.896 | 0.99158 | 3.4447 | 3.3234 | 37.122 | 0.98462 | 4.9384 | 2.8267
| 2.0 | 40.720 | 0.99116 | 3.3233 | 2.8096 | 36.946 | 0.98386 | 4.7852 | 2.4440
| 3.0 | 40.524 | 0.99075 | 3.3796 | 2.4356 | 36.509 | 0.98239 | 6.6945 | 2.2058

##  Appendix 0.B Details for optimizing neural representations from blur

The SIREN architecture [28] used for the neural representation consists of 4
hidden layers, each one with 192 nodes, and one out-most linear layer. We use
Adam optimizer with 400 iterations. The learning rate (lr) and the gradient
weight $\lambda$ are set to $5\times 10^{-4}$ and $8\times 10^{-6}$,
respectively. Likewise, we employ gradient clipping and the cosine annealing
scheduler with minimum lr of $5\times 10^{-6}$.

##  Appendix 0.C Additional results

More visual comparisons for our blur formation model are shown in Fig.
0.C.0.1, and Fig. 0.C.0.2. Additional deblurring results are illustrated in
Fig. 0.C.0.3 and Fig. 0.C.0.4.

---
Figure 0.C.0.1: Visual results of blur formation in VirtualCMB: Macro,
Trucking, and Standard scenes.
---
Figure 0.C.0.2: Visual results of blur formation in RealCMB.

---
Figure 0.C.0.3: Examples of sharp restoration in VirtualCMB.
---
Figure 0.C.0.4: Examples of sharp restoration in RealCMB.


# Clinical-Longformer and Clinical-BigBird: Transformers for long clinical
sequences

Yikuan Li, Ramsey M. Wehbe, Faraz S. Ahmad, Hanyin Wang, Yuan Luo
Feinberg School of Medicine
Northwestern University
{yikuan.li, ramsey.wehbe, faraz.ahmad, hanyin.wang<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Transformers-based models, such as BERT, have dramatically improved
performance for various natural language processing tasks. The clinical
knowledge enriched model, namely ClinicalBERT, also achieved state-of-the-art
results when performed on clinical named entity recognition and natural
language inference tasks. One of the core limitations of these transformers is
the substantial memory consumption due to their full self-attention mechanism.
To overcome this, long sequence transformer models, e.g. Longformer and
BigBird, were proposed with the idea of sparse attention mechanism to reduce
the memory usage from quadratic to the sequence length to a linear scale.
These models extended the maximum input sequence length from 512 to 4096,
which enhanced the ability of modeling long-term dependency and consequently
achieved optimal results in a variety of tasks. Inspired by the success of
these long sequence transformer models, we introduce two domain enriched
language models, namely Clinical-Longformer and Clinical-BigBird, which are
pre-trained from large-scale clinical corpora. We evaluate both pre-trained
models using 10 baseline tasks including named entity recognition, question
answering, and document classification tasks. The results demonstrate that
Clinical-Longformer and Clinical-BigBird consistently and significantly
outperform ClinicalBERT as well as other short-sequence transformers in all
downstream tasks. We have made our source code available at
https://github.com/luoyuanlab/Clinical-Longformer the pre-trained models
available for public download at: https://huggingface.co/yikuan8/Clinical-
Longformer.

##  1 Introduction

Transformer-based models have been wildly successful in setting state-of-the-
art results to a broad range of natural language processing (NLP) tasks,
including question answering, document classification, machine translation,
text summarization and others [1, 2, 3]. The successes are also replicated in
the clinical and biomedical domain via pre-training language models using
clinical or biomedical corpora and then transfer them to a variety of clinical
or biomedical downstream tasks, e.g. computational phenotyping [4, 5],
automatically ICD coding [6], knowledge graph completion [7] and clinical
question answering [8].

The self-attention mechanism [9] is one of the most important components that
lead to the successes of transformer-based models, which allows each token in
the input sequence to independently interact with every other token in the
sequence in parallel. However, the memory consumption of self-attention
mechanism quadratically enlarges with sequence length, resulting in
impracticable training time and easily reaching the memory limits of GPUs.
Consequently, transformer-based models that leverage a full self-attention
mechanism, such as BERT and RoBERTa, always have an input sequence length
limit of 512 tokens. To deal with this limit when modeling long texts using
transformer-based models, the input sequence shall be either truncated to the
first 512 tokens, or segmented to multiple small chunks of 512 tokens. If the
latter method is applied to a text classification task, an aggregation
operation will be added to yield the final output. Both segmentation and
truncation of the input sequence will ignore the long-term dependencies
spanning over 512 tokens and consequently achieve suboptimal results due to
information loss. This drawback of self-attention operation will impact
language model pre-training and then be amplified to downstream tasks. In
clinical NLP, the models that are applied with the transformer-based
approaches also encounter this limitation [10]. For example, the discharge
summaries in MIMIC-III, which are always used to predict hospital re-admission
[11] or mortality [12], have 1,435 words in average, far exceeding the 512
tokens limits of BERT like models. Recently, researchers have been developing
some novel variants of transformers particularly for long sequences that
reduce the memory usage from quadratic to linear to the sequence length [13,
14, 15]. The core idea behind these models is to replace the full attention
mechanism with the sparse attention mechanism, which is a blend of sliding
windows and reduced global attention. These models are able to be fed in up to
4.096 tokens and boost the performance of several question answering as well
as text summarization tasks. However, to the best of our knowledge, these
transformers for long sequences are rarely discussed in the clinical and
biomedical domain. Therefore, we would like to examine the adaptability of
these long sequence models using a series of clinical NLP tasks. In
particular, we make the following contributions:

  1. 1.

We pre-train and publicly release two transformer models for long clinical
sequences, namely Clinical-Longformer and Clinical-BigBird, using large-scale
clinical notes.

  2. 2.

We demonstrate that both Clinical-Longformer and Clinical-BigBird
significantly improve the performance of a variety of downstream clinical NLP
tasks including question answering, named entity recognition and document
classification.

##  2 Related Work

###  2.1 Clinical and Biomedical Transformers

Transformer-based models, especially BERT [1], can be enriched with clinical
and biomedical knowledge through pre-training on large-scale clinical and
biomedical corpora. These domain-enriched models, e.g. BioBERT [16] pre-
trained on biomedical publications and ClinicalBERT [17] pre-trained on
clinical narratives, set the state-of-the-arts when down-stream applied to
clinical and biomedical NLP tasks. Inspired by the success of these domain-
enriched models, more pre-trained models were released to boost the
performance of NLP models when applied to specific clinical scenarios. For
example, Smit et al. [18] proposed CheXbert to annotate thoracic disease
findings from radiology reports, which outperformed previous rule-based
labelers with statistical significance. The model was pre-trained using the
combination of human-annotated and machine-annotated radiology reports. He et
al. [19] introduced DiseaseBERT, which infused disease knowledge to the BERT
model by pre-training on a series of disease description passages that were
constructed from Wikipedia and MeSH terms. DiseaseBERT received superior
results on consumer health question answering tasks. Zhou et al. [20]
developed CancerBERT to extract breast cancer related concepts from clinical
notes and pathology reports, which would facilitate the information extraction
process. However, all of the aforementioned models were built on the basic
BERT architecture, which has a limitation of 512 tokens in the input sequence
length. This limitation may result in the information loss of long-term
dependencies in the training processes.

###  2.2 Transformers for long sequences

Various attention mechanisms have been proposed to handle the large memory
consumption of the attention operations in vanilla Transformer architecture.
Transformer-XL [21] segmented a long sequence into multiple small chunks and
then learned long-term dependency with a left-to-right segment-level
recurrence mechanism. Transformer-XL learns 5.5× dependency compared to the
vanilla transformer models, but loses the advantage of bidirectional
representation of BERT like models. In another study, Reformer [22] applied
two techniques to reduce the space complexity of transformer architecture by
replacing do-product attention operation with locality-sensitive hashing and
sharing the activation function among layers. Reformer was able to process
longer sequences at a faster speed and be more memory-efficient. However, no
performance improvement can be observed when using Reformer. Almost
simultaneously, Longformer [13] and BigBird [14] were proposed to drastically
alleviate the memory consumption of transformer models from quadratic growth
to linear growth to the sequence length, by replacing the pairwise full
attention mechanisms with a combination of sliding window attention and global
attention mechanisms. They are slightly different in terms of implementation
and configuration of the global attention mechanism. Both models support input
sequences up to 4.096 tokens long (8× of BERT) and significantly improve
performance on long-text question answering and summarization tasks. However,
the adaptability of these long sequence transformers to the clinical and
biomedical fields, where the text length always exceeds the limits of BERT
like models, has been rarely investigated. Levine et al. [23] pre-trained
longformer models using clinical notes and then fine-tuned the language model
on document classification tasks include ICD coding. Our study applied similar
approach, while aiming at a more comprehensive comparison to the short
sequence models using heterogeneous NLP tasks.

##  3 Material and Methods

In this section, we first introduce the clinical data set we use as the pre-
training corpus, followed by the pre-training process of Clinical-Longformer
and BigBird. Then, we enumerate the down-stream tasks we used to fine-tune our
pre-trained model. We also provide the technical details of pre-training and
fine-tuning for reproducing purposes. We have made our source code available
at https://github.com/luoyuanlab/Clinical-Longformer the pre-trained models
available for public download at: https://huggingface.co/yikuan8/Clinical-
Longformer.

###  3.1 Datasets

Similarly to [17, 11], we leveraged approximately 2 million clinical notes
extracted from the MIMIC-III [24] dataset, which is the largest publicly
available EHR dataset that contains clinical narratives of over 40,000
patients admitted to the intensive care units. We only applied minimal pre-
processing steps, including 1) to remove all de-identification placeholders
that were generated to protect the PHI (protected health information); 2) to
replace all characters other than alphanumericals and punctuation marks; 3) to
convert all alphabetical characters to lower cases; and 4) to strip extra
white spaces.

###  3.2 Pre-training

Longformer [13] and BigBird [14] are the two most remarkable transformer
models that are designed for long input sequences. Both models extend the
maximum input sequence length to 4,096 tokens, which is 8× of the conventional
transformer-based models by introducing the localized sliding window and
global attention mechanisms to reduce the computational expenses of full self-
attention mechanisms. The differences between two models are how the global
attention is realized and the selection of loss function in fine-tuning.
BigBird also contains some random localized attention operations. The
performance difference between two models on downstream tasks is minimal.
Therefore, we would like to pre-train both models and compare their
performance on clinical NLP tasks. We refer readers to the original papers of
Longformer [13] and BigBird [14] for more technical details.

Byte-level BPE (Byte-Pair-Encoding) [25] was applied to tokenize the clinical
corpus. We initialized Clinical-Longformer and Clinical-BigBird from the pre-
trained weights of the base version of Longformer and the ITC version of
BigBird, respectively (although the ETC version performs better, HuggingFace,
the largest shared transformers community only provides the implementation and
the pre-trained weights of ITC version). Both models were distributed in
parallel to 6 32GB Tesla V100 GPUs. FP16 precision was enabled to accelerate
training. We pre-trained Clinical-Longformer for 200,000 steps with batch size
of 6×3, and Clinical-BigBird for 300,000 steps with batch size of 6×2. The
learning rates were 3e-5 for both models. The entire pre-training process took
more than 2 weeks for each model.

###  3.3 Tasks

In this study, we adapted the pre-trained Clinical-Longformer and Clinical-
BigBird to 9 clinical NLP datasets. These 9 NLP datasets broadly cover various
NLP tasks including extractive question answering, named entity recognition,
natural language inference, and document classification. We relied on these
NLP tasks to validate the performance improvement of our proposed models. The
statistics and descriptions of all dataset can be found in Table 1.

Table 1: Description and statistics of downstream NLP tasks

Dataset | Task | Data Source | Domain | Sample Size | Avg. Seq. Length | Max Seq. Length
---|---|---|---|---|---|---
MedNLI | Inference | MIMIC | Clinical | 14,049 | 38.9 | 409
i2b2 2006 | NER | i2b2 | Clinical | 66,034 | 867 | 3,986
i2b2 2010 | NER | i2b2 | Clinical | 43,947 | 1,459.3 | 6,052
i2b2 2012 | NER | i2b2 | Clinical | 13,108 | 793.6 | 2,900
I2b2 2014 | NER | i2b2 | Clinical | 83,466 | 5,133.5 | 14,370
emrQA-Medication | QA | i2b2 | Clinical | 255,908 | 1,880.3 | 6,109
emrQA-Relation | QA | i2b2 | Clinical | 141,243 | 1,460.3 | 6,050
emrQA-HeartDisease | QA | i2b2 | Clinical | 30,731 | 5,292.7 | 14,060
OHSUMed | Multiclass Classif. | MEDLINE | Biomedical | 4,056 | 301.1 | 861
OpenI | Multilabel Classif. | IndianaU | Clinical | 3,684 | 69.7 | 294
MIMIC-AKI | Binary Classif. | MIMIC | Clinical | 16,536 | 1463.1 | 20,857

####  3.3.1 Question Answering

Question answering (QA) is an NLP discipline, which targets to automatically
answer questions raised from human languages [26]. In the clinical context, QA
system answers physicians’ questions by understanding the clinical narratives
extracted from electronic health record systems to support decision making.
emrQA [27] is the most frequently used benchmark dataset in clinical QA, which
contains more than 400,000 question-answer pairs semi-automatically generated
from past i2b2 challenges. emrQA falls into the category of extractive
question answering that aims at identifying the answer spans from reference
contexts instead of generating answers in a word-by-word fashion. Researchers
have attempted to solve emrQA tasks by using word embedding models [28],
conditional random fields (CRFs) [29] and transformer-based models [30], among
which transformer-based models defeated their competitors in terms of
performance. In our experiments, we investigated the performance of our pre-
trained models using the three largest emrQA subsets: Medication, Relation and
Heart Disease. We evaluated QA performance with two commonly used metrics:
exact match (EM) and F1-score. Exact match represents that the predicted spans
and categories exactly and strictly match with the ground-truth annotations.
F1-score is a looser metric derived from token-level precision and recall,
which aims at measuring the overlap between the predictions and the targets.
We generated train-dev-test splits by following the instruction of [28], where
the training set of relation and medication subsets were randomly under-
sampled to reduce training time. Based on their experience, the performance
would not be compromised after under-sampling.

####  3.3.2 Named Entity Recognition

Named-entity recognition is a token-level classification task that seeks to
identify the named entities and classify them to predefined categories. This
genre of NLP tasks has broad applications in the clinical and biomedical
domain. e.g. de-identification of PHI, and medical concept extraction from
clinical notes. Prior studies have shown that Transformer-based models [16]
significantly outperformed the models built on pre-trained static word
embedding [31] or LSTM networks [32]. We examined our pre-trained models using
four data challenges of: 1) i2b2 2006 [33], to de-identify PHI from medical
discharge notes; 2) i2b2 2010 [34], to extract and annotate medical concepts
from patient reports; 3) i2b2 2012 [35], to identify both clinical concepts
and events relevant to the patient’s clinical timeline from discharge
summaries; and 4) i2b2 2014 [36], to identify PHI information from
longitudinal clinical narratives. We followed the processing steps of[17],
which converted the raw data from all four tasks to IOB
(inside–outside–beginning) tagging format proposed by Ramshaw et al, and then
created train-dev-test splits. [37]. Based on the new tagging format, we
evaluated the model performance with F1-score similarly to QA tasks.

####  3.3.3 Document Classification

Document classification is one of the most common NLP tasks, where we assign a
sentence or document to one or more classes or categories. In the clinical
domain, researchers pay more attention to identifying the diagnosis, or to
predicting the onset of a certain disease from clinical notes. We applied the
following four document classification data set to evaluate the pre-trained
models from different perspectives.

MIMIC-AKI [38, 39] MIMIC-AKI is a binary classification task, where we would
like to predict the possibility of AKI (acute kidney injury) for critically
ill patients using the clinical notes within the first 24 hours following
intensive care unit (ICU) admission. We evaluated the model performance using
AUC and F1 score.

OpenI [40] OpenI is a publicly available chest x-ray (CXR) dataset collected
by Indiana University. The dataset provides around 4,000 radiology reports and
their associated human annotated MeSH terms. Given the small sample size, we
only used openI as the testing set. The pre-trained language models were fine-
tuned using MIMIC-CXR [41], another publicly available chest x-ray dataset.
Unlike openI, the ground-truth labels for MIMIC-CXR were automatically
generated using NLP approaches. The annotated findings overlap between two CXR
data sources are the diagnosis of: Cardiomegaly, Edema, Consolidation,
Pneumonia, Atelectasis, Pneumothorax and Pleural Effusion. In experiments, we
would like to retrieve the presences of all possible thoracic findings from
CXR reports. This task can be considered as a multi-label classification task.
We would report the sample number weighted average of AUC as proposed in [42].

MedNLI [43] Natural language inference (NLI) is the task of determining the
relationship between sentence pairs. MedNLI is a collection of sentence pairs
extracted from MIMIC-III [24] and annotated by two board-certified
radiologists. The relationship between the premise sentence and the hypothesis
sentence could be entailment, contradiction or neutral. Transformer-based
models process NLI tasks also as document classification by merging the
sentence pair together and placing a delimiter token right after the end of
the first sentence. We followed the original splits as [43] and used accuracy
to evaluate the performance.

OHSUMed [44] OHSUMed is a collection of journal titles and/or abstracts
extracted from MEDLINE, which is maintained and published by the National
Library of Medicine. Unlike other data sources aforementioned, this dataset is
the only one that the free-text raw data is not obtained from clinical notes
but academic journals. Following [45], only those cardiovascular diseases
abstracts that are associated with one and only one of the 23 disease
categories are included in this study. We also used accuracy to evaluate this
multi-class classification task.

###  3.4 Baseline Models

Both Clinical-Longformer and Clinical-BigBird were compared with the
pioneering BERT, ClinicalBERT and BioBERT models. BERT [1] model is the first-
of-its-kind transformer architecture that achieved state-of-the-art results in
eleven NLP tasks. Both masked language modeling and next sentence prediction
were used to learn contextualized word representation from BooksCorpus and
English Wikipedia in the pre-training stage. BioBERT [16] is the first
biomedical domain specific BERT variant that was pre-trained from PubMed
abstracts and PMC full-text articles. The weights of BioBERT were initialized
from BERT. BioBERT yielded optimal performance in biomedical QA, NER and
relation extraction tasks. ClinicalBERT [17], initialized from BioBERT, was
further pre-trained using clinical notes that were also extracted from MIMIC-
III. ClinicalBERT boosted the performance for MedNLI and four i2b2 NER tasks
that were also included in our study. Both BioBERT and ClinicalBERT discarded
the next sentence prediction task and only selected the masked language
modeling as the pre-training strategy.

###  3.5 Experimental setup

For the token-level classification, including QA and NER, a classification
head was added to the output of each token obtained from the transformer-based
architecture. The maximum sequence length was selected to be 3,072 for
Clinical-Longformer and Clinical-BigBird, and 384 for all the other three
baseline models. For the sequence-level classification, the predicted outcomes
were yielded from the [CLS] token added to the beginning of each sentence or
document. Given that some clinical notes are extremely long, which may even
exceed the length limits of all models, we first truncated each
document/sentence/sentence-pair to 4096 tokens, which met the length limits of
Clinical-Longformer and Clinical-BigBird. The document/sentence/sentence-pair
was further segmented to snippets of 512 tokens in order to accommodate for
the length requirement of BERT/BioBERT/ClinicalBERT. A pooling layer was
applied to aggregate the output from short snippets. We conducted our
experiments using four 32GB gpus. The batch size during training was 4×4 for
Clinical-Longformer, 4×3 for Clinical-BigBird, and 4×16 for all other models.
Batch sizes were doubled when evaluating or testing. Half precision was
applied to both Clinical-Longformer and Clinical-BigBird. The only hyper-
parameter we tuned was learning rate. We tried 1e-5, 2e-5 and 5e-5 for the
experiments of each model on each task. We fine-tuned 6 epochs for each set-
up. The best-performing model parameters were determined by the performance on
the developing split. The experiments were implemented with python 3.8.0,
pytorch 1.9.0 and Transformer 4.9.0.

##  4 Results and Discussion

Full results for QA, NER and classification tasks are presented in Table 2, 3,
4, respectively. In question answering tasks, both Clinical-Longformer and
Clinical-BigBird outperformed the pioneering short-sequence transformer models
2 percents across all three emrQA subsets when evaluated by F1 score. When
considering the stricter EM metric, Clinical-Longformer and Clinical-BigBird
improved 5 percent on the relations subset, but yielded similar results to
ClinicalBERT in the other two subsets. In NER tasks, Clinical-Longformer
consistently led the short-text transformers by more than 2 percents in all 4
i2b2 dataset. Clinical-BigBird also performed better than ClinicalBERT and
BioBERT in every single NER experiments. In document classification tasks, our
two long-sequence transformers achieved better results compared to prior
models on OpenI, MIMIC-AKI and medNLI, which were all generated from clinical
data sources. BioBERT performed slightly better than our models in OHSUMed
dataset. We argue that OHSUMed is the only task that falls under the scope of
biomedical data mining, making it align better with BioBERT which was pre-
trained from biomedical publications.

We observed that Clinical-Longformer and Clinical-BigBird not only improved
the performance of long sequences tasks, but also short sequences. The maximum
sequences of MedNLI and OpenI are smaller than 512 tokens, which can be
entirely fed into the BERT like models. However, the long sequences models
still achieved better results. We attributed these improvements to the pre-
training stages of Clinical-Longformer and Clinical-BigBird, where the
language models can learn more long-term dependencies by extending the limit
of sequence length. In addition, we found that more significant improvement
can be achieved when applying our two models to the dataset with longer
sequences. For example, the performance improvement on i2b2 2014, which has
the largest averaged sequence length in all 4 NER tasks, is almost twice of
the other three subsets. Likewise, Clinical-Longformer more strongly improved
the F1 score of the heart disease subset from emrQA. This helps us confirm
that Clinical-Longformer and Clinical-BigBird are also able to model long-term
dependency in the downstream tasks. We also noticed that Clinical-Longformer
consistently yielded better results when compared to Clinical-BigBird. Given
that Clinical-BigBird also requires more fine-tuning time, we recommend
readers to apply our Clinical-Longformer checkpoint to their own tasks when
they have limited resources.

Our study has several limitations, each of which can lead to further
investigations. First, we only applied Longformer and BigBird to large scale
clinical corpora. In the next step, we plan to release more pre-trained models
for long sequences enriched with other biomedical corpora, e.g. PubMed and PMC
publications. Second, another approach developed to address the memory problem
of long sequences is to simplify or compress the transformer architecture. In
our future work, we will compare this genre of transformers, e.g. TinyBERT to
our current long sequence models. Last, we did not integrate Clinical-
Longformer or Clinical-BigBird into an encoder-decoder framework due to the
memory limits of our GPU cards. Therefore, the experiment on text generation
or summarization is not included in this study. We will definitely incorporate
those tasks into our future study, once we acquire more computational
resources.

Table 2: The performance of transformer-based pre-trained models on question
answering tasks.

Pre-trained Models | emrQA - Medication | emrQA - Relation | emrQA- Heart Disease
---|---|---|---
metrics | EM | F1 | EM | F1 | EM | F1
BERT | 0.240 | 0.675 | 0.833 | 0.924 | 0.650 | 0.698
BioBERT | 0.247 | 0.700 | 0.836 | 0.926 | 0.647 | 0.702
ClinicalBERT | 0.297 | 0.698 | 0.849 | 0.929 | 0.666 | 0.711
Clinical-Longformer | 0.302 | 0.716 | 0.911 | 0.948 | 0.698 | 0.734
Clinical-BigBird | 0.300 | 0.715 | 0.898 | 0.944 | 0.664 | 0.711

Table 3: The performance of transformer-based pre-trained models on named
entity recognition tasks.

Pre-trained Models | i2b2 2006 | i2b2 2010 | i2b2 2012 | i2b2 2014
---|---|---|---|---
metrics | F1 | F1 | F1 | F1
BERT | 0.939 | 0.835 | 0.759 | 0.928
BioBERT | 0.948 | 0.865 | 0.789 | 0.930
ClinicalBERT | 0.951 | 0.861 | 0.773 | 0.929
Clinical-Longformer | 0.974 | 0.887 | 0.800 | 0.961
Clinical-Big Bird | 0.967 | 0.872 | 0.787 | 0.952

Table 4: The performance of transformer-based pre-trained models on document
classification tasks.

Pre-trained Models | OHSUMed | OpenI | MIMIC-AKI | medNLI
---|---|---|---|---
metrics | Accuracy | AUC | AUC | F1 | Accuracy
BERT | 0.717 | 0.952 | 0.514 | 0.293 | 0.776
BioBERT | 0.771 | 0.954 | 0.534 | 0.324 | 0.808
ClinicalBERT | 0.741 | 0.967 | 0.738 | 0.444 | 0.812
Clinical-Longformer | 0.766 | 0.977 | 0.762 | 0.484 | 0.842
Clinical-Big Bird | 0.752 | 0.972 | 0.755 | 0.480 | 0.827

##  5 Conclusion

In this study, we introduced Clinical-Longformer and Clinical-BigBird, two
pre-trained language models that are suitable for long sequence clinical NLP
tasks. We compared these two models with the pioneering short sequence
transformer based models, e.g. ClinicalBERT, in named entity recognition,
question answering, and document classification tasks. Results demonstrated
that Clinical-Longformer and Clinical-BigBird achieved better results in both
long sequence and short sequence benchmark dataset. Future studies will be
investigated on other models, for example TinyBERT, that are developed to
solve the memory consumption of long sequences as well as the generalizability
of our proposed models to clinical text generation and summarization.

## Acknowledgments

This was was supported in part by NIH grants U01TR003528 and R01LM013337.

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [3] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  arXiv preprint arXiv:2005.14165, 2020.
  * [4] Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi.  Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction.  NPJ digital medicine, 4(1):1–13, 2021.
  * [5] Liang Yao, Zhe Jin, Chengsheng Mao, Yin Zhang, and Yuan Luo.  Traditional chinese medicine clinical records classification with bert and domain specific corpora.  Journal of the American Medical Informatics Association, 26(12):1632–1636, 2019.
  * [6] Zachariah Zhang, Jingshu Liu, and Narges Razavian.  Bert-xml: Large scale automated icd coding using bert pretraining.  arXiv preprint arXiv:2006.03685, 2020.
  * [7] Liang Yao, Chengsheng Mao, and Yuan Luo.  Kg-bert: Bert for knowledge graph completion.  arXiv preprint arXiv:1909.03193, 2019.
  * [8] Andrew Wen, Mohamed Y Elwazir, Sungrim Moon, and Jungwei Fan.  Adapting and evaluating a deep learning language model for clinical why-question answering.  JAMIA open, 3(1):16–20, 2020.
  * [9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [10] Shang Gao, Mohammed Alawad, Michael Todd Young, John Gounley, Noah Schaefferkoetter, Hong-Jun Yoon, Xiao-Cheng Wu, Eric B Durbin, Jennifer Doherty, Antoinette Stroup, et al.  Limitations of transformers on clinical text classification.  IEEE journal of biomedical and health informatics, 2021.
  * [11] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  arXiv preprint arXiv:1904.05342, 2019.
  * [12] Maria Mahbub, Sudarshan Srinivasan, Ioana Danciu, Alina Peluso, Edmon Begoli, Suzanne Tamang, and Gregory D Peterson.  Unstructured clinical notes within the 24 hours since admission predict short, mid & long-term mortality in adult icu patients.  Plos one, 17(1):e0262182, 2022.
  * [13] Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  arXiv preprint arXiv:2004.05150, 2020.
  * [14] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al.  Big bird: Transformers for longer sequences.  In NeurIPS, 2020.
  * [15] Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang.  Etc: Encoding long and structured inputs in transformers.  arXiv preprint arXiv:2004.08483, 2020.
  * [16] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics, 36(4):1234–1240, 2020.
  * [17] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott.  Publicly available clinical bert embeddings.  arXiv preprint arXiv:1904.03323, 2019.
  * [18] Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew P Lungren.  Chexbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert.  arXiv preprint arXiv:2004.09167, 2020.
  * [19] Yun He, Ziwei Zhu, Yin Zhang, Qin Chen, and James Caverlee.  Infusing disease knowledge into bert for health question answering, medical inference and disease name recognition.  arXiv preprint arXiv:2010.03746, 2020.
  * [20] Sicheng Zhou, Liwei Wang, Nan Wang, Hongfang Liu, and Rui Zhang.  Cancerbert: a bert model for extracting breast cancer phenotypes from electronic health records.  arXiv preprint arXiv:2108.11303, 2021.
  * [21] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  arXiv preprint arXiv:1901.02860, 2019.
  * [22] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  arXiv preprint arXiv:2001.04451, 2020.
  * [23] Simon Levine and aargh2d2.  simonlevine/clinical-longformer v0.1-alpha, December 2020.
  * [24] Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark.  Mimic-iii, a freely accessible critical care database.  Scientific data, 3(1):1–9, 2016.
  * [25] Changhan Wang, Kyunghyun Cho, and Jiatao Gu.  Neural machine translation with byte-level subwords.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9154–9160, 2020.
  * [26] Philipp Cimiano, Christina Unger, and John McCrae.  Ontology-based interpretation of natural language.  Synthesis Lectures on Human Language Technologies, 7(2):1–178, 2014\.
  * [27] Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng.  emrqa: A large corpus for question answering on electronic medical records.  arXiv preprint arXiv:1809.00732, 2018.
  * [28] Xiang Yue, Bernal Jimenez Gutierrez, and Huan Sun.  Clinical reading comprehension: A thorough analysis of the emrqa dataset.  arXiv preprint arXiv:2005.00574, 2020.
  * [29] Minki Kang, Moonsu Han, and Sung Ju Hwang.  Neural mask generator: Learning to generate adaptive word maskings for language model adaptation.  arXiv preprint arXiv:2010.02705, 2020.
  * [30] Sarvesh Soni and Kirk Roberts.  Evaluation of dataset selection for pre-training and fine-tuning transformer language models for clinical question answering.  In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5532–5538, 2020.
  * [31] Wonjin Yoon, Chan Ho So, Jinhyuk Lee, and Jaewoo Kang.  Collabonet: collaboration of deep neural networks for biomedical named entity recognition.  BMC bioinformatics, 20(10):55–65, 2019.
  * [32] Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang, Curtis Langlotz, and Jiawei Han.  Cross-type biomedical named entity recognition with deep multi-task learning.  Bioinformatics, 35(10):1745–1752, 2019.
  * [33] Özlem Uzuner, Yuan Luo, and Peter Szolovits.  Evaluating the state-of-the-art in automatic de-identification.  Journal of the American Medical Informatics Association, 14(5):550–563, 2007.
  * [34] Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall.  2010 i2b2/va challenge on concepts, assertions, and relations in clinical text.  Journal of the American Medical Informatics Association, 18(5):552–556, 2011.
  * [35] Weiyi Sun, Anna Rumshisky, and Ozlem Uzuner.  Evaluating temporal relations in clinical text: 2012 i2b2 challenge.  Journal of the American Medical Informatics Association, 20(5):806–813, 2013.
  * [36] Amber Stubbs and Özlem Uzuner.  Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/uthealth corpus.  Journal of biomedical informatics, 58:S20–S29, 2015.
  * [37] Lance A Ramshaw and Mitchell P Marcus.  Text chunking using transformation-based learning.  In Natural language processing using very large corpora, pages 157–176. Springer, 1999.
  * [38] Yikuan Li, Liang Yao, Chengsheng Mao, Anand Srivastava, Xiaoqian Jiang, and Yuan Luo.  Early prediction of acute kidney injury in critical care setting using clinical notes.  In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 683–686. IEEE, 2018.
  * [39] Mengxin Sun, Jason Baron, Anand Dighe, Peter Szolovits, Richard G Wunderink, Tamara Isakova, and Yuan Luo.  Early prediction of acute kidney injury in critical care setting using clinical notes and structured multivariate physiological measurements.  MedInfo, 264:368–372, 2019.
  * [40] Dina Demner-Fushman, Sameer Antani, Matthew Simpson, and George R Thoma.  Design and development of a multimodal biomedical information retrieval system.  Journal of Computing Science and Engineering, 6(2):168–177, 2012\.
  * [41] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng.  Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.  Scientific data, 6(1):1–8, 2019.
  * [42] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M Summers.  Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9049–9058, 2018.
  * [43] Alexey Romanov and Chaitanya Shivade.  Lessons from natural language inference in the clinical domain.  arXiv preprint arXiv:1808.06752, 2018.
  * [44] William Hersh, Chris Buckley, TJ Leone, and David Hickam.  Ohsumed: An interactive retrieval evaluation and new large test collection for research.  In SIGIR’94, pages 192–201. Springer, 1994.
  * [45] Liang Yao, Chengsheng Mao, and Yuan Luo.  Graph convolutional networks for text classification.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370–7377, 2019.