# SL-CycleGAN: Blind Motion Deblurring in Cycles using Sparse Learning

Ali Syed Saqlain
NCEPU, Beijing, China
ncepu.edu.cn
<EMAIL_ADDRESS>Li-Yun Wang
Portland State University, U.S.A
pdx.edu
<EMAIL_ADDRESS>Fang Fang
NCEPU, Beijing, China
ncepu.edu.cn
<EMAIL_ADDRESS>
###### Abstract

In this paper, we introduce an end-to-end generative adversarial network (GAN)
based on sparse learning for single image blind motion deblurring, which we
called SL-CycleGAN. For the first time in blind motion deblurring, we propose
a sparse ResNet-block as a combination of sparse convolution layers and a
trainable spatial pooler k-winner based on HTM (Hierarchical Temporal Memory)
to replace non-linearity such as ReLU in the ResNet-block of SL-CycleGAN
generators. Furthermore, unlike many state-of-the-art GAN-based motion
deblurring methods that treat motion deblurring as a linear end-to-end
process, we take our inspiration from the domain-to-domain translation ability
of CycleGAN, and we show that image deblurring can be cycle-consistent while
achieving the best qualitative results. Finally, we perform extensive
experiments on popular image benchmarks both qualitatively and quantitatively
and achieve the record-breaking PSNR of 38.087 dB on GoPro dataset, which is
5.377 dB better than the most recent deblurring method.

Figure 1: Photos of Triton by Voyager, Neptune’s satellite. Images taken from
NASA’s official website [46]. From left to right: first column, the blurry
input, second, the restored results from [36] and finally, our restored
results. The synthetic motion blur in the input images is of magnitude of 96
pixels with 0 degrees angle.

##  1 Introduction

Motion blur is one of the most recursive problems of paramount importance in
the field of computer vision and digital photography. It is mainly caused by
the streaking of fast moving objects in an image or video frames, also some of
the other reasons could be the camera shake or long exposure time [37, 68, 74,
47, 60]. An interesting way to understand motion blur is to understand the
concept of relative motion, _e.g_., motion of an object relative to the
observer in an instance of time. During a single exposure time, the image
captured by the camera, especially when an object is moving in the captured
image may represent a scene over a continuous interval of time. Such motion of
an object in a captured image, causes motion blur artifacts or more
specifically displacement of pixels. Removal of motion blur from the images,
is to obtain the clean sharp images from the blurry inputs by minimizing the
mismatch between the latent sharp image and the restored sharp image.
Considering the fact, that motion blur in real-world is commonly shift-variant
or non-uniform in nature, and the depth or the density of blur may fluctuate
over different regions of an image [6, 12, 22, 14]. The earlier blind
deconvolution methods assume the estimation of unknown blur kernel by studying
image priors [33, 40, 84, 73, 69, 82, 51], Weiner deconvolution [71, 33], or
Richardson-Lucy bayesian approach [58]. However, such outdated methods
inevitably need the handcrafted image priors, while not to mention the cost of
computational resources and the exponential increase in complexity.

Thanks to the recent advancements in deep learning, and its tremendous feature
learning ability from training to testing in blind image deblurring task [59,
63, 45, 66, 78], we do not need to rely on conventional approaches anymore.
Especially, considering the revolution that GANs [18] have brought not only in
general-purpose computer vision tasks [29, 30], but also in blind motion
deblurring task [35, 36, 61, 5, 80]. Despite the colossal success of GANs in
blind motion deblurring, the quality of restored images from the blurry inputs
is still straggling. No doubt, either scale-wise stacking of convolution
layers [63, 17, 48, 45, 59, 66, 78, 16, 76] or versions of GAN-based DeblurGAN
models [35, 36] have significantly improved the performance of restoration
both in terms of qualitative and quantitative analysis.

To address the problem of non-uniform blind image deblurring, and to propose
such a GAN-based blind image motion deblurring network, that, unlike other
GAN-based models does not treat the restoration of sharp image as a linear
end-to-end process from input to output. Instead, our proposed approach (SL-
CycleGAN) treats blind motion deblurring as a domain-to-domain translation
problem. What’s even more interesting is, we take the inspiration for this
research from the sparse representation learning of [1], not only this, but we
also combine Hawkins _et al_. [21] research on Hierarchical temporal memory
(HTM) and “A thousand brains: A new theory of intelligence” by Jeff Hawkins
[20]. By observing other GAN-based state-of-the-art methods for blind motion
deblurring [35, 36, 7], the results achieved by our proposed framework
outperform state-of-the-art motion deblurring methods, and speak for
themselves both qualitatively and quantitatively. Fig. 1 shows the supreme
reconstruction ability of our proposed method against DeblurGAN-v2 [36] on
low-light space images.

Our contributions in this research paper are summarized as follow:

  * •

The Framework: There is no denial that, GAN-based models are notorious in
nature when it comes to problems such as mode-collapse and vanishing gradient
[3, 56]. Therefore, a thoughtful choice of network that is able to tackle such
issues is of vital importance. We adopt CycleGAN [83] for its amazing ability
of domain-to-domain translation. Unlike, other GAN-based models for blind
motion deblurring, our proposed network is cycle-consistent, that means, not
only the generators of our network are able to deblur the blurry input but
also they are able to reconstruct synthetic non-uniform motion blur similar to
the original blurry input.

  * •

Sparse Convolutions: Our second contribution is the adoption of intrinsic
advantages of high dimensional sparse representation through sparse
convolutions, similar to [1]. The primary reason why we choose sparse
convolutions over standard convolutions layers in our proposed generator
architecture is, sparse representations are more robust to noise and
interference.

  * •

HTM: Hierarchical Temporal Memory (HTM) is an algorithm that models how
neocortex of a human brain performs complex world calculations such as
understanding of visual patterns, context of spoken language, perceiving the
information through touch and other sensory organs [21]. Our final
contribution is, utilizing a trainable HTM spatial pooler such as k-winner [1]
to replace non-linearity ReLU$\left(\cdot\right)$ with k-winner in the
residual-block of the generator network. The reason of k-winner as a
replacement for classic ReLU is, it is naturally more robust to variance in
noise and interference from random signals. In addition, k-winner constraints
the output of each layer to the most active non-zero units.

##  2 Related work

###  2.1 Motion Deblurring

Earlier methods treat blind motion image deblurring as an image deconvolution
problem [60, 15, 11, 72]. Similarly, sparse-based methods before the
introduction of CNNs explore the sparse image gradients in the input blurry
images [34, 40, 50, 54, 64, 72, 73]. Meanwhile, other similar motion
deblurring approaches focus more on the advantages of patch-wise estimation
[44] and estimation of dark channel image priors [51]. However, these
conventional image deconvolution methods assume the blur to be uniform, while
real-world blur is mostly non-uniform or shift-variant. Since the introduction
of deep learning and CNNs, the blind motion deblurring community has seen
exponential improvement in the quality of restored sharp images such as [63,
17, 48]. Nah _et al_. [45] proposed a deep scale-wise convolution network for
dynamic scene motion deblurring. Unlike conventional deconvolution methods,
[45] eliminates the need of knowing explicit blur kernel in advance.
Similarly, Schuler _et al_. [59] proposed deep CNN network for blind motion
deblurring in a coarse-to-fine scheme. Tao _et al_. [66] proposed a encoder-
decoder scale-wise recurrent network architecture for blind motion deblurring.
Zhang _et al_. [78] proposed a sequential CNN architecture for dynamic scene
deblurring, while achieving impressive results in comparison with [45] and
[66]. Gao _et al_. [16] proposed a parameter selective sharing scheme, and a
multi-scale encoder-decoder model with nested skip connections for dynamic
scene deblurring. Cai _et al_. [7] proposed a dynamic scene motion deblurring
network that investigates the dark and bright channel image priors in the
input blurry images.

Figure 2: Architecture of SL-CycleGAN generators. The encoder block of the
generator contains strided convolutions layers with stide of $\frac{1}{2}$.
Each convolution layer in the encoder block and TransposeConv layer in the
decoder block is followed by the InstanceNorm and non-linearity ReLU. Besides,
the generators network contains 9-residual blocks. Fig. 2a shows the
conventional Res-block architecture with Conv layers, InstanNorm, and ReLU,
while Fig. 2b shows the modified Res-block with Sparse-Conv layer,
InstanceNorm, and ReLU is replaced by HTM k-winn. Each Sparse Res-block
contains a Sparse-conv layer, InstanceNorm, and k-winn.

###  2.2 GANs for Motion Deblurring

Generative adversarial networks (GANs) [18] since their introduction, have
been widely used for computer vision tasks. A GAN architecture is a deep
neural network that consists of mainly a generator $G$ and a discriminator
$D$. A generator network that generates a fake generated sample $G(z)$,
normally takes a random noise vector $z$ as an input, more specifically in a
deblurring scenario, an input blurry image is passed through the generator
network and it outputs a fake sharp version of the input image. While the
discriminator network acts as a classifier by discriminating between the real
data sample $x$ and the generated sample $G(z)$. Both of these adversaries
play a minimax game and keep getting better and better. Theoretically, the
main goal of such adversarial network is to approximate the generated
distribution ${{p}_{z}}$ to the real data distribution ${{p}_{d}}$. The
minimax objective function for GANs can be formulated as,

$\underset{G}{\mathop{\min}}\,\underset{D}{\mathop{\max}}\,\left[{{\mathbb{E}}_{x\sim{{p}_{d}}}}\log(D(x))\right]+\left[{{\mathbb{E}}_{z\sim{{p}_{z}}}}\log(1-D(G(z)))\right]$
(1)

Based on the success of GANs, Kupyn _et al_. [35] proposed a conditional GAN-
framework (DeblurGAN) for single image blind motion deblurring. DeblurGAN
consists of a generator and a discriminator network for deblurring task, while
utilizing Wasserstein loss function [4] and optimization criteria with an
additional gradient penalty to tackle GAN related isssues [19]. Kupyn _et al_.
[36] proposed the improved version of previous DeblurGAN [35], called
DeblurGAN-v2. The DeblurGAN-v2 modified the original architecture of the
generator network of DeblurGAN by incorporating the Feature pyramid network
(FPN) [43] for improved quality. Similarly a relativistic local and global
discriminator network is introduced [28] with Inception-ResNet-v2 [65] as a
backbone of the network. Shao _et al_. [61] proposed a GAN-based deblurring
framework that explores the dark and bright channel image priors. Asim _et
al_. [5] proposed a blind image deblurring network based on deep generative
priors. However, their proposed method lack the experimental analysis on real-
world benchmarks for deblurring. Zhang _et al_. [80] proposed a image
deblurring and denoising network by combining the noisy and blurry image pairs
acquired in a burst. Similarly, Lin _et al_. [42] deployed a GAN-based
framework for blind dynamic scene deblurring. Several other approaches either
exploit the network architecture in scale-wise convolutions [2, 81, 26], or
the deep generative and discriminative priors [57, 41] for dynamic scene blind
motion deblurring.

##  3 SL-CycleGAN Network Architecture

The detailed architecture of SL-CycleGAN generators is demonstrated in Fig. 2.
Given a pair of blurry and sharp images
$\left\\{{{x}_{i}}\right\\}_{i=1}^{N}\in{{X}_{blurry}}$ and
$\left\\{{{y}_{i}}\right\\}_{j=1}^{M}\in{{Y}_{sharp}}$, the generators of SL-
CycleGAN learn the translations from ${{X}_{blurry}}$ to ${{Y}_{sharp}}$.
Taking the inspiration from the original CycleGAN model [83], SL-CycleGAN also
introduces two generator networks ${{G}_{X}}$ and ${{G}_{Y}}$. Whereas,
${{G}_{X}}$ learns the translation function from ${X}\to{Y}$ such that
${{G}_{X}}:{{X}_{blur}}\to{{Y}_{sharp}}$. Similarly, the second generator
${{G}_{Y}}$ learns the mapping from ${Y}\to{X}$ such that
${{G}_{Y}}:{{Y}_{sharp}}\to{{X}_{blur}}$. A pair of adversarial discriminators
${{D}_{X}}$ and ${{D}_{Y}}$ are also proposed. While ${{D}_{X}}$ learns to
differentiate between the blurry input ${{x}_{i}}$ and the translated image
${{G}_{Y}}(\hat{y})$. Similarly, ${{D}_{Y}}$ differentiates between the latent
sharp image ${{y}_{i}}$ and the translated image ${{G}_{X}}(\hat{x})$. The
architecture of our discriminator networks is similar to the PatchGAN
$70\times 70$ discriminator [24].

###  3.1 Cycle-consistent Deblurring

As mentioned earlier in Sec. 1, GANs have been known for mode collapse. The
main reason behind mode collapse in GANs, is its adversarial nature and the
choice of the objective function for optimization purposes. Theoretically
speaking, a generator function that maps
${{G}_{X}}:{{X}_{blur}}\to{{Y}_{sharp}}$ outputs a distribution of translated
image ${{p}_{data}}(\hat{y})$ such that the output image $\hat{y}$ is similar
to the original sharp image $y$. The assumption that the generated probability
distribution ${{p}_{data}}(\hat{y})$ strictly correlates to the original data
distribution ${{p}_{data}}({y})$ requires the generator ${G}_{X}$ to be
stochastic in nature [18]. However during inference, such theoretical
assumption does not assure that the generator will learn meaningful
translations without being the victim of mode collapse.

In order to avoid mode collapse in generators and to improve the optimization
ability of the network, Zhu _et al_. [83] argued that the adversarial
objective function of generators should be coupled with the term “cycle-
consistent”. The cycle-consistency term ensures that the generators
${{G}_{X}}$ and ${{G}_{Y}}$ are the inverse mapping functions of each other.
It can be defined as,

$\begin{split}{{L}_{cycle}}\text{(}{{G}_{X}},{{G}_{Y}})={{\mathbb{E}}_{x\sim{{p}_{data}}(x)}}\left[{{\left\|{{G}_{Y}}({{G}_{X}}(\hat{x}))-x\right\|}_{1}}\right]\\\
+{{\mathbb{E}}_{y\sim{{p}_{data}}(y)}}\left[{{\left\|{{G}_{X}}({{G}_{Y}}(\hat{y}))-y\right\|}_{1}}\right]\end{split}$
(2)

where ${{L}_{cycle}}$ in Eq. 2 represents the L1 norm, ${{p}_{data}(x)}$ and
${{p}_{data}(y)}$ represent the distributions of blurry and sharp images.

Unlike our close GAN-based competitors for blind motion deblurring [35, 36,
61, 80], we consider cycle-consistency an essential factor for blind motion
deblurring, and show that our network outperforms all the state-of-the-art
methods in blind motion dynamic scence deblurring in Sec. 4.

###  3.2 Sparse Convolutions and HTM

The history behind sparse representations is nothing new, in fact, Olshausen
_et al_. [49] showed that deploying sparse embeddings and sparse objective
functions in encoders can lead to the representations that are similar to the
learnt representations in primate visual cortex. Similarly, Chen _et al_. [10]
developed hierarchical sparse representations that are similar to hierarchical
feature detectors. The weights for each unit in sparse convolution layers in
our Resnet architecture are randomly sampled from a sparse subset of the
previous layer. Additionally, the output of each layer is bounded to only $k$
most non-zero active units. The number of non-zero products in each layer is
(sparsity of layer $l$)$\times$(sparse weights of layer $l+1$). Fig. 2a
represents a conventional arrangement of a residual-block, where each conv
layer is followed by InstanceNorm and ReLU as an activation function. Fig. 2b
is our modified structure for residual-block, which we called Sparse ResNet-
block. To select the most active $k$ non-zero units and for each unit to be
equally active in order to be robust to noise and interference, boosting
techniques are applied to sparse convolution layers which can be defined as,

$c_{i}^{l}(t)=(1-\alpha)c_{i}^{l}(t-1)+\alpha\cdot\left[i\in
topIndice{{s}^{l}}\right]$ (3)

Eq. 3 represents the HTM boosting duty cycle through k-winner, which
calculates the running average of each active unit cycle. Where $c_{i}^{l}(t)$
is the unit duty cycles for each unit $i$ in layer $l$ at time $t$. The
boosting coefficient $b_{i}^{l}={{e}^{\beta({{{\hat{a}}}^{l}}-c_{i}^{l}(t))}}$
is then measured for each unit based on the target and current average duty
cycle. Where ${{{\hat{a}}}^{l}}$ denotes the number of units that are expected
to be active, while the boosting factor $\beta$ is a positive parameter
responsible for controlling the strength of boosting.

In order to construct sparse convolutions with HTM k-winner in Resnet
architecture, k-winner is applied to the output of InstanceNorm in each
residual-block respectively with stride of 1 and kernel size of $3\times 3$.

###  3.3 Loss functions

In this section, we discuss the loss functions for our proposed SL-CycleGAN,
the overall loss function is the combination of three different loss
functions.

Adversarial loss: The adversarial objective functions is an essential
component for blind motion deblurring in GANs. The classic Jensen-Shannon
divergence (JSD) based minimax loss function for GANs is proposed by [18] is
defined as in Eq. 1. However, the objective fucntion in Eq. 1 suffers from the
serious issues such as mode collapse and vanishing gradient. Thus, a
conventional minimax objective function is not a good choice for our blind
motion deblurring task. Instead, we choose the objective function of [19] with
gradient penalty term. The adversarial functions of our proposed network can
be defined as follow,

$\begin{split}{{L}_{adv}}({{G}_{X}},{{D}_{Y}},{{X}_{blur}},{{Y}_{sharp}})={{\mathbb{E}}_{y\sim{{p}_{data}}(y)}}\left[{{D}_{Y}}(y)\right]\\\
-{{\mathbb{E}}_{x\sim{{p}_{data}}(x)}}\left[{{D}_{Y}}({{G}_{X}}(x))\right],\end{split}$
(4)
$\begin{split}{{L}_{adv}}({{G}_{Y}},{{D}_{X}},{{Y}_{sharp}},{{X}_{blur}})={{\mathbb{E}}_{x\sim{{p}_{data}}(x)}}\left[{{D}_{X}}(x)\right]\\\
-{{\mathbb{E}}_{y\sim{{p}_{data}}(y)}}\left[{{D}_{X}}({{G}_{Y}}(y))\right]\end{split}$
(5)

where ${{G}_{X}}$ and ${{G}_{Y}}$ are the inverse mapping functions of each
other. The adversarial functions for both the generators and discriminators in
Eq. 4 and Eq. 5 are combined along with cycle-consistency loss ${{L}_{cycle}}$
from Eq. 3 during the inference.

Perceptual loss: We observe that by incorporating only adversarial and cycle-
consistency loss, the quality of the restored images is slightly degraded. To
further improve the quality of restored images, we adopt the perceptual loss
of pre-trained VGG-19 by Jonhnson _et al_. [27]. The perceptual loss can be
defined as,

${{L}_{perc}}=\frac{1}{{{W}_{i,j}}{{H}_{i,j}}}\sum\limits_{w=1}^{{{W}_{i,j}}}{\sum\limits_{h=1}^{{{H}_{i,j}}}{{{\left({{\phi}_{i,j}}{{({{I}_{S}})}_{w,h}}-{{\phi}_{i,j}}{{({{G}_{\theta}}({{I}_{B}}))}_{w,h}}\right)}^{2}}}}$
(6)

where ${{H}_{i,j}}$ and ${{W}_{i,j}}$ in Eq. 6 indicate the height and width
of the conv3$\times$3 layers in the pre-trained VGG19 network.
${{\phi}_{i,j}}$ indicates the obtained feature maps by the j-th convolution
layer after the activation function and before the i-th maxpooling layer.
${{I}_{S}}$ and ${{G}_{\theta}}({{I}_{B})}$ represent the real sharp and the
restored deblurred images.

Overall Loss Function: The overall loss function for proposed SL-CycleGAN can
be defined as,

${{L}_{SL-
CycleGAN}}={{L}_{adv}}+{{\lambda}_{cyc}}{{L}_{cycle}}+{{\lambda}_{perc}}{{L}_{perc}}$
(7)

where ${{\lambda}_{cyc}}$ represents the relative coefficient of adversarial
functions for ${G}_{X}$ and ${G}_{Y}$. ${\lambda}_{perc}$ is the hyper-
parameter for perceptual loss ${{{L}}_{perc}}$.

##  4 Experimental evaluation

Method |  | Year |  | PSNR |  | SSIM
---|---|---|---|---|---|---
DeepDeblur [45] |  | 2016 |  | 30.12 |  | 0.9021
DeblurGAN [35] |  | 2018 |  | 28.70 |  | 0.958
DeblurGAN-v2-Inception [36] |  | 2019 |  | 29.55 |  | 0.934
DeblurGAN+ [61] |  | 2020 |  | 28.62 |  | 0.959
DBGAN [79] |  | 2020 |  | 31.10 |  | 0.9424
RNNDeblur [78] |  | 2018 |  | 29.1872 |  | 0.9306
SRN-Deblur [66] |  | 2018 |  | 30.26 |  | 0.9342
DBCPeNet [7] |  | 2020 |  | 31.10 |  | 0.945
MTRNN [52] |  | 2019 |  | 31.15 |  | 0.945
DMPHN [76] |  | 2019 |  | 31.50 |  | 0.9483
SRN+PSS+NSC [16] |  | 2019 |  | 31.58 |  | 0.9478
Learning Even-Based Motion Deblurring [25] |  | 2020 |  | 31.79 |  | 0.949
SAPHNet [62] |  | 2020 |  | 32.02 |  | 0.953
RADNet [55] |  | 2020 |  | 32.15 |  | 0.953
BANet [67] |  | 2021 |  | 32.44 |  | 0.957
MPRNet [75] |  | 2021 |  | 32.66 |  | 0.959
MIMO-UNet++ [13] |  | 2021 |  | 32.68 |  | 0.959
HINet [9] |  | 2021 |  | 32.71 |  | 0.959
SL-CycleGAN (Ours) |  | 2021 |  | 38.087 |  | 0.954

Table 1: Quantitative comparison of Blind image deblurring on GoPro dataset
[45]. Our proposed method SL-CycleGAN achieves the highest PSNR of 38.087 dB
on blind image motion deblurring task. Figure 3: Deblurring results of test
images from GoPro dataset. (a) Blurry inputs. (b) Magnified blurry image
patches. (c) Corresponding sharp image patches. (d) Deblurring results of
[66]. (e) Deblurring results of [35]. (f) Deblurring results of [36]. (g)
Deblurring results of [61]. (h) Deblurring results of [80]. (i) Finally,
deblurring results of our proposed SL-CycleGAN. Figure 4: Visual comparison of
test images from Kohler dataset [32]. (a) Input blurry images. (b) Magnified
blurry image patches. (c) Magnified real sharp image patches. (d) Deblurring
results of [66]. (e) Deblurring results of [35]. (f) Deblurring results of
[36]. (g) Deblurring results of [7]. (h) Deblurring results of our proposed
method.

###  4.1 Experiment settings

We have used Pytorch [53] for all our experiments on Nvidia GTX 1080ti with
11G GPU. We performed experiments on three image benchmarks, GoPro dataset
[45], Kohler dataset [32], and Lai dataset [38]. We resized all the images in
all three image benchmarks to 256$\times$256 for training and testing and
apply data augmentation. For optimization of the generators and the
discriminators, we use the Adam optimizer [31] with ${{\beta}=0.999}$ and
batch size of 1. We train our model on all these three image benchmarks for
200 training epochs each with an initial learning rate of 0.0002 for first 100
epochs and linearly decay to zero over next 100 iterations. For all the
experiments we set the values of ${{\lambda}_{cyc}}=10$ and
${{\lambda}_{perc}}=100$ in Eq. 7. We use the gradient penalty term of [19]
for the discriminator networks, which is set to 10. We do not use dropout
layer in our modified Sparse ResNet architecture, since [1] in their research
show that the utilization of k-winner with sparse convolutions replaces the
need of dropout layers in the network architecture. The training time of our
proposed network (SL-CycleGAN) on one dataset for total of 200 training epochs
took 2 days to complete, which is 6 days in total for three datasets.

Method |  | PSNR |  | SSIM
---|---|---|---|---
Whyte _et al_. [70] |  | 27.02 |  | 0.809
Xu _et al_. [73] |  | 27.40 |  | 0.810
Sun _et al_. [63] |  | 25.21 |  | 0.772
DeepDeblur [45] |  | 26.48 |  | 0.807
DeblurGAN [35] |  | 25.86 |  | 0.802
DeblurGAN-v2 [36] |  | 26.10 |  | 0.816
SRN-Deblur [66] |  | 26.75 |  | 0.837
DMPHN [76] |  | 24.21 |  | 0.7562
Zhang _et al_. [78] |  | 25.71 |  | 0.800
Kim _et al_. [23] |  | 24.68 |  | 0.794
DBCPeNet [7] |  | 26.79 |  | 0.839
SL-CycleGAN (ours) |  | 30.818 |  | 0.843

Table 2: Quantitative comparison on Kohler dataset [32]. Our proposed SL-
CycleGAN achieves significant improvement both in terms of PSNR and SSIM.

Method |  | PSNR |  | SSIM
---|---|---|---|---
Fergus _et al_. [15] |  | 22.870 |  | 0.682
Cho [11] |  | 23.272 |  | 0.699
Xu _et al_. [73] |  | 25.586 |  | 0.773
Krishnan _et al_. [34] |  | 23.070 |  | 0.716
Levin _et al_. [39] |  | 21.855 |  | 0.651
Whyte _et al_. [70] |  | 23.232 |  | 0.667
Sun _et al_. [63] |  | 24.649 |  | 0.756
Xu [72] |  | 25.319 |  | 0.765
Zhang _et al_. [77] |  | 22.918 |  | 0.679
Chakrabarti _et al_. [8] |  | 25.389 |  | 0.769
Nah _et al_. [45] |  | 24.224 |  | 0.713
Gong _et al_. [17] |  | 23.805 |  | 0.694
DeblurGAN [35] |  | 24.561 |  | 0.741
DeblurGAN-v2 [36] |  | 25.634 |  | 0.754
SRN-Deblur [66] |  | 25.231 |  | 0.752
SL-CycleGAN (ours) |  | 27.935 |  | 0.766

Table 3: Quantitative comparison on Lai dataset [38]. Our Proposed approach
shows superior performance than all the other methods in terms of PSNR.

###  4.2 Image Benchmarks

Evaluation on GoPro Dataset: GoPro dataset was proposed by Nah _et al_. [45],
which consists of 3214 images in total for deblurring task, 2103 training
image pairs of blurred and sharp images while the rest of 1111 images are
reserved for testing purposes. It is the most commonly used benchmark for
blind image deblurring task. The quantitative evaluation on GoPro dataset is
presented in Tab. 1. While Tab. 1 presents the timeline of all the state-of-
the-art deep learning-based deblurring methods starting from year 2016-2021
both in terms of PSNR and SSIM. Our proposed method SL-CycleGAN outperforms
all the state-of-the-art methods on GoPro deblurring task, while achieving the
record-breaking PSNR of 38.087 dB, which is 5.377 dB better than the most
recent deblurring method HiNet [9]. Similarly, the average SSIM value of our
proposed network remains in the list of top five most recent deblurring
methods. The qualitative results on GoPro dataset are given in Fig. 3. In
comparison with the state-of-the-art blind deblurring methods [66, 35, 36, 61,
80], our proposed approach restores the sharp images from the blurry inputs
that are similar to the real sharp images and can be clearly seen in Fig. 3.
The resemblance between our restored and the real sharp image patches is quite
high in comparison with other approaches.

Evaluation on Kohler Dataset: Kohler _et al_. [32] proposed a real-world
deblurring datasat that consists of 4 latent sharp images and 48 corresponding
blurry images of varying blur kernel intensities. It is the most commonly used
benchmark for blind image deblurring comparison. The quantitative comparison
of blind image deblurring on Kohler dataset is given in Tab. 2. Our SL-
CycleGAN outperforms the other state-of-the-art methods by achieving the
average PSNR of 30.818 dB and SSIM of 0.843, while our closest competitor
DBCPeNet [7] achieves the PSNR of 26.79 dB and SSIM of 0.839. Similarly,
DeblurGAN [35], DeblurGAN-v2 [36], SRN-Deblur [66] and DMPHN [76] show
quantitatively inferior performance than our proposed approach. The visual
comparison of test images from kohler dataset is presented in Fig. 4. We can
see from Fig. 4 that our deblurred sharp image patches retain the texture
details and the sharpness similar to the original latent sharp image patches.
In comparison with [66, 35, 36, 7], our proposed model shows the ability to
understand the distribution of non-uniform blur over different image regions
even when the subject of focus lacks significant light reflection.

Figure 5: Visual ablation study and analysis on GoPro [45], Kohler [32], and
Lai [38] datasets. First row: Images from GoPro. Second row: Images from
Kohler dataset. Third row: Images from Lai dataset. Meanwhile the first column
represents the blurry inputs, second column: represent deblur results of
CycleGAN, third column: CycleGAN with perceptual loss and sparse convs, fourth
column: CycleGAN and sparse convs, and finally CycleGAN + perceptual loss +
sparse + k-winn (SL-CycleGAN).

Quantitative evaluation on Lai Dataset: Lai _et al_. [38] proposed a benchmark
for blind image deblurring task, which contains 100 real-world blurred images,
they also generated synthetic dataset with 200 generated blurred images
containing images of both uniform and non-uniform blur. We present the
quantitative comparison on Lai dataset in Tab. 3. We can observe from Tab. 3
that our proposed method achieves 2.301 dB improvement in PSNR than the second
highest DeblurGAN-v2 [36]. Meanwhile, the visual results based on the ablation
study and analysis on Lai dataset are shown in Fig. 5, which we further
discuss in Sec. 4.3 along with the ablation analysis on GoPro and Kohler
datasets.

GoPro | PSNR(dB) | SSIM | MS-SSIM
---|---|---|---
CycleGAN | 31.835 | 0.844 | 0.986
CycleGAN+VGG-19(perceptual)+Sparse | 37.852 | 0.954 | 0.997
CycleGAN+Sparse | 33.135 | 0.876 | 0.990
CycleGAN+VGG-19(perceptual)+Sparse+k-winn (SL-CycleGAN) | 38.087 | 0.954 | 0.997
Kohler |
CycleGAN | 29.870 | 0.814 | 0.983
CycleGAN+VGG-19(perceptual)+Sparse | 29.985 | 0.814 | 0.984
CycleGAN+Sparse | 30.461 | 0.823 | 0.985
CycleGAN+VGG-19(perceptual)+Sparse+k-winn (SL-CycleGAN) | 30.818 | 0.843 | 0.987
Lai |
CycleGAN | 25.034 | 0.662 | 0.970
CycleGAN+VGG-19(perceptual)+Sparse | 27.581 | 0.764 | 0.983
CycleGAN+Sparse | 27.564 | 0.757 | 0.983
CycleGAN+VGG-19(perceptual)+Sparse+k-winn (SL-CycleGAN) | 27.935 | 0.766 | 0.984

Table 4: Quantitative ablation study on GoPro [45], Kohler [32] and Lai [38]
datasets.

###  4.3 Ablation Study

We conduct an ablation study on the components of SL-CycleGAN and observe the
impact and effectiveness of these components both qualitatively and
quantitatively. We present the visual ablation study and on three image
benchmarks in Fig. 5, while considering original CycleGAN [83] as an starting
point. Meanwhile, we gradually keep adding modifications to the generator
networks such as replacing standard conv layers in the ResNet by sparse-convs
and adding VGG-19 perceptual loss, then eliminating perceptual loss and
leaving only sparse conv layers. Finally, we modify the network by integrating
perceptual loss, sparse-convs and replace ReLU with k-winner in the ResNet
generators architecture. We call the final version of our network SL-CycleGAN
(CycleGAN + perceptual + sparse-convs + k-winn). We can see from Fig. 5 that
all our sparse versions of the network perform better visually than just
CycleGAN, especially the final version SL-CycleGAN produces visually appealing
results. Similarly in Tab. 4 of ablation study, SL-CycleGAN outperforms all
the preceding versions quantitatively.

Limitations: Keeping in mind that “Honesty is the best policy”. We observe
that during the inference on GoPro dataset, some of the restored images by SL-
CyleGAN show slightly dim light in comparison with the original bright sharp
images. However, we consider it as an inherited issue from the original
CycleGAN and cycle-consistency loss [83].

##  5 Conclusion

This paper introduces a novel blind image deblurring network SL-CycleGAN, that
for the first time, utilizes sparse representation learning with HTM k-winner
for improved image deblurring and is more robust towards noise and
interference. Meanwhile, achieving the best qualitative and quantitative
results on popular image benchmarks.

## References

  * [1] Subutai Ahmad and Luiz Scheinkman.  How can we be so dense? the benefits of using highly sparse representations.  arXiv preprint arXiv:1903.11257, 2019.
  * [2] Raied Aljadaany, Dipan K Pal, and Marios Savvides.  Douglas-rachford networks: Learning both the image prior and data fidelity terms for blind image deconvolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10235–10244, 2019.
  * [3] Martin Arjovsky and Léon Bottou.  Towards principled methods for training generative adversarial networks.  arXiv preprint arXiv:1701.04862, 2017.
  * [4] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In International conference on machine learning, pages 214–223. PMLR, 2017.
  * [5] Muhammad Asim, Fahad Shamshad, and Ali Ahmed.  Blind image deconvolution using deep generative priors.  IEEE Transactions on Computational Imaging, 6:1493–1506, 2020.
  * [6] John Bardsley, Stuart Jefferies, James Nagy, and Robert Plemmons.  Blind iterative restoration of images with spatially-varying blur.  Optics Express, 14(5):1767–1782, 2006.
  * [7] Jianrui Cai, Wangmeng Zuo, and Lei Zhang.  Dark and bright channel prior embedded network for dynamic scene deblurring.  IEEE Transactions on Image Processing, 29:6885–6897, 2020.
  * [8] Ayan Chakrabarti, Todd Zickler, and William T Freeman.  Analyzing spatially-varying blur.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 2512–2519. IEEE, 2010.
  * [9] Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen.  Hinet: Half instance normalization network for image restoration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 182–192, 2021.
  * [10] Yubei Chen, Dylan M Paiton, and Bruno A Olshausen.  The sparse manifold transform.  arXiv preprint arXiv:1806.08887, 2018.
  * [11] Sunghyun Cho and Seungyong Lee.  Fast motion deblurring.  In ACM SIGGRAPH Asia 2009 papers, pages 1–8. 2009.
  * [12] Sunghyun Cho, Yasuyuki Matsushita, and Seungyong Lee.  Removing non-uniform motion blur from images.  In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.
  * [13] Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko.  Rethinking coarse-to-fine approach in single image deblurring.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4641–4650, 2021.
  * [14] Florent Couzinié-Devy, Jian Sun, Karteek Alahari, and Jean Ponce.  Learning to estimate and remove non-uniform image blur.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1075–1082, 2013.
  * [15] Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T Roweis, and William T Freeman.  Removing camera shake from a single photograph.  ACM Transactions on Graphics (TOG), 25(3):787–794, 2006.
  * [16] Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia.  Dynamic scene deblurring with parameter selective sharing and nested skip connections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3848–3856, 2019.
  * [17] Dong Gong, Jie Yang, Lingqiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton Van Den Hengel, and Qinfeng Shi.  From motion blur to motion flow: A deep learning solution for removing heterogeneous motion blur.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2319–2328, 2017.
  * [18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, volume 27, pages 2672–2680, 2014.
  * [19] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville.  Improved training of wasserstein gans.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 5769–5779, 2017.
  * [20] Jeff Hawkins.  A thousand brains: A new theory of intelligence, 2021.
  * [21] Jeff Hawkins, Subutai Ahmad, Donna Dubinsky, et al.  Cortical learning algorithm and hierarchical temporal memory.  Numenta Whitepaper, 1:68, 2011.
  * [22] Michael Hirsch, Christian J Schuler, Stefan Harmeling, and Bernhard Schölkopf.  Fast removal of non-uniform camera shake.  In 2011 International Conference on Computer Vision, pages 463–470. IEEE, 2011.
  * [23] Tae Hyun Kim, Byeongjoo Ahn, and Kyoung Mu Lee.  Dynamic scene deblurring.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3160–3167, 2013.
  * [24] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [25] Zhe Jiang, Yu Zhang, Dongqing Zou, Jimmy Ren, Jiancheng Lv, and Yebin Liu.  Learning event-based motion deblurring.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3320–3329, 2020.
  * [26] Jianbo Jiao, Wei-Chih Tu, Shengfeng He, and Rynson WH Lau.  Formresnet: Formatted residual learning for image restoration.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 38–46, 2017.
  * [27] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [28] Alexia Jolicoeur-Martineau.  The relativistic discriminator: a key element missing from standard gan.  In International Conference on Learning Representations, 2018.
  * [29] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [30] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  * [31] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [32] Rolf Köhler, Michael Hirsch, Betty Mohler, Bernhard Schölkopf, and Stefan Harmeling.  Recording and playback of camera shake: Benchmarking blind deconvolution with a real-world database.  In European conference on computer vision, pages 27–40. Springer, 2012.
  * [33] Dilip Krishnan and Rob Fergus.  Fast image deconvolution using hyper-laplacian priors, supplementary material.  In Neural Information Pro-cessing Systems Conference. Citeseer, 2009\.
  * [34] Dilip Krishnan, Terence Tay, and Rob Fergus.  Blind deconvolution using a normalized sparsity measure.  In CVPR 2011, pages 233–240. IEEE, 2011.
  * [35] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas.  Deblurgan: Blind motion deblurring using conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8183–8192, 2018.
  * [36] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang.  Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8878–8887, 2019.
  * [37] Reginald L Lagendijk and Jan Biemond.  Basic methods for image restoration and identification.  In The essential guide to image processing, pages 323–348. Elsevier, 2009.
  * [38] Wei-Sheng Lai, Jia-Bin Huang, Zhe Hu, Narendra Ahuja, and Ming-Hsuan Yang.  A comparative study for single image blind deblurring.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1701–1709, 2016.
  * [39] Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman.  Understanding and evaluating blind deconvolution algorithms.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1964–1971. IEEE, 2009.
  * [40] Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman.  Efficient marginal likelihood optimization in blind deconvolution.  In CVPR 2011, pages 2657–2664. IEEE, 2011.
  * [41] Lerenhan Li, Jinshan Pan, Wei-Sheng Lai, Changxin Gao, Nong Sang, and Ming-Hsuan Yang.  Learning a discriminative prior for blind image deblurring.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6616–6625, 2018.
  * [42] Jen-Chun Lin, Wen-Li Wei, Tyng-Luh Liu, C-C Jay Kuo, and Mark Liao.  Tell me where it is still blurry: Adversarial blurred region mining and refining.  In Proceedings of the 27th ACM International Conference on Multimedia, pages 702–710, 2019.
  * [43] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [44] Tomer Michaeli and Michal Irani.  Blind deblurring using internal patch recurrence.  In European conference on computer vision, pages 783–798. Springer, 2014.
  * [45] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee.  Deep multi-scale convolutional neural network for dynamic scene deblurring.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3883–3891, 2017.
  * [46] NASA.  https://photojournal.jpl.nasa.gov/.
  * [47] Shree K Nayar and Moshe Ben-Ezra.  Motion-based motion deblurring.  IEEE transactions on pattern analysis and machine intelligence, 26(6):689–698, 2004.
  * [48] Mehdi Noroozi, Paramanand Chandramouli, and Paolo Favaro.  Motion deblurring in the wild.  In German conference on pattern recognition, pages 65–77. Springer, 2017.
  * [49] Bruno A Olshausen and David J Field.  Sparse coding with an overcomplete basis set: A strategy employed by v1?  Vision research, 37(23):3311–3325, 1997.
  * [50] Jinshan Pan, Zhe Hu, Zhixun Su, and Ming-Hsuan Yang.  $l\\_0$-regularized intensity and gradient prior for deblurring text images and beyond.  IEEE transactions on pattern analysis and machine intelligence, 39(2):342–355, 2016.
  * [51] Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang.  Blind image deblurring using dark channel prior.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1628–1636, 2016.
  * [52] Dongwon Park, Dong Un Kang, Jisoo Kim, and Se Young Chun.  Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training.  In European Conference on Computer Vision, pages 327–343. Springer, 2020.
  * [53] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [54] Daniele Perrone and Paolo Favaro.  A logarithmic image prior for blind deconvolution.  International journal of computer vision, 117(2):159–172, 2016\.
  * [55] Kuldeep Purohit and AN Rajagopalan.  Region-adaptive dense network for efficient motion deblurring.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11882–11889, 2020.
  * [56] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [57] Dongwei Ren, Kai Zhang, Qilong Wang, Qinghua Hu, and Wangmeng Zuo.  Neural blind deconvolution using deep priors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3341–3350, 2020.
  * [58] William Hadley Richardson.  Bayesian-based iterative method of image restoration.  Journal of the Optical Society of America (1917-1983), 62(1):55, 1972.
  * [59] Christian J Schuler, Michael Hirsch, Stefan Harmeling, and Bernhard Schölkopf.  Learning to deblur.  IEEE transactions on pattern analysis and machine intelligence, 38(7):1439–1451, 2015.
  * [60] Qi Shan, Jiaya Jia, and Aseem Agarwala.  High-quality motion deblurring from a single image.  Acm transactions on graphics (tog), 27(3):1–10, 2008.
  * [61] Wen-Ze Shao, Yuan-Yuan Liu, Lu-Yue Ye, Li-Qian Wang, Qi Ge, Bing-Kun Bao, and Hai-Bo Li.  Deblurgan+: revisiting blind motion deblurring using conditional adversarial networks.  Signal Processing, 168:107338, 2020.
  * [62] Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan.  Spatially-attentive patch-hierarchical network for adaptive motion deblurring.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3606–3615, 2020.
  * [63] Jian Sun, Wenfei Cao, Zongben Xu, and Jean Ponce.  Learning a convolutional neural network for non-uniform motion blur removal.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 769–777, 2015.
  * [64] Libin Sun, Sunghyun Cho, Jue Wang, and James Hays.  Edge-based blur kernel estimation using patch priors.  In IEEE International Conference on Computational Photography (ICCP), pages 1–8. IEEE, 2013.
  * [65] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  * [66] Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia.  Scale-recurrent network for deep image deblurring.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8174–8182, 2018.
  * [67] Fu-Jen Tsai, Yan-Tsung Peng, Yen-Yu Lin, Chung-Chi Tsai, and Chia-Wen Lin.  Banet: Blur-aware attention networks for dynamic scene deblurring.  arXiv preprint arXiv:2101.07518, 2021.
  * [68] Ruxin Wang and Dacheng Tao.  Recent progress in image deblurring.  arXiv preprint arXiv:1409.6838, 2014.
  * [69] Manuel Werlberger, Thomas Pock, and Horst Bischof.  Motion estimation with non-local total variation regularization.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 2464–2471. IEEE, 2010.
  * [70] Oliver Whyte, Josef Sivic, Andrew Zisserman, and Jean Ponce.  Non-uniform deblurring for shaken images.  International journal of computer vision, 98(2):168–186, 2012.
  * [71] Norbert Wiener.  Extrapolation, interpolation, and smoothing of stationary time series: with engineering applications.  MIT press Cambridge, 1950.
  * [72] Li Xu and Jiaya Jia.  Two-phase kernel estimation for robust motion deblurring.  In European conference on computer vision, pages 157–170. Springer, 2010.
  * [73] Li Xu, Shicheng Zheng, and Jiaya Jia.  Unnatural l0 sparse representation for natural image deblurring.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1107–1114, 2013.
  * [74] Yitzhak Yitzhaky and Norman S Kopeika.  Identification of blur parameters from motion blurred images.  Graphical models and image processing, 59(5):310–320, 1997.
  * [75] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  Multi-stage progressive image restoration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14821–14831, 2021.
  * [76] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz.  Deep stacked hierarchical multi-patch network for image deblurring.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5978–5986, 2019.
  * [77] Haichao Zhang, David Wipf, and Yanning Zhang.  Multi-image blind deblurring using a coupled adaptive sparse prior.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1051–1058, 2013.
  * [78] Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson WH Lau, and Ming-Hsuan Yang.  Dynamic scene deblurring using spatially variant recurrent neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2521–2529, 2018.
  * [79] Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li.  Deblurring by realistic blurring.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2737–2746, 2020.
  * [80] Shuang Zhang, Ada Zhen, and Robert L Stevenson.  Deep motion blur removal using noisy/blurry image pairs.  Journal of Electronic Imaging, 30(3):033022, 2021.
  * [81] Xinyi Zhang, Hang Dong, Zhe Hu, Wei Sheng Lai, Fei Wang, and Ming Hsuan Yang.  Gated fusion network for joint image deblurring and super-resolution.  In 29th British Machine Vision Conference, BMVC 2018, 2019.
  * [82] Xinxin Zhang, Ronggang Wang, Yonghong Tian, Wenmin Wang, and Wen Gao.  Image deblurring using robust sparsity priors.  In 2015 IEEE International Conference on Image Processing (ICIP), pages 138–142. IEEE, 2015.
  * [83] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
  * [84] Daniel Zoran and Yair Weiss.  From learning models of natural image patches to whole image restoration.  In 2011 International Conference on Computer Vision, pages 479–486. IEEE, 2011.