# Single-image Defocus Deblurring by Integration of Defocus Map Prediction
Tracing the Inverse Problem Computation

Qian Ye1, Masanori Suganuma1,2, Takayuki Okatani1,2
1 Graduate School of Information Sciences, Tohoku University
2 RIKEN Center for AIP
{qian, suganuma<EMAIL_ADDRESS>
###### Abstract

In this paper, we consider the problem in defocus image deblurring. Previous
classical methods follow two-steps approaches, i.e., first defocus map
estimation and then the non-blind deblurring. In the era of deep learning,
some researchers have tried to address these two problems by CNN. However, the
simple concatenation of defocus map, which represents the blur level, leads to
suboptimal performance. Considering the spatial variant property of the
defocus blur and the blur level indicated in the defocus map, we employ the
defocus map as conditional guidance to adjust the features from the input
blurring images instead of simple concatenation. Then we propose a simple but
effective network with spatial modulation based on the defocus map. To achieve
this, we design a network consisting of three sub-networks, including the
defocus map estimation network, a condition network that encodes the defocus
map into condition features, and the defocus deblurring network that performs
spatially dynamic modulation based on the condition features. Moreover, the
spatially dynamic modulation is based on an affine transform function to
adjust the features from the input blurry images. Experimental results show
that our method can achieve better quantitative and qualitative evaluation
performance than the existing state-of-the-art methods on the commonly used
public test datasets.

##  1 Introduction

Defocus blur is inevitable when the rays from a scene not lying on the focal
plane of the camera converge to a region rather than a point on the image
plane and the region is called the circle of confusion (COC) [25]. Using a
large aperture allows more light to pass through the lens in a shorter
exposure time. But this results in a shallow depth of field (DoF), thereby
causing defocus blur. Shallow DOF is useful to make the subject stand out from
the blurry background and foreground. But on the other hand, defocus blur
causes visual information losses, which is important for other tasks like
image understanding. Thus, recovering the blurry images can help improve the
performance on these tasks.

However, defocus deblurring is still a challenging task due to the spatial
variant blur, i.e., the level of blur for each pixel is different. For
example, the scene on the focal plane is captured sharply while the scene out
of the focal plane is captured in blurry. And the level of defocus is usually
depicted by the point spread function (PSF), resulting in a pixel-wise defocus
map.

The defocus blur can be modeled by

$I_{b}=K*I_{c}+N$ (1)

where $I_{b}$ is the blurry image, $I_{c}$ is the clean image, $K$ is the blur
kernel and $N$ is the additive noise [3]. Thus the natural idea for defocus
deblurring is to follow two steps to address this problem, i.e., first to
estimate the defocus map indicating the level of defocus blur, then apply non-
blind deconvolution[29, 38, 11]. These classical methods can achieve
satisfying results under the low or medium level defocus blur but it is hard
for these methods to achieve sharp results under high level blur. Some image
priors are utilized to improve the performance but these priors works well
under some particular scenes but may fail to cover the real-world scenes. What
is more, the classical methods need iterative optimization to achieve sharp
results, which is time-consuming.

In the era of deep learning, many CNN-based based approaches are proposed for
deblurring problem. Nah et al. [21] propose multi-scale structure for dynamic
scene deblurring and achieve good visual results. Kupyn et al. [14, 15]
utilize the generative adversarial networks (GANs) [8] for the deblurring. Cho
et al. [5] propose a multi-input multi-output U-net (MIMO-UNet) to improve the
performance while reduce the computational cost. Zamir et al. [35] propose a
multi-stage and multi-scale architecture for motion blur removal. And Chen et
al. [4] propose a baseline model that achieves better performance and lower
inference time compared with previous methods. But the above methods focus on
the motion blur and therefore may be unsuitable for the defocus deblurring.

Recently, there are also some CNN-based methods proposed for defoucs blurring
[1, 17, 28]. Abuolaim et al. [1] use the UNet structure to recover the sharp
image in an end-to-end manner. Lee et al. [17] propose an iterative filter
adaptive module to handle spatially-varying and large defocus blur and a
training scheme based on defocus disparity estimation and re-blurring to boost
the deblurring quality. Son et al. [28] propose to simulate inverse kernels by
kernel sharing parallel atrous convolution block.

However, there still is a CNN-based method that follows first defocus map
estimation and then non-blind deblurring [20]. Compared with previous CNN-
based methods, the two-steps CNN-based methods can achieve higher performance
since the bind deblurring problem is much more complicated than the non-blind
deblurring problem, while the two steps CNN-based methods can utilize the
important information from the defocus map. After the estimation of the
defocus map using convolutional neural network, Ma et al. [20] concatenates
the defocus map and the defocus image as input of the neural network to
achieve all sharp images. We argue that direct concatenation can not fully use
the information in defocus map, which represents the blur level.

Considering the spatial variant property of the defocus blur and the blur
level indicated in the defocus map, we employ the defocus map as conditional
guidance to adjust the features from the input blurring images instead of
simple concatenation. Then we propose a simple but effective network with
spatial modulation based on the defocus map. To achieve this, we design a
network consisting of three sub-networks, including the defocus map estimation
network, a condition network that encodes the defocus map into condition
features and the defocus deblurring network that performs spatially dynamic
modulation based on the condition features. And the spatially dynamic
modulation is achieved by an affine transform function to adjust the features
from the input blurry images. Experimental results show that our method can
achieve better quantitative and qualitative evaluation performance than the
existing state-of-the-art methods on the commonly used public test datasets.

##  2 Related Work

In this section, we briefly introduce the related works, including defocus map
estimation, non-blind deblurring, and single image defocus deblurring.

###  2.1 Defocus Map Estimation

There are different approaches to defocus map estimation, which can be roughly
divided into three categories: edge-based, region-based, and CNN-based
methods.

The basic idea of the edge-based methods is to compute a sparse defocus map
only at the edges of the images and then propagate the map to the whole image
[24, 7, 2, 38, 19]. In [24], the input image is re-blurred by the Gaussian
kernels, and then the defocus amount along the edges is computed by the rate
of the gradients of the re-blurred image. Similar to [24], Elder and Zucker
[7] propose a method that simultaneously detects image edges and estimates
blur. These two studies [24, 7] only focus on the sparse defocus map
estimation. Liu et al. [19] propose a two-parameter model to improve the
performance of the defocus map estimation on the edges. Bae and Durand [2] try
to estimate the defocus map on the whole image. After the sparse defocus map
estimation, their method uses a bilateral filter to remove outliers. It then
uses a colorization-scheme-based interpolation method to achieve the full
defocus map. Zhuo and Sim [38] propose using alpha Laplacian Matting to
propagate the sparse defocus map to the whole image. These methods suffer from
inaccurate defocus map estimation for the image areas far from the edges. In
[11], a connected edge filter is proposed to smooth the initial sparse blur
map based on pixel connectivity within detected edge contours. Then a fast-
guided filter is used to propagate the sparse blur map through the whole
image.

Region-based methods directly estimate the defocus amount from the local
patches centered at the current pixel [30, 27]. For each local patch in the
image, Trouvé et al. [30] use a maximum likelihood method to select the local
blur from a set of PSF candidates. In [6], a machine learning approach based
on the regression tree fields is used to train a model able to regress a
coherent defocus blur map of the image, labeling each pixel by the scale of a
defocus point spread function. Shi et al. [27] propose a method based on
dictionary learning using sharp and slightly blurred patches.

Recently, CNN-based methods have been proposed for defocus map estimation. Yan
and Shao [34] propose a method that first classifies the blur type and then
estimates the blur parameter using a general regression neural network. Zhao
et al. [37] propose a method that detects defocus blur by using a bottom-top-
bottom fully convolutional network. Since defocus blur detection only
classifies each pixel as blur pixel and the non-blur pixel while the defocus
map estimation needs to estimate the blue level of each pixel so the defocus
blur detection can be considered as the loose formulation of the defocus map
estimation. Lee et al. [16] propose an end-to-end CNN-based method (DME-Net)
for spatially varying defocus map estimation, for which they also created a
synthetic dataset. They employ a domain adaptation method to address the gap
between real and synthetic datasets. Theirs is the first truly CNN-based
defocus map estimation method. Similar to [16], Ma et al., [20] train their
model in an end-to-end manner for defocus map estimation. These methods suffer
from the lack of sufficient real data.

###  2.2 Non-Blind Defocus Deblurring

Non-blind defocus deblurring, which assumes either the defocus map or the blur
kernel to be known, is an ill-posed problem since some information will
inevitably be lost during the blurring process. Most classical methods impose
some image priors to regularize the solution, for example, patch-based prior
[39], hyper-Laplacian prior [13], and local color prior [10]. These methods
usually need computationally expensive iterative optimization. Recently,
researchers have shown that CNN-based methods will be better than these
classical approaches in terms of accuracy and efficiency [26, 33, 36].
However, most existing methods, such as Schuler et al. [26] and Xu et al.
[33], need to be trained for each specific blur kernel; when we encounter
images with unseen kernels, we need to retrain the networks.

###  2.3 Single Image Defocus Deblurring

Many studies have been conducted for single-image defocus deblurring.
Conventional methods typically decompose the problem into two steps. The first
step is to estimate the defocus map of an input image, which indicates the
blur level for each pixel [29, 38, 11]. The second step is the non-blind
deconvolution [13, 18], where the defocus map estimated in the first step is
used. Most of the methods employing this two-step approach focus on improving
the accuracy of the first step, i.e., the defocus map estimation, since the
small error on the defocus map will significantly deteriorate the final
deblurring performance [11, 22, 27, 6].

Another approach is to train a network to directly predict the deblurred image
from a blurry input image in an end-to-end fashion. Abuolaim et al. [1] use a
network with the UNet structure to do this. Lee et al. [17] propose an
iterative filter adaptive module to handle spatially-varying and large defocus
blur. They also propose a training scheme based on defocus disparity
estimation and re-blurring to boost deblurring quality. Son et al. [28]
propose to simulate inverse kernels by kernel sharing parallel atrous
convolution block. However, these methods based on direct prediction depend
too much on the training data, and we believe their performance is suboptimal.

Figure 1: Architecture of the proposed network.

##  3 Proposed Method

We propose a network to remove spatial variant defocus blur from a single
blurry image. Inspired by the previous deblurring methods that decompose the
blind deblurring task into defocus map estimation task and non-blind
deblurring [29, 38, 11, 20]. Our network also first estimates the defocus map
as an intermediate result. For CNN-based based methods, they directly use the
feed-forward networks to learn the mapping from the input blurring images,
while the previous CNN-based two steps method directly concatenates the
defocus map with the input blurring images as the input as the non-blind
deblurring network. The defocus map, i.e., knowing the blur level for each
pixel, is very important information for the non-blind deblurring. But
previous method utilizes the defocus map by simple concatenation. We think
this leads to suboptimal results.

Considering the spatial variant property of the defocus blur and the blur
level indicated in the defocus map, we employ the defocus map as conditional
guidance to adjust the features from the input blurring images instead of
simple concatenation. For better usage of the defocus map, we employ an affine
transform to adjust the features from the input blurring images for each
pixel. By considering the same blur level on regions in blurring image, we
further decompose the parameters for scaling in the affine transform into
spatial and channel dimensions to reduce the redundancy and improve the
capacity.

Figure 1 shows our network which consists of three sub-network: defocus map
estimation network, the condition network and the deblurring network.

Figure 2: Architecture of the conditioned res-block.

###  3.1 Defocus Map Estimation Network

The defocus map estimation network utilizes several residual blocks to ease
the training procedure and maximize the information flow, which takes the
blurry images as input and then estimates the defocus map.

###  3.2 Condition Network

The condition network only consists of three convolutional layers, which takes
the estimated defocus map as input and maps it into the feature space as
conditions that are afterward used to modulate the intermediate features in
the defocus deblurring network. The key to reconstructing the all-sharp images
is to recover the missing details in the out-of-focus regions in the input
blurrying images. Different areas in one image have different contents and
blur levels. Further, different images also have different contents and blur
levels. Therefore, it is necessary to deal with input images with location-
specific and image-specific operations. However, the convolutional layer is
spatial invariant. Inspired by spatial feature transform (SFT) [31], we
introduce a network with SFT to perform a spatial variant adjustment. Here the
condition network is to generate the conditions for SFT from the defocus map.

Figure 3: Architecture of the feature transform.

###  3.3 Defocus Deblurring Network

Defocus deblurring network takes not only the blurring image but also the
conditions from the condition network as input. There are some conditioned
residual blocks in the defocus deblurring network and the Figure 2 shows the
details of conditioned residual blocks. The key component in conditioned
residual blocks is the SFT layer. The structure of SFT layer is shown in
Figure 3. The SFT learns a mapping function that generates the paired
modulation parameters $\gamma$ and $\beta$ based on the defocus map as prior.
The learned parameters adaptively adjust the output by an affine transform for
each pixel to the intermediate features on the defocus deblurring network.
Specifically, the SFT layer can be described as,

$SFT(F)=\gamma\odot F+\beta$ (2)

$F\in\mathbb{R}^{C\times H\times W}$ is the intermediate features in defocus
deblurring network and $\gamma\in\mathbb{R}^{C\times H\times W}$ and
$\beta\in\mathbb{R}^{C\times H\times W}$ are the parameters for modulation.
$\odot$ is the element-wise multiplication.

Figure 4: Architecture of the decomposition transform.

Further, as shown in Figure 1, there are some regions with the same value in
defocus map causing the redundancy on the conditions. Inspired by channel
attention [9] and spatial attention [32], we perform decomposition on the
generation for $\gamma$ in the channel dimension $\mathbb{R}^{C\times 1\times
1}$ and spatial dimension $\mathbb{R}^{1\times H\times W}$ as shown in Figure
4. It should be noted that we do not apply the decomposition transform to the
generation of $\beta$ since there are differences for each pixel value, even
though their blur levels are the same. Thus, the $\beta$ without decomposition
transform can provide the detailed compensation for the deblurring network. By
using this modulation strategy, our method can better use the important
information in defocus map.

###  3.4 Loss Function

The proposed network consists of three sub-network: defocus map estimation
network, condition network and the deblurring network. Following the previous
work [20], we use the $L_{1}$ norm loss and $L_{2}$ norm loss for training.
Specifically, for defocus map estimation, the loss is computed as follows,

$L_{dme}=||DM_{e}-DM_{gt}||_{1}$ (3)

where $DM_{e}$ is the estimated defocus map and the $DM_{gt}$ is the ground
truth for the defocus map.

While for the defocus deblurring, we exploit two losses: $L_{db}$ and
$L_{wd}$. $L_{db}$ is defined as

$L_{df}=||I_{df}-I_{gt}||_{1}$ (4)

while $L_{wd}$ is defined as

$L_{wd}=||W_{dm}*(I_{df}-I_{gt})||_{2}$ (5)

where $I_{df}$ is the deblurring result of the deblurring network and the
$I_{gt}$ is the all-sharp ground truth image. $W_{dm}$ is defined as

$W_{dm}=\frac{DM_{gt}}{mean(DM_{gt})}$ (6)

Similar to [20], we train our network for three stages. In stage one and stage
two, the two networks are jointly trained for 400 and 200 epochs,
respectively. For stage one, the ground truth defocus map is set as the input
of the defocus deblurring network, which avoids divergence caused by random
output of the defocus map estimation network. For stage two, the output of the
defocus map estimation network is set as the input of the defocus deblurring
network to jointly training the whole network. For both stage one and stage
two, the loss used for training is

$Loss_{1}=\lambda_{1}\times L_{dme}+\lambda_{2}\times L_{df}$ (7)

In stage three, we use the following loss to finetune the network for another
400 epochs,

$Loss_{2}=\lambda_{2}\times L_{df}+\lambda_{3}\times L_{wd}$ (8)

In this work, we set the weights for loss function as $\lambda_{1}=0.2$,
$\lambda_{2}=0.9$ and $\lambda_{3}=0.1$

##  4 Experiments

###  4.1 Experimental Settings

####  4.1.1 Training Data

To train our network, we use the defocus image deblurring dataset in [20]
which consists of both the defocus map ground truth and all-sharp image ground
truth.

####  4.1.2 Test Data

Followed the previous work [20], we evaluate the proposed method on the
Realistic dataset [6] and the DED test dataset [20].

####  4.1.3 Implementation Details

For training, we use the Adam optimizer [12] with $\beta_{1}=0.9$,
$\beta_{2}=0.999$ with initial learning rate $1\times 10^{-4}$ and set the
batch size to 16. And the number of epochs for each stage is mentioned above.
The input blurring image, the defocus map ground truth and the corresponding
all-sharp ground truth are randomly cropped into patches with 256$\times$256\.
We also apply other data augmentation like random rotation and flipping to
avoid over-fitting. We implement our model by PyTorch [23] platform and train
our model on NVIDIA GeForce RTX 2080 GPUs.

###  4.2 Experimental Resutls

We first evaluate the performance of defocus map estimation. We compare the
proposed method with the methods of Karaali and Jung [11] and the recent deep
learning-based DME-Net [16] and DID-ANet [20] for defoucs map estimation. Mean
absolute error (MAE) and mean squared error (MSE) are used as the evaluation
metrics. Table 1 shows the quantitative results on Realistic dataset. The
proposed method achieves almost the same performance with DID-ANet [20] since
we adopt the same network with DID-ANet [20] and is comparable with DMENet
[16] on Realistic dataset.

We compare the proposed method with the DME-Net [16] that estimates the
defocus map by CNN and achieves deblurring by conventional deconvolution [13],
two CNN-based methods for defocus deblurring (DPDDNet [1] and IFAN [17]),
three CNN-based state-of-arts methods (MPRNet [35], MIMO-UNet [5] and NAFT
[4]) for motion deblurring and two-steps CNN-based method (DID-ANet [20]). All
the methods are fine-tuned on the training set of DED dataset. And the
training settings are set separately based on the original paper. The epochs
for fine-tuning are set to 600 to ensure the models are convergent.

Table 1: Quantitative Results for Defocus Map Estimation Methods | MAE, Realistic | MSE, Realistic
---|---|---
Karraali [11] | 0.3102 | 0.1245
DMENet [16] | 0.1191 | 0.0242
DID-ANet [20] | 0.1331 | 0.0312
Ours | 0.1303 | 0.0299

We use the Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM)
for evaluation metrics. The quantitative results are shown in Table 2 where
the best results are in bold. For Realistic dataset, the proposed method
achieves better performance than the other methods in terms of PSNR and SSIM.
The CNN-based methods for motion blur and defocus blur do not achieve good
performance compared with the two-steps methods (e.g., DID-ANet [20]), since
they do not utilize the information from the defocus map which is essential
for defocus image blurring. Although The proposed method achieves almost the
same performance with DID-ANet [20] in the defoucs map estimation, it still
achieves much higher performance than DID-ANet [20] with more than 0.6db
improvement, which shows the effectiveness of the proposed method. And also
compared DME-Net [16] with conventional deconvolution [13], the CNN-based
methods (e.g., DID-ANet [20]) can achieve much better performance.

Table 2: Quantitative Results for Defocus Image Deblurring | PSNR, Realistic | SSIM, Realistic
---|---|---
DMENet [16] | 24.0397 | 0.7180
DPDDNet [1] | 24.7232 | 0.7640
IFAN [17] | 24.9261 | 0.8211
MPRNet [35] | 25.2738 | 0.7869
MIMO-UNet [5] | 25.3430 | 0.7962
NAFT [4] | 24.3461 | 0.7484
DID-ANet [20] | 25.9491 | 0.8204
Ours | 26.5535 | 0.8465
Figure 5: Visual comparison of defocus image deblurring on realistic (08).
Figure 6: Visual comparison of defocus image deblurring on realistic (09).
Figure 7: Visual comparison of defocus image deblurring on DED dataset (604).
Figure 8: Visual comparison of defocus image deblurring on DED dataset (794).
Figure 9: Visual comparison of defocus image deblurring on DED dataset (971).

Several visualization results are shown in Figure 5-9. Among them, Figure 5
and 6 are from Realistic test set while Figure 7-9 are from DED test set.
Since the image size for the DED test set is large, we crop the images from
DED test set and zoom them for a better view.

The images from Figure 5 and 6 are almost the same contents but have different
focal plane. The image (08) in the Figure 5 focuses on the tree then the
background is blurry. As shown in Figure 5, the structures on the road and the
fence are clearer in our result. While in Figure 6, ours results have clearer
texture compared with other methods. As shown in Figure 7, our method can
achieve the sharper edge on the wall with fewer artifacts. The light region in
input image shown in Figure 8 is blurry, while our method can successfully
recover the sharp shape of the light region. The hand and the watch are more
realistic and clearer in our result.

In contrast, the previous works cannot address the defocus blur very well. The
DME-Net [16] estimates the defocus map and then uses the deconvolution
deblurring [13] to achieve defocus deblurring. Their results look blurry
compared with ours. On the other hand, the CNN-based based methods (DPDDNet
[1] and IFAN [17]) also achieve blurry results though the model are finetuned
on the DED dataset. For the state-of-the-art method in motion deblurring
methods, NAFT [4] cannot deal with the defocus blur very well. For the two-
steps methods, DID-ANet achieves better performance than the above methods.
However, the important information in the defocus map can not be fully
utilized by simple concatenation. On the contrary, the proposed method
exploits the information from the defocus map by the SFT with the
decomposition method, which leads to higher PSNR and SSIM than previous
methods, as shown in Table 2

Table 3: Ablation Study on Results for Defocus Image Deblurring Methods | PSNR, Realistic | SSIM, Realistic
---|---|---
baseline-S1 | 25.6443 | 0.8077
+SFT-S1 | 26.1286 | 0.8294
+SFT-Dec-S1 | 26.0837 | 0.8289
+SFT-FDec-S1 | 25.9939 | 0.8253
baseline-S2 | 25.7936 | 0.8118
+SFT-S2 | 26.2533 | 0.8307
+SFT-Dec-S2 | 26.4275 | 0.8310
+SFT-FDec-S2 | 26.1735 | 0.8307
baseline-S3 | 25.9638 | 0.8191
+SFT-S3 | 26.3973 | 0.8379
+SFT-Dec-S3 | 26.5535 | 0.8465
+SFT-FDec-S3 | 26.4327 | 0.8400
+SFT-Dec-end | 26.1143 | 0.8259

###  4.3 Ablation Study

We demonstrate the effectiveness of each component in the proposed method on
the Realistic dataset. The results are shown in Table 3. For the baseline
model, we remove the SFT with the decomposition method. For the
”baseline+SFT”, it means that we use the original SFT in the baseline. For the
”baseline+SFT-Dec”, it means that we use the SFT with the decomposition on
$\gamma$ into the baseline. For the ”baseline+SFT-FDec”, it means that we use
the SFT with the decomposition on both $\gamma$ and $\beta$ into the baseline.
S1, S2 and S3 indicate the different training stages, respectively while
”-end” means the model is trained in an end-to-end manner with the supervision
of the defocus map and the all sharp ground truth.

As shown in Table 3, for all these methods, the performance increases after
each stage training. While the final model ”baseline+SFT-Dec-s3” achieves the
best performance compared with other ablated networks. At the same time, if
the final model is trained in an end-to-end manner, the performance decreases
by a large margin. By introducing the SFT, the performance increases about
0.4db in PSNR and 0.02 in SSIM compared with the baseline model. While
incorporating the decomposition method, the performance increases by about
0.6db in PSNR and 0.03 in SSIM compared with the baseline model. However, if
we also apply the decomposition method to the $\beta$, the performance drops
about 0.1db in PSNR. This is because the $\beta$ provides the detailed
information for modulation but the decomposition method will remove these
details to some extent.

##  5 Conclusion

In this paper, we propose a new method for defocus image deblurring by first
defocus map estimation and then the defocus deblurring direction. The defocus
map is important information for defocus image deblurring since it contains
the blur level for each pixel. To remove the spatial variant blur, we
introduce the spatial feature transform and the decomposition technique to
perform spatial modulation based on the defocus map. The experimental results
have validated the effectiveness of the proposed approach.

## References

  * [1] Abdullah Abuolaim and Michael S Brown.  Defocus deblurring using dual-pixel data.  In European Conference on Computer Vision, pages 111–126. Springer, 2020.
  * [2] Soonmin Bae and Frédo Durand.  Defocus magnification.  In Computer graphics forum, volume 26, pages 571–579. Wiley Online Library, 2007.
  * [3] Patrizio Campisi and Karen Egiazarian.  Blind image deconvolution: theory and applications.  CRC press, 2017.
  * [4] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun.  Simple baselines for image restoration.  arXiv preprint arXiv:2204.04676, 2022.
  * [5] Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko.  Rethinking coarse-to-fine approach in single image deblurring.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4641–4650, 2021.
  * [6] Laurent D’Andrès, Jordi Salvador, Axel Kochale, and Sabine Süsstrunk.  Non-parametric blur map regression for depth of field extension.  IEEE Transactions on Image Processing, 25(4):1660–1673, 2016.
  * [7] James H Elder and Steven W Zucker.  Local scale control for edge detection and blur estimation.  IEEE Transactions on pattern analysis and machine intelligence, 20(7):699–716, 1998.
  * [8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [9] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7132–7141, 2018.
  * [10] Neel Joshi, C Lawrence Zitnick, Richard Szeliski, and David J Kriegman.  Image deblurring and denoising using color priors.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1550–1557. IEEE, 2009.
  * [11] Ali Karaali and Claudio Rosito Jung.  Edge-based defocus blur estimation with adaptive scale selection.  IEEE Transactions on Image Processing, 27(3):1126–1137, 2017.
  * [12] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
  * [13] Dilip Krishnan and Rob Fergus.  Fast image deconvolution using hyper-laplacian priors.  Advances in neural information processing systems, 22, 2009.
  * [14] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas.  Deblurgan: Blind motion deblurring using conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8183–8192, 2018.
  * [15] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang.  Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8878–8887, 2019.
  * [16] Junyong Lee, Sungkil Lee, Sunghyun Cho, and Seungyong Lee.  Deep defocus map estimation using domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12222–12230, 2019.
  * [17] Junyong Lee, Hyeongseok Son, Jaesung Rim, Sunghyun Cho, and Seungyong Lee.  Iterative filter adaptive network for single image defocus deblurring.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2034–2042, 2021.
  * [18] Anat Levin, Rob Fergus, Frédo Durand, and William T Freeman.  Image and depth from a conventional camera with a coded aperture.  ACM transactions on graphics (TOG), 26(3):70–es, 2007.
  * [19] Shaojun Liu, Fei Zhou, and Qingmin Liao.  Defocus map estimation from a single image based on two-parameter defocus model.  IEEE Transactions on Image Processing, 25(12):5943–5956, 2016.
  * [20] Haoyu Ma, Shaojun Liu, Qingmin Liao, Juncheng Zhang, and Jing-Hao Xue.  Defocus image deblurring network with defocus map estimation as auxiliary task.  IEEE Transactions on Image Processing, 31:216–226, 2021.
  * [21] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee.  Deep multi-scale convolutional neural network for dynamic scene deblurring.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3883–3891, 2017.
  * [22] Jinsun Park, Yu-Wing Tai, Donghyeon Cho, and In So Kweon.  A unified approach of multi-scale deep and hand-crafted features for defocus estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1736–1745, 2017.
  * [23] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [24] Alex Paul Pentland.  A new sense for depth of field.  IEEE transactions on pattern analysis and machine intelligence, (4):523–531, 1987.
  * [25] Michael Potmesil and Indranil Chakravarty.  Synthetic image generation with a lens and aperture camera model.  ACM Transactions on Graphics (TOG), 1(2):85–108, 1982.
  * [26] Christian J Schuler, Harold Christopher Burger, Stefan Harmeling, and Bernhard Scholkopf.  A machine learning approach for non-blind image deconvolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1067–1074, 2013.
  * [27] Jianping Shi, Li Xu, and Jiaya Jia.  Just noticeable defocus blur detection and estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 657–665, 2015.
  * [28] Hyeongseok Son, Junyong Lee, Sunghyun Cho, and Seungyong Lee.  Single image defocus deblurring using kernel-sharing parallel atrous convolutions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2642–2650, 2021.
  * [29] Yu-Wing Tai and Michael S Brown.  Single image defocus map estimation using local contrast prior.  In 2009 16th IEEE International Conference on Image Processing (ICIP), pages 1797–1800. IEEE, 2009.
  * [30] Pauline Trouvé, Frédéric Champagnat, Guy Le Besnerais, and Jérôme Idier.  Single image local blur identification.  In 2011 18th IEEE International Conference on Image Processing, pages 613–616. IEEE, 2011.
  * [31] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Recovering realistic texture in image super-resolution by deep spatial feature transform.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 606–615, 2018.
  * [32] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [33] Li Xu, Jimmy S Ren, Ce Liu, and Jiaya Jia.  Deep convolutional neural network for image deconvolution.  Advances in neural information processing systems, 27, 2014.
  * [34] Ruomei Yan and Ling Shao.  Blind image blur estimation via deep learning.  IEEE Transactions on Image Processing, 25(4):1910–1921, 2016.
  * [35] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  Multi-stage progressive image restoration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14821–14831, 2021.
  * [36] Jiawei Zhang, Jinshan Pan, Wei-Sheng Lai, Rynson WH Lau, and Ming-Hsuan Yang.  Learning fully convolutional networks for iterative non-blind deconvolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3817–3825, 2017.
  * [37] Wenda Zhao, Fan Zhao, Dong Wang, and Huchuan Lu.  Defocus blur detection via multi-stream bottom-top-bottom fully convolutional network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3080–3088, 2018.
  * [38] Shaojie Zhuo and Terence Sim.  Defocus map estimation from a single image.  Pattern Recognition, 44(9):1852–1858, 2011.
  * [39] Daniel Zoran and Yair Weiss.  From learning models of natural image patches to whole image restoration.  In 2011 International Conference on Computer Vision, pages 479–486. IEEE, 2011.