# Real time dense anomaly detection
by learning on synthetic negative data

Anja Delić, Matej Grcić, Siniša Šegvić Master research seminar at University
of Zagreb Faculty of Electrical Engineering and Computing

###### Abstract

Most approaches to dense anomaly detection rely on generative modeling or on
discriminative methods that train with negative data. We consider a recent
hybrid method that optimizes the same shared representation according to
cross-entropy of the discriminative predictions, and negative log likelihood
of the predicted energy-based density. We extend that work with a jointly
trained generative flow that samples synthetic negatives at the border of the
inlier distribution. The proposed extension provides potential to learn the
hybrid method without real negative data. Our experiments analyse the impact
of training with synthetic negative data and validate contribution of the
energy-based density during training and evaluation.

###### Index Terms:

dense anomaly detection, simultaneous semantic segmentation and outlier
detection, energy-based density estimation, normalizing flow

##  I Introduction

Existing deep learning methods solve the computer vision task of semantic
segmentation with high precision. However, models are typically evaluated on
closed-world datasets with a fixed taxonomy and their behaviour in real-world
applications can be unpredictable. We therefore study methods that detect
objects that deviate from the training distribution by assigning a pixel vise
anomaly score. This task has also been known as dense anomaly, out-of-
distribution (OoD) or novelty detection. Current state of the art in this
field relies on training on mixed content images obtained by pasting negative
data atop inlier images. However, anomaly detection methods that utilize real
negative datasets can give rise to over-optimistic performance metrics because
of the bias introduced by the possibility that some pasted negatives can be
related to anomalies in the test dataset. We try to avoid this bias by
extending discriminative [1] [2] and hybrid [3] methods that utilize real
negatives with the capability to learn with generated synthetic negative data
[4, 5]. This work analyzes methods that were proposed in [6]. We extend that
work by providing real-time experiments on the StreetHazards dataset [7].

##  II Related work

Anomaly detection methods can be discriminative or generative. Most
discriminative methods require training with auxiliary negative data that is
pasted over regular training images[1, 2]. Pixel-wise OoD predictions can be
produced by a separate OoD head sharing features with the standard semantic
segmentation head or by measuring uncertainty of the predictive distributions.
Real negative data can be replaced with synthetic negative patches obtained by
sampling a jointly trained generative model[8, 9, 10, 4, 5].

Generative methods detect outliers according to the estimated density of the
generative distribution of the inlier training data. The data likelihood has
been estimated by leveraging an energy-based model[3] or a normalizing
flow[11, 12]. However, estimating the data likelihood with a generative model
can be suboptimal because the generative model may assign higher likelihood to
outliers than inliers[13, 14]. On the other hand, discriminative methods
highly depend on the choice of negative data. A recent hybrid method known as
DenseHybrid [3] aims at achieving a synergy between these two approaches. It
builds on a recent image-wide approach [15]. DenseHybrid[3] combines
discriminative and generative learning by reinterpreting discriminative logits
as a logarithm of unnormalized joint distribution of data input and labels.

##  III Learning with synthetic negatives

We extend the training procedure proposed od DenseHybrid to jointly train a
normalizing flow (NF) that generates samples at the border of the inlier
manifold [8, 5].

For the segmentation model we use the original DenseHybrid loss [3] that
combines standard segmentation loss $L_{cls}$, outlier head loss $L_{d}$ and
energy loss $L_{x}$:

$\displaystyle L_{\mathrm{seg}}(\theta,\gamma)=$
$\displaystyle-\mathbb{E}_{\mathbf{x},\mathbf{y}\in D_{\text{in}}}[\ln
P(\mathbf{y}\mid\mathbf{x})+\ln P(d_{\text{in}}\mid\mathbf{x})]$ (1)
$\displaystyle-\beta\cdot\mathbb{E}_{\mathbf{x}\in
D_{\text{out}}}[\ln(P(d_{\text{out}}\mid\mathbf{x}))-\ln\hat{p}(\mathbf{x})].$

We try two loss functions for training the NF. The first approach corresponds
to the Jensen-Shannon divergence between $P(y|\mathbf{x})$ and the uniform
distribution to the negative log-likelihood of the cut out inlier patches:

$L_{\mathrm{flow}}(\psi)=L_{\mathrm{mle}}(\psi)+\beta\cdot
L_{\mathrm{jsd}}(\psi,\theta).$ (2)

The first term attracts the generative distribution towards inliers while the
second term pushes it away from the inliers. Thus, the normalizing flow
generates data at the boundary of the training distribution. We stop the
gradients from $L_{d}$ and $L_{x}$ from backpropagating to the NF so that only
gradients from (2) impact the NF. We refer to this method as NF-Hybrid-JS.

In the second approach, we remove the JS divergence from the NF training
objective (2) and let the gradients from $L_{d}$ and $L_{x}$ to propagate
through the segmentation model back to the NF. We refer to this method as NF-
Hybrid-LdLx.

$L_{\mathrm{mle}}(\psi)=-\mathbb{E}_{\mathbf{\operatorname{crop}(x)}\in
D_{\text{in}}}[\ln p_{\psi}(\operatorname{crop}(\mathbf{x}))]$ (3)

TABLE I: Anomaly detection, closed-set and open-set performance on StreetHazards [7] Method | Aux. data | Anomaly score | AP | $\text{FPR}_{95}$ | AUROC | closed $\overline{\text{IoU}}$ | open $\overline{\text{IoU}}$
---|---|---|---|---|---|---|---
DenseHybrid [3] | ✓ | DH | 23.81 | 18.76 | 93.69 | 57.52 | 37.59
OP$\times$MS | 17.08 | 21.94 | 92.13 | 57.52 | 37.39
NFlowJS [4] | ✗ | JSD | 12.12 | 25.64 | 90.81 | 60.34 | 42.94
NF-Hybrid-JS | ✗ | DH | 16.61 | 21.76 | 91.83 | 56.74 | 35.72
NF-Hybrid-LdLx | ✗ | DH | 16.67 | 19.26 | 92.39 | 56.76 | 37.88
OP$\times$MS | 14.35 | 23.03 | 90.96 | 56.76 | 34.57
NF-Hybrid-Ld | ✗ | DH | 15.62 | 22.78 | 91.31 | 56.79 | 35.07
OODHead | ✓ | DH | 24.63 | 16.45 | 94.51 | 58.51 | 44.29
OP | 22.46 | 31.26 | 91.71 | 58.51 | 32.78
| OP$\times$MS | 22.52 | 19.06 | 93.54 | 58.51 | 40.13
NF-OODHead | ✗ | DH | 12.13 | 21.73 | 92.03 | 60.11 | 41.76
OP | 3.83 | 69.12 | 70.03 | 60.11 | 9.8
OP$\times$MS | 8.87 | 23.99 | 90.55 | 60.11 | 36.92

We validate the influence of the energy loss in (1) by removing it from the
segmentation model training objective. That results in a discriminative
segmentation model with an additional outlier head similar to approaches
presented in [1] [2]. We present experiments with real (method OODHead) and
synthetic negatives (NF-OODHead). When using synthetic negatives, the
normalizing flow is trained by minimizing negative log-likelihood (3).

##  IV Anomaly Score

During inference the anomaly detector recognizes anomalies by thresholding a
predefined anomaly score. A simple approach is to define the anomaly score as
a discriminative probability that the sample is an outlier[1]:

$s_{OP}(\mathbf{x})=P(d_{\text{out}}=1\mid\mathbf{x}).$ (4)

Performance on small anomalous objects can be improved by multiplying the
outlier probability with max-softmax[2]:

$s_{OP\times
MS}(\mathbf{x})=P(d_{\text{out}}=1\mid\mathbf{x})\cdot(1-\operatorname{max}_{c}(P(y_{c}|\mathbf{x}))).$
(5)

The DenseHybrid anomaly score is defined as the ratio of dataset posterior and
data likelihood[3]:

$s_{DH}(\mathbf{x})=\ln\frac{P(d_{\text{out}}\mid\mathbf{x})}{p(\mathbf{x})}.$
(6)

Anomaly score can also be defined as JS divergence between $P(y|\mathbf{x})$
and the uniform distribution[4]. In later text we refer to these methods as
outlier probability (OP), outlier probability multiplied by maximum softmax
(OP$\times$MS), DenseHybrid score (DH) and JS divergence score (JSD).

##  V Experiments

We evaluate methods described in Section III on StreetHazards dataset [7]. In
all our experiments, we use SwiftNet-RN18 [16] as the segmentation model
because it shows good semantic segmentation performance with real-time
inference. We create mixed content images by pasting negatives atop inlier
crops obtained by randomly jittering in range [0.5, 2], random horizontal
flipping and taking a square crop of size 768. In experiments that utilize
real negative data, we paste instances from ADE20k dataset [17]. We use
DenseFlow-25-6[18] for generating synthetic negatives. The normalizing flow
generates rectangular patches with arbitrary spatial dimensions. In
experiments we use square dimensions between 16 and 216 pixels. The NF
pretrained on Vistas is trained with the Adamax optimizer with learning rate
$10^{-6}$. The joint training is a two step process. For initialization we use
weights pretrained on ImageNet. We first train our model on StreetHazards
train for 80 epochs as a standard segmentation model. Then we fine tune for
additional 40 epochs on mixed content images following the anomaly aware
procedures as described in Section III. We use Adam optimizer with initial
learning rate $10^{-5}$ that is decayed through a cosine schedule to
$10^{-7}$. We set the loss hyperparameter $\beta$ for $L_{x}$ and $L_{JSD}$ to
0.03 and for $L_{d}$ 0.3. In OODHead and NF-OODHead experiments we use
$\beta=1$. We train with batch size 16. We evaluate our experiments with
different anomaly scores as described in Section IV. We use temperature
scaling with temperature $T=2$ for OP, OP$\times$MS and JSD scores.

Table I presents the experimental performance. To make our results comparable
with previous work, we reproduce experiments [3] and [4], DenseHybrid and
NFlowJS respectively, following the original setup but with SwiftNet-RN18 as
the semantic segmentation model. We evaluate DenseHybrid with the proposed DH
score and with OP$\times$MS to be comparable with our experiments. We see a
performance drop when using OP$\times$MS.

The NF-Hybrid-JS approach combines DenseHybrid and NFlowJS. The difference to
NFlowJS is an improvement in training the segmentation model by adding the
energy term and an outlier head in DenseHybrid manner. We use the DH anomaly
score in evaluation. NF-Hybrid-JS outperforms NFlowJS but not DenseHybrid.

The NF-Hybrid-LdLx approach achieves the best average precision score in
comparison to our other experiments that use synthetic negatives and NFlowJS.
We stop the gradients from the energy loss from propagating to the NF (method
NF-Hybrid-Ld) so that only the outlier head influences the NF but this leads
to worse performance. This suggests that the energy loss can be beneficial for
training the NF.

TABLE II: Computational overhead in frames per second of used anomaly detectors when inferring with RTX A4000 on StreetHazards test images Method | FPS
---|---
OH | 74.43
OP$\times$MS | 71.69
DH | 69.86
JSD | 52.41

OODHead experiment is an ablation of the DenseHybrid approach. A standard
approach for evaluating a two head model would be to use OP anomaly score. We
also evaluate OP$\times$MS and DH. OP$\times$MS outperforms OP and is best
seen in a big drop in FPR metric. DH score outperforms both OP$\times$MS and
OP. It is interesting that OODHead outperforms DenseHybrid which could be
caused by insufficient capacity of the segmentation model. When using DH score
we see that we can improve the discriminative anomaly aware model by using the
data likelihood only in evaluation.

We replace the auxiliary negatives with synthetic negatives generated by a
normalizing flow (NF-OODHead) and evaluate using OP, OP$\times$MS and DH
anomaly scores. We see that this method highly depends on the choice of
anomaly score. There is a noticeable performance gap between OP and
OP$\times$MS scores.

We measure open-set performance using the open-IoU metric proposed in [3]. If
we compare experiments that differ only in using the energy term in the loss
function, DenseHybrid and OODHead or also NF-Hybrid-JS and NFlowJS, we can
observe a drop in both open and closed IoU when learning in a hybrid manner.
This indicates that the segmentation model SwiftNetRN-18 does not have enough
capacity for hybrid learning.

Table II shows computational overhead for the open set recognition task over
the SwiftNetRN-18 segmentation model for anomaly detectors described in IV. We
measured the inference time with NVIDIA RTX A4000 on StreetHazards test images
of size 720 $\times$ 1280 pixels.

##  VI Conclusion

We extend a hybrid dense anomaly detection approach with a normalizing flow
model that generates synthetic negatives following methods that were proposed
in [6]. This allows to learn outlier-aware models without training on real
negative data. Our experiments show improvement of hybrid approach over other
methods that use synthetic negatives. However, we observed a performance drop
when training with the hybrid approach on real negative data which indicates
that the chosen semantic segmentation model SwiftNetRN-18 does not have
sufficient capacity for hybrid learning. We show that the normalizing flow can
generate meaningful anomalies when influenced by the energy loss and outlier
head. We evaluate our training approaches using different anomaly scores and
show advantage of the hybrid anomaly score even when the segmentation model is
trained in a discriminative manner. These findings should be validated on
larger models and more datasets. Interesting directions for future work
include reducing the gap between closed and open set performance and
considering different anomaly detection approaches.

## References

  * [1] P. Bevandić, I. Krešo, M. Oršić, and S. Šegvić, “Simultaneous semantic segmentation and outlier detection in presence of domain shift,” in 41st DAGM German Conference, DAGM GCPR. Springer, 2019.
  * [2] P. Bevandić, I. Krešo, M. Oršić, and S. Šegvić, “Dense open-set recognition based on training with noisy negative images,” Image and Vision Computing, 2022.
  * [3] M. Grcić, P. Bevandić, and S. Šegvić, “DenseHybrid: Hybrid Anomaly Detection for Dense Open-Set Recognition,” in European Conference on Computer Vision, ECCV 2022. Springer, 2022.
  * [4] M. Grcić, P. Bevandić, and S. Šegvić, “Dense anomaly detection by robust learning on synthetic negative data,” arxiv preprint arXiv:2112.12833, 2021.
  * [5] M. Grcić, P. Bevandić, and S. Šegvić, “Dense open-set recognition with synthetic outliers generated by real NVP,” in 16th International Joint Conference on Computer Vision,Imaging and Computer Graphics Theory and Applications, VISIGRAPP, 2021.
  * [6] M. Grcić, P. Bevandić, and S. Šegvić, “Hybrid Open-set Segmentation with Synthetic Negative Data,” arXiv preprint arXiv:2301.08555, 2023.
  * [7] D. Hendrycks, S. Basart, M. Mazeika, A. Zou, J. Kwon, M. Mostajabi, J. Steinhardt, and D. Song, “Scaling out-of-distribution detection for real-world settings,” ICML, 2022.
  * [8] K. Lee, H. Lee, K. Lee, and J. Shin, “Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples,” in International Conference on Learning Representations, 2018.
  * [9] L. Neal, M. Olson, X. Fern, W.-K. Wong, and F. Li, “Open set learning with counterfactual images,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [10] Z. Zhao, L. Cao, and K. Lin, “Revealing distributional vulnerability of explicit discriminators by implicit generators,” CoRR, 2021.
  * [11] H. Blum, P.-E. Sarlin, J. Nieto, R. Siegwart, and C. Cadena, “The fishyscapes benchmark: Measuring blind spots in semantic segmentation,” International Journal of Computer Vision, 2021.
  * [12] L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using real NVP,” in International Conference on Learning Representations, 2017\.
  * [13] Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan, “Do deep generative models know what they don’t know?,” in International Conference on Learning Representations, 2019.
  * [14] P. Kirichenko, P. Izmailov, and A. G. Wilson, “Why normalizing flows fail to detect out-of-distribution data,” in Advances in Neural Information Processing Systems, 2020.
  * [15] W. Grathwohl, K.-C. Wang, J.-H. Jacobsen, D. Duvenaud, M. Norouzi, and K. Swersky, “Your classifier is secretly an energy based model and you should treat it like one,” in International Conference on Learning Representations, 2020.
  * [16] M. Orsic, I. Kreso, P. Bevandic, and S. Segvic, “In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 12 607–12 616.
  * [17] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
  * [18] M. Grcić, I. Grubišić, and S. Šegvić, “Densely connected normalizing flows,” Advances in Neural Information Processing Systems, 2021.


# Leveraging Classic Deconvolution and Feature Extraction in Zero-Shot Image
Restoration

Tomáš Chobola<EMAIL_ADDRESS>Technical University of
Munich, Munich, Germany  Helmholtz AI, Helmholtz Munich - German Research
Center for Environmental Health, Neuherberg, Germany  Gesine Müller  Georg-
August-University Göttingen, Göttingen, Germany  Veit Dausmann  GEOMAR
Helmholtz Centre for Ocean Research Kiel, Kiel, Germany  Anton Theileis
GEOMAR Helmholtz Centre for Ocean Research Kiel, Kiel, Germany  Jan Taucher
GEOMAR Helmholtz Centre for Ocean Research Kiel, Kiel, Germany  Jan Huisken
Georg-August-University Göttingen, Göttingen, Germany  Tingying Peng
<EMAIL_ADDRESS>Helmholtz AI, Helmholtz Munich - German
Research Center for Environmental Health, Neuherberg, Germany

###### Abstract

Non-blind deconvolution aims to restore a sharp image from its blurred
counterpart given an obtained kernel. Existing deep neural architectures are
often built based on large datasets of sharp ground truth images and trained
with supervision. Sharp, high quality ground truth images, however, are not
always available, especially for biomedical applications. This severely
hampers the applicability of current approaches in practice. In this paper, we
propose a novel non-blind deconvolution method that leverages the power of
deep learning and classic iterative deconvolution algorithms. Our approach
combines a pre-trained network to extract deep features from the input image
with iterative Richardson-Lucy deconvolution steps. Subsequently, a zero-shot
optimisation process is employed to integrate the deconvolved features,
resulting in a high-quality reconstructed image. By performing the preliminary
reconstruction with the classic iterative deconvolution method, we can
effectively utilise a smaller network to produce the final image, thus
accelerating the reconstruction whilst reducing the demand for valuable
computational resources. Our method demonstrates significant improvements in
various real-world applications non-blind deconvolution tasks.

##  1 Introduction

Image deconvolution is a classic problem in computer vision and imaging
sciences. It aims to recover a sharp image $x$ out of a blurred and noisy
representation

$y=x*k+b,$ (1)

where $k$ is the blur kernel, $b$ is additive noise and $*$ denotes the
convolution operator. Traditionally, methods that solve this problem are split
into two steps: first, the blur kernel $k$ is obtained either through
estimation or imaging system calibration. Then, image restoration is performed
(i.e., non-blind deconvolution). Given the reduction or removal of high
frequencies caused by convolving an image with a blur kernel and unknown noise
caused by measurement errors, image deconvolution is essentially a challenging
ill-posed inverse problem.

Recently, multiple studies of deep neural networks designed for sharp image
recovery have been conducted [7, 20, 50, 18, 10, 33, 8]. While those methods
are able to achieve impressive performance on computer vision benchmarks [25,
22, 41, 29], the pre-trained models are not directly applicable to scientific
imaging given the arising domain gap, and therefore it decreases their
generalisation performance. Additionally, re-training or fine-tuning is often
not possible given the sparsity of ground truth sharp images in scientific
imaging fields such as medical or biological microscopy [1, 54, 17]. This
problem cannot be alleviated by using images from other scientific fields due
to the image diversity, which results in large domain shifts and gaps between
different types of microscopy images.

To address the limitations arising from insufficient training data, various
deep self-supervised techniques have been developed for deblurring and
denoising tasks, as evidenced in previous works [42, 13, 32, 3]. However,
these methods primarily cater to general image restoration tasks and do not
account for the distinctive features and complexities inherent in scientific
images. Scientific images often exhibit unique noise patterns, intricate
structures, and specialized imaging modalities, necessitating tailored
restoration approaches. Therefore, the development of specialized deep self-
supervised methods becomes imperative to effectively tackle restoration
challenges specific to scientific images. Moreover, it is important to
emphasize that existing methods often neglect the consideration of
computational costs and model size. In the context of scientific image
restoration, these aspects hold crucial importance due to limited
computational resources and the need for efficiently processing large
quantities of data. Therefore, the development of deep self-supervised methods
that also take into account computational efficiency and model compactness is
highly beneficial in real-life scientific applications.

Given the reasons stated above, in many scientific fields deconvolution is
still handled by classic algorithms [34, 23, 47] that are fast, but known to
be prone to degradation and noise amplification. To improve the image quality,
total variation (TV) has become the standard for noise reduction and
regularisation in image reconstruction, but in recent years its limitations
such as loss in textures, changes in smooth intensity or patchy artefacts have
been reported [2, 15, 21].

The core contributions of this paper can be summarised as follows:

  1. 1.

We propose CiDeR111https://github.com/ctom2/cider (Classic Deconvolution and
Feature Extraction for Zero-Shot Image Restoration), a self-supervised
approach for non-blind deconvolution embedding the Richardson-Lucy algorithm,
a classic method for image deconvolution, into a deep learning model.

  2. 2.

We integrate the continuity prior as a form of Hessian regularisation into the
loss function, reducing the noise in the resulting sharp image while
preserving the underlying image structures.

  3. 3.

We apply our proposed CiDeR to real microscopy images, addressing a variety of
image restoration tasks across different microscopy modalities, achieving
enhanced visual quality and improved image details.

Figure 1: The architecture of CiDeR, a novel non-blind deconvolution model
that integrates the Richardson-Lucy algorithm into a zero-shot optimisation
framework.

##  2 Related Work

In this section, we discuss various classic and deep image deconvolution
methods and their shortcomings. We also mention methods used in specific
scientific domains where image enhancement plays a key role.

###  2.1 Classic Deconvolution Algorithms

One of the earliest methods for non-blind deconvolution are the Wiener filter
[47] and Richardson-Lucy [34] that impose assumptions that image noise follows
Gaussian and Poisson distributions, respectively. Besides, Landweber iteration
[23] is another popular algorithm for sharp image recovery, which is a special
case of gradient descent. While those methods are fast and simple in their
implementations, they are prone to degradation and noise amplification. To
improve the reconstructed image quality, multiple optimisation methods
employing various priors, such as Laplacian, hyper-Laplacian [24, 19] or total
variation regularisation [36, 43], have been developed. Yuan et al. [49]
proposed an inter-scale and intra-scale non-blind deconvolution method to
obtain fine details while suppressing image artefacts. Manually designed
priors usually rely on statistical features of the natural images, which is
not suitable for all image types and can lead to inference problems.

###  2.2 Deep Deconvolution Models

Recently, deep models have been used for image deconvolution problems [51, 50,
38, 48] to learn the features needed for image restoration instead of
engineering them. Gong et al. [14] incorporated deep neural networks into a
fully parameterised gradient descent scheme to learn an implicit image prior.
The combination of the Wiener filter and CNNs into a deep supervised method
was proposed by Dong et al. [7], where the deconvolution is performed on
learned image features and not in the image space to improve the level of
detail and noise suppression. Jointly learning spatially-variant data and
regularisation terms within the MAP framework, as proposed by Dong et al. [8],
better captures the properties of clear images. While those methods achieve
impressive performance on benchmark computer vision datasets, they require
large quantities of ground truth data to train that are not accessible in
biological or medical imaging sciences. Additionally, using models trained on
images from a domain that differs from the target domain introduces a domain
shift, which could significantly reduce the performance in the target domain
and therefore the overall model usability.

Moreover, there has been a growing interest in the development of deep
restoration methods that leverage classic deconvolution algorithms for
volumetric data reconstruction, aiming to enhance processing efficiency [26,
4]. These approaches combine the principles of classical deconvolution with
advancements in deep learning and optimisation techniques to efficiently
restore volumetric datasets.

The concept of deep image prior (DIP) [42, 13] has been successfully applied
to denoising, super-resolution and deconvolution [32]. Chen et al. [3] applied
an ensemble of DIP models for image deblurring. These generative models
alleviate the need for training datasets as they are optimised only given an
input image and a kernel, hence being more flexible and adaptive. While these
methods perform well in general image restoration, they often overlook the
distinctive features and attributes of scientific images, demanding customised
solutions to effectively tackle their specific challenges. It is worth noting
that these methods do not take into account the importance of limited
computational resources. Therefore, developing tailored solutions for
scientific image restoration remains essential.

###  2.3 Domain Specific Deconvolution

Increasing the spatial resolution through deconvolution is a widely studied
problem in the microscopy imaging field. Zhao et al. [53] proposed a
processing pipeline for fluorescence microscopy consisting of background
removal, upsampling reconstruction based on sparsity and continuity priors and
iterative deconvolution with Richardson-Lucy [34]. Image deconvolution
algorithms in domains such as optical, electron and X-ray microscopy [45, 40,
37, 11, 53] rely on a specific imaging setup or image features characteristic
for the target domain. This prevents the use of such algorithms in different
fields and makes their window of applicability extremely narrow.

##  3 Proposed Method

We propose a combination of a classic non-blind iterative deconvolution
algorithm and deep learning. The objective of the method is to restore a
representation of the sharp image based purely on the blurry input image and
the appropriate kernel in a self-supervised manner. By embedding the
Richardson-Lucy algorithm [34] into the method we omit the need for using an
extensively large neural network to produce the sharp image.

###  3.1 Richardson-Lucy Algorithm

The Richardson-Lucy algorithm is a restoration technique widely used in
various fields, including astronomy [46], microscopy [53], and medical imaging
[5]. The primary objective of the method is to enhance the quality of images
degraded by blur and noise during the acquisition process (following Equation
1). The algorithm works iteratively and at each iteration it estimates an
intermediate restored image $x^{(i)}$, where $i$ represents the iteration
number. The update rule is as follows,

$x^{(i)}=x^{(i-1)}\cdot\left(\frac{y}{x^{(i-1)}*k}*k^{\top}\right).$ (2)

The restoration begins with an initial estimate of the original image
$x^{(0)}$ which can be set to the observed image $y$ or any other reasonable
initialisation. The algorithm iterates until a convergence criterion is met or
a predefined number of iterations is reached. One of the significant
advantages of the Richardson-Lucy algorithm is its ability to handle
deconvolution even in cases where the noise characteristics are unknown or
complex.

###  3.2 Feature Deconvolution With Richardson-Lucy

Similarly to [7], we address the deconvolution task by utilising deep features
instead of the conventional image space representation. The standard image
space deconvolution often proves to be insufficient in effectively removing
artifacts and restoring fine details in degraded images [6, 39]. To overcome
these limitations, we exploit deep features, which capture high-level abstract
information learned from convolutional neural networks, to guide the
restoration process more effectively.

Incorporating a set of linear filters $\\{f_{j}\\}_{j=1}^{n}$ generated
through deep neural networks, that are designed to extract feature information
from the degraded input, we can establish the relationship between $y$, $x$,
and $k$ within the feature space as follows,

$\mathbf{F}_{j}\mathbf{y}=\mathbf{KF}_{j}\mathbf{x}+\mathbf{F}_{j}\mathbf{b},$
(3)

where $\mathbf{F}_{j}$, $\mathbf{K}$, $\mathbf{y}$, $\mathbf{x}$, and
$\mathbf{b}$ denote the matrix/vector forms of $f_{j}$, $k$, $y$, $x$, and
$b$. Then, the Richardson-Lucy algorithm can be directly applied to the latent
features $\\{\mathbf{F}_{j}\mathbf{y}\\}$ as follows,

$\mathbf{F}_{j}\mathbf{x}^{(i)}=\mathbf{F}_{j}\mathbf{x}^{(i-1)}\left(\mathbf{K}^{\top}\frac{\mathbf{F}_{j}\mathbf{y}}{\mathbf{KF}_{j}\mathbf{x}^{(i-1)}}\right).$
(4)

Upon completing the iterations, we acquire the features of the desired latent
clear image denoted as $\\{{\mathbf{F}_{j}\hat{\mathbf{x}}}\\}$.

###  3.3 Self-Supervised Image Synthesis

In order to produce high-quality images, we adopt the DIP assumptions [42, 13,
32] and utilise a generative network $G_{x}$. This network takes the features
of the latent clear image as input and it has the structure of an asymmetric
Autoencoder [35] with skip-connections [42]. Unlike traditional DIP
approaches, where the generative network must possess significant modeling
capacity to generate images with rich textures and salient structures, we can
reduce the overall model size. This is because the deconvolution objective has
already been performed in the latent space, and thus, the purpose of the
network shifts to image synthesis rather than performing the deconvolution
process itself.

To impose regularisation on the generator, our primary approach involves
utilising the structural similarity index (SSIM) [44]. In conjunction with the
SSIM, we incorporate a regularisation term $\mathcal{R}(\cdot)$ aimed at
capturing important image priors as follows,

$\mathcal{L}(x,k,y)=\alpha\cdot\mathcal{L}_{\text{SSIM}}(x*k,y)+\lambda\cdot\mathcal{R}(x).$
(5)

This combination facilitates an effective means of enforcing structural
coherence and enhancing the overall quality of the generated outputs. The
integration of SSIM and the image prior regularisation empowers our generator
to produce more visually appealing and contextually meaningful results,
ensuring the preservation of essential characteristics present in the original
data. Specifically, we adopt the Hessian prior, which operates effectively in
the image space,

$\mathcal{R}(z)=\lVert{z_{xx}}\rVert_{1}+\lVert{z_{yy}}\rVert_{1}+2\lVert{z_{xy}}\rVert_{1},$
(6)

where $z_{i}$ denotes the second-order partial derivatives of $z$ along the
$x$ and $y$ axes. The purpose of the regularisation is to reduce the noise
artifacts arising from the deconvolution steps and to preserve the underlying
image structure. Figure 1 illustrates the full model architecture used to
synthesise the restored image.

###  3.4 Enhancing Microscopy Image Restoration

In the context of real microscopy image restoration, we initiate the process
with a pre-processing step involving background removal. This step helps in
enhancing the clarity of the images and preparing them for further
restoration. Additionally, to optimise the synthesis process, we modify the
loss function by introducing sparsity as an additional prior, alongside the
Hessian continuity prior. The inclusion of the sparsity prior is well-founded,
as it has demonstrated significant improvements in the reconstruction of
super-resolution microscopy images [52]. By leveraging both the Hessian and
sparsity priors, our approach aims to achieve superior results by enhancing
the visual quality of the microscopy images, and providing more accurate
representations of the underlying structures.

####  3.4.1 Background Removal

To address background interference in microscopy images, such as light
diffraction or scattering effects, we utilise the modified iterative wavelet
transform method [12], as described by Zhao et al. [52]. This approach is
applied to improve the reliability and visual quality of the subsequent image
restoration processes.

We use the residual image arising from setting the values over the mean value
of the original image $y$ to zero to estimate the background as follows: (i)
The background is iteratively estimated from the lowest frequency wavelet
bands related to the input image using 2D Daubechies-6 wavelet filters to
decompose the signal up to the 7${}^{\text{th}}$ level. (ii) An inverse
wavelet transform on the lowest band of the frequency information to the
spatial domain is performed to prevent the unintended removal of important
information. The result is then combined with $\sqrt{y}/2$ into a single image
whose pixels consist of the minimum of those two. (iii) The output of the
previous step is then used as the input in the next iteration. Following [52],
we set the number of iterations to 3 to estimate the background. The final
estimated background is then subtracted from the original input image $y$.

####  3.4.2 Sparsity Prior

To cater specifically to the unique characteristics of microscopy data, we
adapt our approach by augmenting the loss function $\mathcal{L}$ with sparsity
regularisation. The sparsity-aware loss function ensures that the
reconstruction process not only preserves continuity but also encourages the
final result to align with the sparse characteristics typically exhibited in
microscopy data. It is important to note that this sparsity regularisation is
deliberately included for microscopy image restoration, and may not be
utilised in general image reconstruction scenarios. This approach allows us to
achieve superior synthesis results by yielding more accurate representations
of the underlying content.

To quantify sparsity, we utilise the $\ell_{1}$ norm on the synthesized image,
which leads to the following adapted loss function,

$\displaystyle\mathcal{L}(x,k,y)$
$\displaystyle=\alpha\cdot\mathcal{L}_{\text{SSIM}}(x*k,y)$ (7)
$\displaystyle+\lambda\cdot\mathcal{R}(x)+\beta\cdot\lVert{x\rVert}_{1}.$ (8)

Input: $\mathbf{y},\mathbf{k}$

Output: $\hat{\mathbf{x}}$

$\mathbf{F}_{1:n}\mathbf{y}\leftarrow\text{feature\\_extractor}(\mathbf{y})$

$\mathbf{F}_{1:n}\hat{\mathbf{x}}\leftarrow\text{Richardson-
Lucy}(\mathbf{F}_{1:n}\mathbf{y})$

for  _$i\leftarrow 1$ to $T$_ do

$\mathbf{x}\leftarrow G_{x}^{t-1}(\mathbf{F}_{1:n}\hat{\mathbf{x}})$

Compute the gradient w.r.t. $G_{x}$

Update $G_{x}^{t}$ using the NAdam algorithm [9]

end for

$\hat{\mathbf{x}}=G_{x}^{T}(\mathbf{F}_{1:n}\hat{\mathbf{x}})$

Algorithm 1 CiDeR image synthesis

(a)

(b)

Figure 2: A comparison between the original raw light-sheet microscopy image
of a starfish (a) and the deconvolved output obtained using CiDeR (b) reveals
significant improvements. The deconvolved image effectively reduces haze and
light diffraction artifacts within the sample, resulting in clearer and more
accurate structures that were previously distorted due to image degradation.

##  4 Experimental Results

We implemented CiDeR using PyTorch [30]. For extracting features from the
degraded images, we employed a pre-trained model from [7], that was fine-tuned
on images from the Berkeley segmentation [28] and Waterloo Exploration [27]
datasets. This feature extractor consists of one convolutional layer and three
residual blocks [16], enabling the extraction of $n=16$ features.

The generator $G_{x}$ for the synthesis of the final image is optimised over
$T=3000$ iterations using the NAdam algorithm [9]. The initial learning rate
is set as $0.01$ and is decayed by multiplying $0.5$ when reaching $2,000$,
$2,300$ and $2,700$ iterations.

###  4.1 Results with Real Degradation

We assess the performance of CiDeR on two real microscopy modalities: light-
sheet and underwater microscopy with simulated microscopy kernels. In Figure
2, we present the primary comparison between the raw light-sheet image and the
restoration result obtained using CiDeR. The restoration with CiDeR enhances
the separation between cells, elevates the contrast, and improves the overall
image quality. Additionally, Figure 3 provides a comparison between CiDeR and
SelfDeblur [32] on an additional light-sheet image. Both methods employ a
self-supervised approach for image restoration, but SelfDeblur is susceptible
to overfitting, leading to the introduction of artifacts and image distortions
that degrade the final result. In contrast, the incorporation of more fitting
image priors in the optimisation objective of CiDeR prevents such degradation,
yielding a more faithful and visually pleasing restoration outcome.

In Figure 4, we present a comparison between the classic Richardson-Lucy
algorithm [34] and CiDeR. The raw image obtained by an underwater microscope
exhibits low contrast, and many features of the biological object remain
obscured. After restoration, both methods enhance the visibility of the
features, but CiDeR outperforms the Richardson-Lucy algorithm in significantly
improving the clarity of the object. With the ability of CiDeR to maintain
object structures while elevating the overall image quality, it demonstrates
its potential as a robust and effective solution for microscopy image
restoration tasks.

###  4.2 Results with Simulated Blur

Image | SelfDeblur [32] | CiDeR (ours)
---|---|---
1 | 0.8619 | 0.9108
2 | 0.8935 | 0.9043
3 | 0.9026 | 0.9174
4 | 0.7876 | 0.8496
Average | 0.8614 | 0.8956
Table 1: Quantitative comparison of CiDeR and SelfDeblur [32] on the dataset introduced by Levin et al. [25], employing the structural similarity index (SSIM) as the metric. For each row, the value represents the average per image over 8 different degradation models. Image | SelfDeblur [32] | CiDeR (ours)
---|---|---
1 | 29.10 | 30.99
2 | 30.35 | 30.71
3 | 32.05 | 32.92
4 | 26.73 | 27.89
Average | 29.56 | 30.63
Table 2: Quantitative comparison of CiDeR and SelfDeblur [32] on the dataset introduced by Levin et al. [25], employing the peak signal-to-noise ratio (PSNR) as the metric. For each row, the value represents the average per image over 8 different degradation models. | SelfDeblur [32] | CiDeR (ours)
---|---|---
No. parameters | 2,357,345 | 322,913
Table 3: Overview of the number of learnable parameters.

(a)

(b)

(c)

Figure 3: Visual comparison involving three images: (a) the raw image of a
starfish embryo captured by a light-sheet microscope, (b) the deconvolution of
a background-subtracted image using SelfDeblur [32], and (c) the image
restoration result obtained using CiDeR.

(a)

(b)

(c)

Figure 4: Visual comparison involving three images: (a) the raw image captured
by an underwater microscope, (b) the deconvolution of a background-subtracted
image using the classic Richardson-Lucy algorithm [34], and (c) the image
restoration result obtained using CiDeR. The result from CiDeR exhibits
improved visual features and fidelity.

(a)

(b)

(c)

(d)

Figure 5: Visual comparison of the deconvolution results of an image from the
dataset of Levin et al. [25]. Subfigure (a) shows the degraded image, while
subfigure (b) displays the ground-truth sharp image. Subfigures (c) and (d)
demonstrate the reconstruction results obtained from SelfDeblur [32] and
CiDeR, respectively.

We compare CiDeR with SelfDeblur [32], which is the state-of-the-art self-
supervised non-blind deconvolution method, using the dataset of Levin et al.
[25]. The dataset comprises 32 images, that were generated using 4 sharp
images and 8 distinct kernels to simulate various image degradations. Since
both models are evaluated in a non-blind setting, we utilise the kernels
estimated by [25] to facilitate the restoration process. Tables 1 and 2 list
the average SSIM and PSNR metrics per each image. In terms of both metrics,
CiDeR outperforms SelfDeblur while having significantly less parameters as
shown in Table 3.

##  5 Discussion & Future Work

Image degradation is a common problem in a wide range of field, including
telescope imaging, microscopy imaging, or medical imaging. Leveraging the
Richardson-Lucy algorithm and DIP assumptions, our image restoration technique
proves to be a valuable tool for reconstructing images in these diverse
fields, with a potential to significantly improve the accuracy of downstream
tasks. In addition to its restoration capabilities, our method benefits from
the incorporation of a classic deconvolution algorithm, which results in a
substantial reduction in the model size. This reduction proves particularly
advantageous in fields where computational resources are scarce, as it
significantly lowers the overall computational cost involved in image
restoration. As a result, our approach not only produces high-quality
reconstructions but also offers an efficient and resource-friendly solution
for restoring images, making it better-suited for application in resource-
constrained environments compared to existing methods.

Unlike the feature-based Wiener deconvolution model [7], CiDeR takes a
different approach by not relying solely on an end-to-end trained feature
refinement model. As a result, it effectively addresses the issue of domain
gap, since the final image generator is directly optimised on the input image.
This direct optimisation mitigates potential challenges associated with domain
mismatches, ensuring a more faithful image restoration process. Furthermore,
the use of the Wiener filter introduces computational bottlenecks, primarily
due to its reliance on the Fourier transform. Such computational costs are
particularly evident when dealing with volumetric or 3D data processing [31].
In contrast, using Richardson-Lucy algorithm offers a significant advantage in
terms of computational efficiency, especially for volumetric data [26, 4].
This efficiency opens up exciting possibilities for future work to explore
feature-based 2.5D or 3D image restoration tasks, allowing for more extensive
and sophisticated applications in three-dimensional data processing.

##  6 Conclusion

In this paper, we propose a novel zero-shot image restoration method called
CiDeR that effectively combines a pre-trained feature extraction network with
a classic deconvolution algorithm, namely Richardson-Lucy. Our approach
leverages deconvolved features to synthesize a reconstructed image, offloading
the actual deconvolution task to the classical algorithm. As a result, the
self-supervised generator solely focuses on learning the synthesis process,
allowing us to design a smaller network compared to existing state-of-the-art
methods. This reduction in model size proves beneficial as it conserves
computational resources, making our approach particularly well-suited for
scenarios where such resources are limited.

Furthermore, our method is designed to be self-supervised, a crucial advantage
in biomedical fields where data scarcity is prevalent, and building end-to-end
networks is often not possible. Through extensive testing on a classic
computer vision benchmark dataset and real microscopy images, the results
demonstrate that our method excels in image restoration tasks. This enhanced
image quality is particularly crucial for downstream tasks, where accurate
images with high fidelity play an important role. Overall, CiDeR exhibits
promising potential for various practical applications, which makes it a
valuable tool that addresses the challenges of image restoration in domains
where data is scarce and computational resources are limited.

## Acknowledgement

Tomáš Chobola is supported by the Helmholtz Association under the joint
research school ”Munich School for Data Science - MUDS”.

## References

  * [1] Dominik Bauer, Tom Russ, Barbara Waldkirch, Christian Tönnes, William Segars, Lothar Schad, Frank Zöllner, and Alena-Kathrin Golla.  Generation of annotated multimodal ground truth datasets for abdominal medical image registration.  International Journal of Computer Assisted Radiology and Surgery, 16, 05 2021.
  * [2] Jian-Feng Cai, Bin Dong, Stanley Osher, and Zuowei Shen.  Image restoration: total variation, wavelet frames, and beyond.  Journal of the American Mathematical Society, 25(4):1033–1089, 2012\.
  * [3] Mingqin Chen, Yuhui Quan, Yong Xu, and Hui Ji.  Self-supervised blind image deconvolution via deep generative ensemble learning.  IEEE Transactions on Circuits and Systems for Video Technology, 2022\.
  * [4] Tomáš Chobola, Gesine Müller, Veit Dausmann, Anton Theileis, Jan Taucher, Jan Huisken, and Tingying Peng.  Lucyd: A feature-driven richardson-lucy deconvolution network.  arXiv preprint arXiv:2307.07998, 2023.
  * [5] Flavio Dell’Acqua, Paola Scifo, Giovanna Rizzo, Marco Catani, Andrew Simmons, Giuseppe Scotti, and Ferruccio Fazio.  A modified damped richardson–lucy algorithm to reduce isotropic background effects in spherical deconvolution.  Neuroimage, 49(2):1446–1458, 2010.
  * [6] Jiangxin Dong, Jinshan Pan, Deqing Sun, Zhixun Su, and Ming-Hsuan Yang.  Learning data terms for non-blind deblurring.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 748–763, 2018.
  * [7] Jiangxin Dong, Stefan Roth, and Bernt Schiele.  Deep wiener deconvolution: Wiener meets deep learning for image deblurring.  Advances in Neural Information Processing Systems, 33:1048–1059, 2020.
  * [8] Jiangxin Dong, Stefan Roth, and Bernt Schiele.  Learning spatially-variant map models for non-blind image deblurring.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4886–4895, 2021.
  * [9] Timothy Dozat.  Incorporating nesterov momentum into adam.  2016\.
  * [10] Thomas Eboli, Jian Sun, and Jean Ponce.  End-to-end interpretable learning of non-blind image deblurring.  In European Conference on Computer Vision, pages 314–331. Springer, 2020.
  * [11] Sebastian Ehn, Franz Michael Epple, Andreas Fehringer, David Pennicard, Heinz Graafsma, Peter Noël, and Franz Pfeiffer.  X-ray deconvolution microscopy.  Biomedical Optics Express, 7(4):1227, Mar. 2016.
  * [12] C Galloway, Eric Le Ru, and P Etchegoin.  An iterative algorithm for background removal in spectroscopy by wavelet transforms.  Applied spectroscopy, 63:1370–6, 12 2009.
  * [13] Yosef Gandelsman, Assaf Shocher, and Michal Irani.  ” double-dip”: Unsupervised image decomposition via coupled deep-image-priors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11026–11035, 2019.
  * [14] Dong Gong, Zhen Zhang, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, and Yanning Zhang.  Learning deep gradient descent optimization for image deconvolution.  IEEE transactions on neural networks and learning systems, 31(12):5468–5482, 2020.
  * [15] Weihong Guo, Jing Qin, and Wotao Yin.  A new detail-preserving regularization scheme.  SIAM journal on imaging sciences, 7(2):1309–1334, 2014.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [17] Suyog Jadhav, Sebastian Acuña, Ida S. Opstad, Balpreet Singh Ahluwalia, Krishna Agarwal, and Dilip K. Prasad.  Artefact removal in ground truth deficient fluctuations-based nanoscopy images using deep learning.  Biomedical Optics Express, 12(1):191, Dec. 2020.
  * [18] Meiguang Jin, Stefan Roth, and Paolo Favaro.  Noise-blind image deblurring.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3510–3518, 2017.
  * [19] Dilip Krishnan and Rob Fergus.  Fast image deconvolution using hyper-laplacian priors.  Advances in neural information processing systems, 22, 2009.
  * [20] Jakob Kruse, Carsten Rother, and Uwe Schmidt.  Learning to push the limits of efficient fft-based image deconvolution.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 4596–4604, 2017.
  * [21] Hiroyuki Kudo, Dong Jian, Katsumi Kamo, Noritaka Horii, Hiromitsu Furukawa, Satoshi Hata, Mitsuhiro Murayama, Kazuhisa Sato, and Shinsuke Miyazaki.  Tomographic image reconstruction using compressed sensing.  KENBIKYO, 51(1):48–53, 2016.
  * [22] Wei-Sheng Lai, Jia-Bin Huang, Zhe Hu, Narendra Ahuja, and Ming-Hsuan Yang.  A comparative study for single image blind deblurring.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1701–1709, 2016.
  * [23] L. Landweber.  An iteration formula for fredholm integral equations of the first kind.  American Journal of Mathematics, 73(3):615–624, 1951.
  * [24] Anat Levin, Rob Fergus, Frédo Durand, and William T Freeman.  Image and depth from a conventional camera with a coded aperture.  ACM transactions on graphics (TOG), 26(3):70–es, 2007.
  * [25] Anat Levin, Yair Weiss, Fredo Durand, and William T. Freeman.  Understanding and evaluating blind deconvolution algorithms.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1964–1971, 2009.
  * [26] Yue Li, Yijun Su, Min Guo, Xiaofei Han, Jiamin Liu, Harshad D Vishwasrao, Xuesong Li, Ryan Christensen, Titas Sengupta, Mark W Moyle, et al.  Incorporating the image formation process into deep learning improves network performance.  Nature Methods, pages 1–11, 2022.
  * [27] Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang.  Waterloo exploration database: New challenges for image quality assessment models.  IEEE Transactions on Image Processing, 26(2):1004–1016, 2016.
  * [28] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416–423. IEEE, 2001.
  * [29] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee.  Deep multi-scale convolutional neural network for dynamic scene deblurring.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3883–3891, 2017.
  * [30] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [31] Dmitry Pekurovsky.  P3dfft: A framework for parallel computations of fourier transforms in three dimensions.  SIAM Journal on Scientific Computing, 34(4):C192–C209, 2012.
  * [32] Dongwei Ren, Kai Zhang, Qilong Wang, Qinghua Hu, and Wangmeng Zuo.  Neural blind deconvolution using deep priors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3341–3350, 2020.
  * [33] Wenqi Ren, Jiawei Zhang, Lin Ma, Jinshan Pan, Xiaochun Cao, Wangmeng Zuo, Wei Liu, and Ming-Hsuan Yang.  Deep non-blind deconvolution via generalized low-rank approximation.  Advances in neural information processing systems, 31, 2018.
  * [34] William Hadley Richardson.  Bayesian-based iterative method of image restoration$\ast$.  J. Opt. Soc. Am., 62(1):55–59, Jan 1972.
  * [35] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [36] Leonid I Rudin, Stanley Osher, and Emad Fatemi.  Nonlinear total variation based noise removal algorithms.  Physica D: nonlinear phenomena, 60(1-4):259–268, 1992.
  * [37] P. Sarder and A. Nehorai.  Deconvolution methods for 3-d fluorescence microscopy images.  IEEE Signal Processing Magazine, 23(3):32–45, 2006.
  * [38] Christian J Schuler, Harold Christopher Burger, Stefan Harmeling, and Bernhard Scholkopf.  A machine learning approach for non-blind image deconvolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1067–1074, 2013.
  * [39] Qi Shan, Jiaya Jia, and Aseem Agarwala.  High-quality motion deblurring from a single image.  Acm transactions on graphics (tog), 27(3):1–10, 2008.
  * [40] Jean-Baptiste Sibarita.  Deconvolution microscopy.  In Microscopy Techniques, pages 201–243. Springer Berlin Heidelberg, May 2005.
  * [41] Libin Sun, Sunghyun Cho, Jue Wang, and James Hays.  Edge-based blur kernel estimation using patch priors.  In IEEE International Conference on Computational Photography (ICCP), pages 1–8, 2013.
  * [42] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Deep image prior.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446–9454, 2018.
  * [43] Yilun Wang, Junfeng Yang, Wotao Yin, and Yin Zhang.  A new alternating minimization algorithm for total variation image reconstruction.  SIAM Journal on Imaging Sciences, 1(3):248–272, 2008.
  * [44] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [45] K. Watanabe, Y. Kotaka, N. Nakanishi, T. Yamazaki, I. Hashimoto, and M. Shiojiri.  Deconvolution processing of haadf stem images.  Ultramicroscopy, 92(3):191–199, 2002.
  * [46] Richard L White.  Image restoration using the damped richardson-lucy method.  In Instrumentation in Astronomy VIII, volume 2198, pages 1342–1348. SPIE, 1994.
  * [47] Norbert Wiener, Norbert Wiener, Cyberneticist Mathematician, Norbert Wiener, Norbert Wiener, and Cybernéticien Mathématicien.  Extrapolation, interpolation, and smoothing of stationary time series: with engineering applications, volume 113.  MIT press Cambridge, MA, 1949.
  * [48] Li Xu, Jimmy S Ren, Ce Liu, and Jiaya Jia.  Deep convolutional neural network for image deconvolution.  Advances in neural information processing systems, 27, 2014.
  * [49] Lu Yuan, Jian Sun, Long Quan, and Heung-Yeung Shum.  Progressive inter-scale and intra-scale non-blind image deconvolution.  Acm Transactions on Graphics (TOG), 27(3):1–10, 2008.
  * [50] Jiawei Zhang, Jinshan Pan, Wei-Sheng Lai, Rynson WH Lau, and Ming-Hsuan Yang.  Learning fully convolutional networks for iterative non-blind deconvolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3817–3825, 2017.
  * [51] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang.  Learning deep cnn denoiser prior for image restoration.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3929–3938, 2017.
  * [52] Weisong Zhao, Shiqun Zhao, Liuju Li, Xiaoshuai Huang, Shijia Xing, Yulin Zhang, Guohua Qiu, Zhenqian Han, Yingxu Shang, De-en Sun, and et al.  Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy.  Nature Biotechnology, 40(4):606–617, 2021.
  * [53] Weisong Zhao, Shiqun Zhao, Liuju Li, Xiaoshuai Huang, Shijia Xing, Yulin Zhang, Guohua Qiu, Zhenqian Han, Yingxu Shang, De-en Sun, et al.  Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy.  Nature biotechnology, 40(4):606–617, 2022.
  * [54] Frank G Zöllner, Amira Šerifović-Trbalić, Gordian Kabelitz, Marek Kociński, Andrzej Materka, and Peter Rogelj.  Image registration in dynamic renal mri—current status and prospects.  Magnetic Resonance Materials in Physics, Biology and Medicine, 33(1):33–48, 2020.