# NAF-DPM: A Nonlinear Activation-Free Diffusion Probabilistic Model for
Document Enhancement

Giordano Cicchetti, and Danilo Comminiello Authors are with the Department of
Information Engineering, Electronics and Telecommunications (DIET), Sapienza
University of Rome, Italy.
Corresponding author’s email<EMAIL_ADDRESS>
###### Abstract

Real-world documents may suffer various forms of degradation, often resulting
in lower accuracy in optical character recognition (OCR) systems. Therefore, a
crucial preprocessing step is essential to eliminate noise while preserving
text and key features of documents. In this paper, we propose NAF-DPM, a novel
generative framework based on a diffusion probabilistic model (DPM) designed
to restore the original quality of degraded documents. While DPMs are
recognized for their high-quality generated images, they are also known for
their large inference time. To mitigate this problem we provide the DPM with
an efficient nonlinear activation-free (NAF) network and we employ as a
sampler a fast solver of ordinary differential equations, which can converge
in a few iterations. To better preserve text characters, we introduce an
additional differentiable module based on convolutional recurrent neural
networks, simulating the behavior of an OCR system during training.
Experiments conducted on various datasets showcase the superiority of our
approach, achieving state-of-the-art performance in terms of pixel-level and
perceptual similarity metrics. Furthermore, the results demonstrate a notable
character error reduction made by OCR systems when transcribing real-world
document images enhanced by our framework. Code and pre-trained models are
available at https://github.com/ispamm/NAF-DPM.

###### Index Terms:

Document enhancement, document deblurring, document binarization, diffusion
models, generative deep learning, OCR.

##  I Introduction

Document images, captured by cameras, scanners, and commercial reading
devices, are typically dirty and noisy. Numerous factors contribute to the low
quality of document images such as low-accurate devices, bad lighting
conditions, improper shooting angles, shadows or blurred text. Additional
noise can be caused by the presence of stains, watermarks, ink artifacts or by
background textures - for instance, colored images, logos, photocopies, and
wrinkled papers. These are the reasons why degraded documents are often
challenging for modern optical character recognition (OCR) systems to be
transcribed. The accuracy of OCR systems is also crucial for all the
downstream tasks that rely on them such as key information extraction, text
mining, text-to-speech (TTS) or automatic machine translation. Therefore,
restoring the original quality of the degraded document is of fundamental
importance.

Figure 1: Random samples of image patches extracted from the Document
Deblurring dataset, DIBCO2017 and H-DIBCO2018 [1, 2, 3]. The first column
refers to degraded images; The second one refers to their counterparts
restored by our framework. These enhanced images are high-quality and very
similar to the original ones. Furthermore, the edges of the text appear to be
sharper and well defined. All this benefits the work of an OCR that will be
able to better recognize text.

One of the most popular tasks within the field of document enhancement is
document deblurring. The goal of this task is to remove blur patterns and
restore the original readability of the documents [4]. Document binarization
is another widely studied problem, whose goal is to classify each pixel either
as a background or a text pixel in order to highlight the relevant content[5].
The major challenges presented by these problems are the removal of noise
while keeping intact the text, the shape of every single character (both
machine-printed or hand-written), and the basic features of documents [6].
Intense noise often leads to the complete degradation of the text, and
reproducing the correct text output is a daunting task. The presence of
watermarks, for example, introduces a large amount of dirt, making the
document difficult to read. While locating the watermark might be an
affordable task, its removal and subsequent recovery of the real occulted
information turns out to be very challenging. An additional challenge is the
inference time, especially in an industrial environment where there is the
need to process high-quality images in a short amount of time. This requires
the development of a framework with high restoration capabilities but at the
same time with low inference latency [7, 8, 9, 10].

In recent years, one of the most promising and effective approaches to address
these challenges has involved deep generative models. Among them, variational
autoencoders (VAEs) [11, 12, 13], generative adversarial networks (GANs)[14,
15, 16], and diffusion probabilistic models (DPMs) [17, 18, 19] have achieved
great notoriety. The latter ones, DPMs, are now considered the most reliable
choice for high-quality image generation problems (both conditioned and
unconditioned). Indeed, it has been shown that they are surprisingly good in
sample quality, surpassing GANs in image generation [20], and they also
feature good mode coverage and diversity, as indicated by high likelihood.
However, generating new items from them often requires thousands of neural
network evaluations, making their application to real-world problems expensive
in terms of time and computational power.

In this paper, we propose a novel generative framework based on DPMs that
solves document enhancement tasks by treating them as conditional image-to-
image translation problems. To design a specific DPM-based framework for
document enhancement tasks, we focus on two main problems: the large sample
diversity, which can lead to inconsistent character restoration, and the huge
inference times.

To reduce the diversity of the model, we introduce a deterministic predictor
before the DPM model that tries to restore as many low-frequency details as
possible. To shorten inference time, we worked on both the architecture of the
framework and the sampling strategy level. Regarding the architecture, we
investigate the power and effectiveness of a nonlinear activation-free network
(NAFNet) [21]. Such a network turned out to be very effective for the image
denoising task along with a substantial reduction in computational resources
needed. Motivated by these results, we design a lightweight variant of NAFNet
to be used as backbone network for a DPM.

In order to guarantee a good sampling quality in a few iterations, we have
adopted a deterministic sampling strategy based on an ordinary differential
equation (ODE) fast solver [22]. Experiments point out that performance
converges in at most 20 iterations, instead of hundreds or even thousands of
iterations required by alternative sampling strategies. Finally, to make our
framework robust to character substitution errors, we have included an
additional differentiable module that simulates the behavior of a commercial
OCR system.

We evaluate the effectiveness of our framework on two representative tasks for
document enhancement: deblurring and binarization. The results demonstrate the
superior performance of our framework with respect to state-of-the-art
methods. Within the document deblurring task, we achieve a significant
improvement on the widely used peak signal-to-noise ratio (PSNR) metric with
respect to DE-GAN [8] and DocDiff [10]. On the same task, we validate the
strength of our framework using a commercial OCR system. Surprisingly, the
images restored by our framework show a high reduction of character errors
detected by OCR, halving the character error rate (CER) compared to DocDiff
[10]. For document binarization we test our framework on the three most recent
datasets from DIBCO Challenges [2, 3, 23]. We reach competitive results for
every dataset, setting a new state-of-the-art for the challenging DIBCO2019
dataset.

##  II Related Work

Document enhancement is a vast area that includes a lot of different subtasks
such as deblurring, background texture removal, page smudging, watermark
removal and handwriting fading. In this section, We group some of the most
relevant works in these areas into tasks: document deblurring and document
binarization. The advent of diffusion models in recent years has brought an
incredible increase in performance in various generative deep learning tasks.
However, there are few works based on this technology in the field of document
enhancement. We have devoted a section to summarize some of the remarkable
work based on conditional diffusion models for the more general category
image-to-image transformation.

###  II-A Document Deblurring

One of the most common disturbances that affect document images is blurring.
Fuzzy characters can be very difficult to be recognized by an OCR system and
sometimes even by the human eye (Fig. 2a). Before the advent of deep learning,
prominent approaches against document deblurring were based on estimating the
blur kernel and then removing it using non-blind deconvolution [24, 25, 26].
Then in 2015 Hradis et al. proposed a method based on convolutional neural
network (CNN) [1]. The trained networks are able to perform blind image
deconvolution without assuming any blur kernel or noise pattern. Later on,
remarkable success was made by GAN-based architectures such as DeblurGAN [27],
DEGAN [8], Blur2sharp [28] and DeblurGAN-CNN [29]. All the aforementioned
models are trained to reduce pixel-level loss ($\mathcal{L}_{1}$ or
$\mathcal{L}_{2}$). Their final aim is to reach high values on distortion
metrics such as peak signal-to-noise ratio (PSNR) and structural similarity
index measure (SSIM). However, it has been shown that distortion metrics and
perceptual quality are at odds with each other [30]. As a result, text
restored from these models has blurred and distorted edges. Recently was
presented the first diffusion model applied to document enhancement tasks:
DocDiff [10]. The images refined by the diffusion-based network appear to be
sharper with respect to previous methods and this directly affects the
performance of OCR systems.

Figure 2: Examples of document images used during this work. Images in
subfigure (a) come from document deblurring OCR test dataset [1] and their
associated task is image deblurring. Images in subfigures (b),(c),(d) come
from the annual Document Image Binarization Competition (DIBCO) [2, 23, 3].
and their associated task is image binarization.

###  II-B Document Binarization

Document binarization aims to classify each pixel of an image as either a
background pixel or a text pixel. At the end of the process, we can obtain a
binary image where only the information concerning text and important features
of the image are retained. Different types of degradation occurring in
documents can be classified as binarization problems: background texture
removal, page smudging, watermark removal, handwriting fading and others (Fig.
2 b,c,d).

Binarization techniques can be divided into threshold-based and numerical-
based methods. Well known threshold-based techniques are Otsu [31], Sauvola
[32] and Niblack algorithms. They are able to separate the foreground from the
background of an image by finding a threshold value that maximizes the
variance between the two classes. Lelore et al. [7] introduced a model called
FAIR. It is based on edge detection to localize the text in a degraded
document image. These approaches guarantee fast computation speed and broad
applicability, but, on the other hand, the binarization process is not ideal
in some circumstances. Numerical-based methods try to classify each pixel
based on the context in which they are surrounded. In 2019, Bezmaternykh et
al. proposed a model based on U-NET (encoder-decoder network with skip-
connection) [33]. Generative adversarial networks also show their potential
for this task [8]. In 2022, DocEnTr was proposed. It is a transformer-based
network that sets the state-of-the-art performance on different document
binarization datasets [9].

###  II-C Document Enhancement with Diffusion Models

The advent of Diffusion Probabilistic Models (DPMs) was a significant
breakthrough in generative deep learning. It has been shown that they are
surprisingly good in sample quality, beating GANs in image generation [20].
They also feature good mode coverage and diversity, indicated by high
likelihood. However, generating new items from them often requires thousands
of neural network evaluations, making their application to real-world problems
expensive in terms of time and computational power. Diffusion models have been
employed in all computer vision tasks: image super resolution, inpainting,
restoration, translation, and editing. Super-Resolution via Repeated
Refinement (SR3) [34] is one of the remarkable works in this field. It uses
DPM to enable conditional image generation, conducting super-resolution
through a stochastic iterative denoising process. Similar to SR3, Palette [19]
makes use of a conditional diffusion model to create a unified framework for
four image generation tasks: colorization, inpainting, uncropping, and JPEG
restoration. Some works try to use a DPM to predict the residual image, which
then can be added to the degraded one to restore its quality [35, 36, 37]. As
already said, the major drawback of diffusion models is their huge inference
time. To speed up the sampling phase the main techniques used are: advanced
SDE/ODE solvers [22, 38, 39], distillation techniques [40, 41, 42] and low-
dimensional diffusion models [43, 44, 45]. The fast ODE solver from [22] is
able to converge in a few iterations without sacrificing quality.

Figure 3: NAF-DPM architecture. An initial predictor retrieves the low-
frequency information, and then a denoiser network estimates the residual
high-frequency details by iterative refinement. The high-frequency information
is restored estimating the residual image that, at the end, is added back to
the image predicted by the initial predictor network. As the backbone network
for the initial predictor we employ an efficient nonlinear activation-free
network (NAFNet [21]). As the backbone network for the denoiser diffusion
model we design a novel and effective variation of NAFNet that takes into
consideration the conditioning of the timestep $t$ of diffusion models in
order to improve performance in denoising and deblurring tasks. It is worth
noting that the initial predictor significantly retrieves the structure of the
document while the denoiser restores all the details that make the text
correct and readable. To better preserve text characters, we finetune our
framework with an additional differentiable module based on convolutional
recurrent neural networks, simulating the behavior of an OCR system during
training

##  III Proposed Approach

###  III-A Overview Of The Proposed NAF-DPM

In our work we propose a generative framework based on a nonlinear activation-
free diffusion probabilistic model (NAF-DPM). It exploits the idea “predict-
and-refine” introduced in [36] and further refined in [37, 10]. As illustrated
in Fig. 3, we first employ an initial predictor that takes the degraded images
and produces an enhanced version by removing as much noise as possible. Then a
DPM network only needs to restore the residual details. The two networks are
jointly trained and a frequency separation technique is employed. The initial
predictor is responsible for the low-frequency information restoration, while
the denoiser is responsible for the high-frequency information restoration.
Various studies have shown that this approach is able to effectively generate
images with much more defined details [37, 46, 47, 10]. To separate the
frequencies of an image we use simple linear filters, similar to [37, 10].
Using a low-pass filter $\phi_{L}$ and a high-pass filter $\phi_{H}$ we can
rewrite an image x using:

$\mathbf{\bar{x}}=\phi_{L}\ast\mathbf{x}+\phi_{H}\ast\mathbf{x}\vspace{0.1cm}$
(1)

To better preserve text characters and to teach the network how to restore all
the details, we finetune our framework with an additional differentiable
module based on convolutional recurrent neural networks, simulating the
behavior of an OCR system during training. This module is used to extract text
from reconstructed images and compute an additional loss function that better
guides the diffusion model to restore the original quality of the text in
document images.

###  III-B Initial predictor

The initial predictor takes the degraded image and retrieves as much low-
frequency information as possible. We use a scaled-down version of a nonlinear
activation-free network (NAFNet), since it has demonstrated remarkable
performance in tasks such as denoising and deblurring [21]. NAFNet takes
inspiration both from a U-Net, for extra-block organization, and from a Vision
Transformer, for intra-block organization. All the components are well
designed to be efficient (in terms of inference time and parameters) and
effective (SOTA performance). The initial predictor takes degraded images
$\mathbf{y}$ and produces an enhanced version
$\mathbf{x}_{I}=i_{\theta}(\mathbf{y})$. We associate to this component two
dedicated loss functions. The first one tries to minimize the mean squared
error between original images and enhanced ones. The second one pushes the
network to restore low-frequency information.

$L_{\text{MSE}}=\left\|\mathbf{x}_{gt}-\mathbf{x}_{I}\right\|_{2}^{2}$ (2)
$L_{\text{LOW}}=\left\|\phi_{L}\ast\left(\mathbf{x}_{gt}-\mathbf{x}_{I}\right)\right\|_{2}^{2}$
(3) $L_{\text{INIT}}=L_{\text{MSE}}+2\ast L_{\text{LOW}}\vspace{0.1cm}$ (4)

Figure 4: Proposed internal structure of each NAF block of our denoiser
diffusion model. We added a time processing branch, depicted in red color.
This branch models the time embedding and transforms it into shift parameters
$\gamma$ and $\beta$ that control the scale and bias terms of each
normalization layer. SCA: Simple Channel Attention; MLP: Multi-Layer
Perceptron.

###  III-C Conditional DPM Refiner

Diffusion models constitute a category of probabilistic generative models that
systematically introduce noise to data and then learn how to reverse this
process for the purpose of generating samples [17, 48]. Following the work
done in [36, 37, 10], our denoiser diffusion model is trained to model the
residual distribution.

Given a ground truth image $\textbf{x}_{gt}$ and its degraded counterpart y,
we define the residual image as the difference between the ground truth
$\textbf{x}_{gt}$ and the enhanced image by the initial predictor
$\textbf{x}_{I}=i_{\theta}(\textbf{y})$.

$\textbf{r}_{0}=\textbf{x}_{gt}-\textbf{x}_{I}\vspace{0.1cm}$ (5)

The forward diffusion process models a Markov chain that, starting from a
residual sample $\mathbf{r}_{0}$, iteratively adds Gaussian noise according to
a transition kernel $q\left(\mathbf{r}_{t}\mid\mathbf{r}_{t-1}\right)$ defined
as follow:

$q\left(\mathbf{r}_{t}\mid\mathbf{r}_{t-1}\right):=\mathcal{N}\left(\mathbf{r}_{t};\sqrt{\alpha_{t}}\mathbf{r}_{t-1},\left(1-\alpha_{t}\right)\mathbf{I}\right),$
(6)

where $(\alpha_{0},\alpha_{1},\dots,\alpha_{T})$ are hyperparameters that
define the noise scheduler and they control the amount of noise that needs to
be added at each timestep. According to this definition, the forward process
transforms residual data distribution $\mathbf{r}_{0}\sim q(\mathbf{r}_{0})$
into a tractable prior standard gaussian distribution
$q\left(\mathbf{r}_{T}\right)\sim\mathcal{N}\left(\mathbf{r}_{T};\mathbf{0},\mathbf{I}\right)$.
In the forward process, there are no learnable parameters and we can obtain
every intermediate state $\textbf{r}_{t}$ by just adding a quantity of noise
defined by the noise scheduler to $\textbf{r}_{0}$:

$q\left(\mathbf{r}_{t}\mid\mathbf{r}_{0}\right):=\mathcal{N}\left(\mathbf{r}_{t};\sqrt{\bar{\alpha}_{t}}\mathbf{r}_{0},\left(1-\bar{\alpha}_{t}\right)\mathbf{I}\right)$
(7) $\bar{\alpha_{t}}:=\prod_{s=0}^{t}\alpha_{s}$ (8)

Using the reparametrization trick, eq. (7), can be written as follow:

$\mathbf{r}_{t}=\sqrt{\bar{\alpha_{t}}}\mathbf{r}_{0}+\sqrt{1-\bar{\alpha_{t}}}\bm{\epsilon}\qquad\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$
(9)

The inverse procedure models another Markov chain that starting from pure
noise $\mathbf{r}_{T}$ and conditioned to $\mathbf{x}_{I}$, progressively
eliminates noise in a recursive manner to predict the exact residual image
$\mathbf{r}_{0}$. This iterative noise elimination is referred to as the
conditional reverse transition
$p_{\theta}(\textbf{r}_{t-1}|\textbf{r}_{t},\textbf{x}_{I})$, which is
estimated by optimizing the trainable parameters $\theta$ within a denoising
network:

$p_{\theta}\left(\mathbf{r}_{t-1}\mid\mathbf{r}_{t},\textbf{x}_{I}\right):=\mathcal{N}\left(\mathbf{r}_{t-1};\bm{\mu}_{\theta}\left(\mathbf{r}_{t},t,\textbf{x}_{I}\right),\bm{\Sigma}_{\theta}\left(\mathbf{r}_{t},t,\textbf{x}_{I}\right)\right)$
(10)

As denoising network $f_{\theta}$, we design a novel and effective variation
of NAFNet that, along with $\textbf{r}_{t}$ and $\textbf{x}_{I}$ as inputs,
takes into consideration also the conditioning of the timestep $t$ of
diffusion models in order to improve performance in denoising and deblurring
tasks. Going into details, following [49], we employ an additional MLP to
model the time embedding and transform it into shift parameters $\gamma$ and
$\beta$. These parameters control scale and bias terms of each normalization
layer that now are conditioned on $t$. The mechanism can be easily visualized
in Fig. 4.

Given a noisy residual sample $\textbf{r}_{t}$, we train the denoiser network
$f_{\theta}$ to directly estimate $\textbf{r}_{0}$ instead of estimating the
amount of noise needed to form $\textbf{r}_{t-1}$. The prediction of
$\textbf{r}_{0}$ and $\textbf{r}_{t-1}$ is equivalent as they can be
transformed into each other through eq. (9). It has been proven that this
approach reduces diversity in favor of generation quality in the first
denoising steps [19, 34, 10]. We associate to this component two dedicated
loss functions. The first one that tries to minimize the distance between the
estimated reverse transition
$p_{\theta}(\textbf{r}_{t-1}\mid\textbf{r}_{t},\textbf{x}_{I})$ and the real
one $q(\textbf{r}_{t-1}\mid\textbf{r}_{t},\textbf{x}_{I})$. The second one
tries to specialize the denoiser network to recover high-frequency details.

$L_{\text{DPM}}=\mathbb{E}\left\|\textbf{r}_{0}-f_{\theta}\left(\textbf{r}_{t},t,\textbf{x}_{I}\right)\right\|_{2}$
(11)
$L_{\text{HIGH}}=\mathbb{E}\left\|\phi_{H}\ast\left(\textbf{r}_{0}-f_{\theta}\left(\textbf{r}_{t},t,\textbf{x}_{I}\right)\right)\right\|_{2}$
(12) $L_{\text{DENOISER}}=L_{\text{DPM}}+2\ast L_{\text{HIGH}}$ (13)

Overall, the used loss function is the combination of eq. (4) and eq. (13):

$L_{\text{TOT}}=\frac{1}{2}L_{\text{INIT}}+L_{\text{DENOISER}}\vspace{0.1cm}$
(14)

The training strategy adopted is detailed described using pseudo-code in
Algorithm 1.

Algorithm 1 Training NAF-DPM

INPUT: $i_{\theta}$ initial predictor, $f_{\theta}$ denoiser network, $T$
Sampling steps, $\bar{\bm{\alpha}}_{0:T}$ noise scheduler;

OUTPUT: $i_{\theta^{\star}}$ trained initial predictor, $f_{\theta^{\star}}$
trained denoiser network;

FOR $i\leftarrow 0$ TO MAX_ITERATIONS DO:

($\textbf{x}_{gt}$,y) $\sim$ q(data) sample images from dataset

$t$ $\sim$ Uniform(1,…,T) sample timestep

$\bm{\epsilon}\sim\mathcal{N}(\textbf{0},\textbf{I})$ sample noise

$\textbf{x}_{I}\leftarrow i_{\theta}(\textbf{y})$ initial prediction

$\textbf{r}_{t}=\sqrt{\bar{\alpha_{t}}}(\textbf{x}_{gt}-\textbf{x}_{I})+\sqrt{1-\bar{\alpha_{t}}}\bm{\epsilon}$
forward diffusion

$\tilde{\textbf{r}_{0}}=f_{\theta}(\textbf{r}_{t},t,\textbf{x}_{I})$ residual
estimation

$L_{\text{TOT}}(\textbf{x}_{gt},\textbf{x}_{I},\tilde{\textbf{r}_{0}})$ loss
computation

Take a gradient descend update on $i_{\theta},f_{\theta}$ using
$L_{\text{TOT}}$

RETURN $I_{\theta^{\star}}$, $f_{\theta^{\star}}$

###  III-D Fast Sampling Strategy

Diffusion models are very powerful generative models but their main weakness
lies in their generative slowness since sampling from them requires hundreds
or even thousands of neural network evaluations. One of the first works that
tries to speed up the generative process of DPM is the strategy introduced by
Song et al. in [18]. They introduced a non-Markovian definition of diffusion
models with the same DPM training objective, but whose reverse process can be
much faster to sample from. Surprisingly, their reverse kernel definition also
allows for a deterministic definition, called denoising diffusion implicit
model (DDIM):

$\begin{array}[]{r}\mathbf{r}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\tilde{\textbf{r}_{0}}+\sqrt{1-\bar{\alpha}_{t-1}}\frac{\textbf{r}_{t}-\sqrt{\bar{\alpha}_{t}}\tilde{\textbf{r}_{0}}}{\sqrt{1-\bar{\alpha}_{t}}}\end{array}$
(15)
$\tilde{\textbf{r}}_{0}=f_{\theta}\left(\textbf{r}_{t},t,\textbf{x}_{I}\right)$
(16)

This is one of the first works that successfully produce higher-quality images
with fewer denoising steps compared to DPM. Taking this approach as a
baseline, in our work, we adopt a different method to speed up the sampling
phase without sacrificing performance: we employ a fast high-order solver for
diffusion ODEs called DPM-solver [22]. It has a convergence order guarantee in
10-20 function evaluations. We decided to use this type of solver since
document enhancement does not require a high degree of diversity and modelling
the sampling phase from our DPM as solving the corresponding diffusion
ordinary differential equation (ODE) could be a suitable choice. The
formulation of DPM-solver assumes a continue-time DPM but we defined and
trained a discrete-time DPM. However, the authors described formally how to
switch from one configuration to another and give black-box coding support for
this operation. The overall sampling strategy is defined in Alg. 2.

Algorithm 2 Sampling from NAF-DPM

INPUT: $i_{\theta^{\star}}$ initial predictor, $f_{\theta^{\star}}$ denoiser
network, $T$ sampling steps, $\bar{\bm{\alpha}}_{0:T}$ noise scheduler, y
degraded image;

OUTPUT: restored images generated by NAF-DPM;

SAMPLING WITH DDIM:

$\textbf{x}_{I}\leftarrow i_{\theta}(\textbf{y})$ first prediction from
initial predictor

$\textbf{r}_{T}\sim\mathcal{N}(\textbf{0},\textbf{I})$ sample from normal
distribution

FOR $t\leftarrow T$ TO $1$ DO:

$\tilde{\textbf{r}}_{0}\leftarrow
f_{\theta}\left(\textbf{r}_{t},t,\textbf{x}_{I}\right)$ denoiser estimation

$\bm{\epsilon}_{t}\leftarrow\frac{\textbf{r}_{t}-\sqrt{\bar{\alpha}_{t}}\tilde{\textbf{r}_{0}}}{\sqrt{1-\bar{\alpha}_{t}}}$
noise computation

$\textbf{r}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\tilde{\textbf{r}}_{0}+\sqrt{1-\bar{\alpha}_{t-1}}\bm{\epsilon}_{t}\hskip
5.69046pt$ refinement

END FOR

RETURN $\textbf{x}_{I}+\textbf{r}_{0}$ initial prediction + residual image

SAMPLING WITH DPM-solver:

$\textbf{x}_{I}\leftarrow i_{\theta}(\textbf{y})$ first prediction from
initial predictor

$\textbf{r}_{T}\sim\mathcal{N}(\textbf{0},\textbf{I})$ sample from normal
distribution

$\textbf{r}_{0}\leftarrow$ DPM-solver( $\bar{\bm{\alpha}}_{0:T}$,
$f_{\theta}$(.), $\textbf{r}_{T}$, $T$, $\textbf{x}_{I}$) Residual

RETURN $\textbf{x}_{I}+\textbf{r}_{0}$ initial prediction + residual image

###  III-E Differentiable OCR-Guided Finetuning

The great generative power of the denoiser network is capable of recovering
high-frequency details from images semi-reconstructed from the initial
predictor. In the case of text documents, when sometimes source characters are
poorly defined, such power can lead to recovering wrong characters. In fact,
during experiments, we noticed that some characters were wrong, although well
reconstructed and well recognized by an OCR system. To limit this phenomenon
we introduce an additional loss term that tries to push the entire network to
reduce the errors in the characters. The main idea is to use a commercial OCR
system. Fed this OCR with the enhanced images from the network and computed
the Levenshtein distance between the extracted text and reference text. In the
end, propagate back the error to make the network aware of mistakes made on
characters. There are two problems with this approach: at first the
Levenshtein distance is not differentiable and so it can not be used as a loss
function.

Figure 5: Our proposed OCR-based finetuning enhances the character readability
and reduces the character error rate made by our network during the sampling
phase. We attach a CRNN module just after the output layer of our network. We
perform word-level text extraction and we feed the CRNN module with all the
extracted word-level image patches. The CRNN emulates the behaviour of a
commercial OCR and recognizes text from input patches. The text is compared to
the reference one and CTC Loss Function is computed. In the end, the weights
of our network are updated using eq. (18).

Then, commercial OCR systems could contain non-differentiable modules that
block the gradient backpropagation. To solve the first problem we employ
connectionist temporal classification (CTC) loss function. CTC is a machine
learning algorithm commonly used for sequence-to-sequence tasks. Given an
input sequence A and an output sequence B, the goal of the CTC is to learn a
mapping function $P(\textbf{{B}}|\textbf{A})$ that maximizes the probability
of the correct output sequence given the input sequence. To solve the second
problem we replace a commercial OCR system with a convolutional recurrent
neural network (CRNN [50]) that is pretrained to emulate the behaviour of the
OCR, following the work by Randika et al. [51].

As illustrated in Fig. 5, we place the CRNN just after the network. From the
restored image we extract word-level text that is fed into CRNN. The output of
this module is compared to reference text and CTC loss is computed. Since this
process requires additional training time with respect to traditional training
we decided to enable the CRNN components only after a predefined number of
iterations of standard training; a sort of finetuning strategy after having
trained the main network for a predefined number of iterations.

$L_{\text{CTC}}=\text{CTC(text}_{\text{extracted}},\text{text}_{\text{reference}})$
(17) $L_{\text{FINETUNE}}=L_{\text{TOT}}+L_{\text{CTC}}\vspace{0.1cm}$ (18)

##  IV Experiments and Results

###  IV-A Document Deblurring

We begin our experiments with document deblurring. For this task, the blurry
document images text dataset is used [1]. It is composed of 66,000 blurry text
images of size 300x300 with their corresponding ground truth counterpart. It
includes also another section composed of 94 images of size 512x512 used for
OCR testing. We randomly select 30,000 images for training and 10,000 images
for validating our model. During training, we randomly extract from each image
patches of size 128x128 performing also random horizontal flip and random
rotation. We jointly train the initial predictor network and denoiser network
for 800k iterations trying to reduce the loss function eq. (14). We fix the
timesteps hyperparameter to $T=100$. During sampling we use both DDIM sampling
algorithm, eq. (15), and DPM-solver, to compare the results coming out from
the two approaches. To assess network performance we use both pixel-level
metrics (PSNR and SSIM [52]), and perceptual similarity metrics (LPIPS [53]
and DISTS [54]).

TABLE I: Ablation studies on sampling methods on the Document Deblurring
Dataset Test Dataset (10k randomly selected 300x300 images) [1].

| Sampling
---
Method
| Sampling
---
Iterations
PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$) | DISTS ($\downarrow$) |
DDIM | 100 | 25.819 | 0.964 | 0.0235 | 0.0497 |
DPM-Solver | 100 | 27.405 | 0.974 | 0.0211 | 0.0406 |
DPM-Solver | 50 | 27.511 | 0.975 | 0.0207 | 0.0400 |
DPM-Solver | 20 | 27.588 | 0.975 | 0.0204 | 0.0398 |
DPM-Solver | 10 | 27.607 | 0.975 | 0.0207 | 0.0409 |

From the results shown in the Table I, we can highlight the superiority of the
sampling method introduced by [22]. This method is able to produce images of
superior quality from the point of view of all metrics examined. In
particular, the PSNR metric is much higher using DPM-solver, a sign that the
predicted images are more accurate with respect to the generated ones using
the DDIM sampling strategy. As warranted by the authors of [22], the method
converges in 10-20 sampling steps. As the number of steps increases, there is
a slight decrease in pixel-level metrics while perceptual ones remain stable.

Nevertheless, we will compare our approach with results achieved by state-of-
the-art methods in the document deblurring problem. For this type of analysis,
we use the document deblurring OCR test dataset that, as previously mentioned,
contains 94 images of size 500x500. We sample using a DPM-solver set at 20
sampling iterations. Results are listed in Table II. NAF-DPM beats all pre-
existing methods based on convolutional neural network (Hradis et al [1]),
conditional generative adversarial network (DE-GAN [8]) and diffusion
probabilistic model (DocDiff [10]) by a large margin. In particular, in
comparison with DocDiff, it can be seen a gain of more than 4db in PSNR
metrics and much more important, there is a clear halving on perceptual
similarity metrics (LPIPS and DISTS).

TABLE II: Results of image deblurring on OCR TEST Dataset (94 images of size
500x500) [1].

Method | Model | PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$) | DISTS ($\downarrow$)
---|---|---|---|---|---
Hradis [1] | CNN | 30.629 | 0.987 | 0.0135 | 0.0258
DE-GAN [8] | cGAN | 28.803 | 0.985 | 0.0144 | 0.0237
DocDiff [10] | DPM | 29.787 | 0.989 | 0.0094 | 0.0339
NAF-DPM | DPM | 34.377 | 0.994 | 0.0046 | 0.0228

Visual results can be appreciated from Fig. 6. It can be seen that images
produced by pure CNN and GAN networks have poorly defined text with blurred
and distorted edges (Hradis et al. [1] and DE-GAN [8]). The strategy “predict
and refine” adopted by DocDiff [10] and by our network allows more readable
text to be retrieved. Unlike DocDiff, our approach seems to be much more
effective in retrieving all the details and text edges. Credit can be given to
the choice of backbone networks. Using a designated and specialized network
for deburring and denoising tasks allows for more precise initial prediction
and consequently also for more specific refinement by the diffusion model. In
addition, we have also a gain in the sampling time, using DPM-solver we are
able to retrieve the vast majority of details in just 10-20 function
evaluations (in Fig. 6 images from our network sampled in 20 DPM-solver
sampling iterations) with respect to DocDiff that requires more than 50
function evaluation to make text edges smoother and sharper (in Fig. 6 images
from DocDiff sampled in 100 DDIM stochastic sampling iterations).

(a)  Figure 6: Document Deblurring: Comparison between our method and other
approaches. From left to right it can be seen the blurred images, and the
restored images by Hradis et al. [1], DE-GAN[8], DOC-DIFF [10], our network
and in the last column the original images. The characters restored by our
network are clearly more readable and accurate. The Generative Adversarial
Network approach produces confused and blurred characters that bring poor
performances from the OCR point of view. DOC-DIFF based on the approach
”Predict and refine” is able to produce more accurate images. The characters
are not so blurred as in DEGAN, however, some of them are substituted by the
wrong character. This behaviour can be due to the high diversity property of a
diffusion model. In our approach, a better suitable backbone network and an
improved training-finetuned strategy effectively reduce the occurrence of
substituted characters with a clear advantage for the accuracy of an OCR
system.

###  IV-B OCR Evaluation

In this section, we focus our attention on the performance of OCR on degraded
and enhanced documents. We compare the results from NAF-DPM with state-of-the-
art results in the document deblurring OCR test dataset [1] using
PaddleOCR111https://github.com/PaddlePaddle/PaddleOCR as OCR. We have manually
labelled the dataset since the original labelling present in [1] was done
using an old OCR system on clean images and indeed it presents a lot of errors
and missing characters. The results are shown in Table III. Our network is
able to substantially reduce the error on characters made by PaddleOcr,
irrefutable evidence that images are well restored and high-frequency details,
fundamentals for the right detection of characters, are preserved. We almost
halve the character error rate (CER) compared to DocDiff [10]. Compared to the
other two benchmarks we have a performance increase of 3.88x with respect to
Hradis et al. [1] and 4.90x with respect to DE-GAN [8]. These results define a
new state-of-the-art (SOTA) for document deblurring OCR test dataset.

In order to further reduce the CER metric, we finetune our network using an
additional CRNN network and the augmented loss function, eq. 18. The adopted
dataset is the Document Deblurring dataset [1]. From the patches we extract
text and corresponding bounding boxes using Paddle-OCR. The extraction is done
offline since it requires a non-negligible amount of time. We pretrain a CRNN
Network on the training patches and the correspondent extracted text for 20
epochs. In this way, the behaviour of PaddleOCR is emulated by a
differentiable CRNN module [50]. We take model checkpoint at 700k iterations
and we finetune it for an additional 100k, so that at the end we can compare
the performance of the two models at equal training state (i.e., 800k training
steps).

TABLE III: Character error rate comparison among different methods on document
deblurring OCR test dataset [1]

Method | PSNR ($\uparrow$) | CER ($\downarrow$)
---|---|---
Hradis [1] | 30.629 | 5.44
DE-GAN [8] | 28.803 | 6.87
DocDiff [10] | 29.787 | 2.78
NAF-DPM | 34.377 | 1.55

The addition of the CRNN module and the new loss function would seem to bring
marginal improvements in the CER metric (see Table IV). In particular, the
results show that this approach succeeds in better training the network and
making it more aware of the characters it is reconstructing. From both the
decreasing CER metric and the qualitative results shown in Fig. 7 we note that
there are some characters exactly reconstructed only in the finetuned version
of NAF-DPM.

TABLE IV: Character error rate comparison between different methods on
document deblurring OCR test dataset [1]

Method | CER ($\downarrow$)
---|---
NAF-DPM w/o finetuning | 1.55
NAF-DPM w/ finetuning | 1.40

Figure 7: Comparison between our method and other approaches on manually
selected words from images of document deblurring OCR test dataset [1]

.

###  IV-C Binarization

We continue our experiments by investigating the robustness of our method to a
different task: document binarization. We train and evaluate our network on
three datasets from the annual Document Image Binarization Competition
(DIBCO). In particular we choose DIBCO2017 [2], H-DIBCO2018 [3] and the
challenging DIBCO2019 [23]. These datasets contain degraded document images
with handwritten and printed text. DIBCO 2019 contains also ancient documents
written on papyrus, which makes this dataset highly challenging. In each
experiment, we pick a dataset as the test dataset (e.g. DIBCO2017) and we
construct the training dataset with the remaining data from annual DIBCOs
(from DIBCO2009 to DIBCO2019). We integrate the training material with the
Bickley Diary dataset [55], Persian Heritage Image Binarization Dataset
(PHIDB) [56], the Synchromedia Multispectral dataset (S-MS) [57] and Palm Leaf
dataset [58]. From each image of the training set we extract 256x256
overlapped patches following the work in [9] and for each experiment, we
jointly train the initial predictor and denoiser for 100000 iterations. We
sample final images using DPM-solver set at 11 sampling iterations.

####  IV-C1 Quantitative Evaluation

Standard metrics used to evaluate the performance of the model are: Peak
signal-to-noise ratio (PSNR), F-measure and pseudo-F-measure ($F_{ps}$).

Table V shows the comparison results of different methods on DIBCO 2017 test
set [2]. Out of these results, we can say that our method is superior to the
current state-of-the-art methods according to PSNR metric. DE-GAN [8] keeps
the best performance in F-measure and $F_{ps}$, although the official results
presented by the authors would seem to outperform the trend of various models
over the years by a large margin. Despite this, NAF-DPM establishes itself as
the best among the others, demonstrating great robustness and stability on the
various degradations present in these documents.

TABLE V: Results of image binarization on DIBCO 2017 and H-DIBCO 2018 Databases [2, 3]. NAF-DPM achieves excellent and consistent results in the various examined datasets. The best algorithms of competitions [2, 3] use pre-processing and post-processing steps to achieve good performance. Jemni et al use additional pertaining data to specialize their network to handwritten document enhancement. In bold the best results, underlined the second best.  Method | Model | Parameters |  Additional Processing | DIBCO 2017 | DIBCO 2018
---|---|---|---|---|---
PSNR($\uparrow$) | F-Measure($\uparrow$) | Fps($\uparrow$) | PSNR($\uparrow$) | F-Measure($\uparrow$) | Fps($\uparrow$)
Competition Top [2, 3] | CNN | - | ✓ | 18.28 | 91.04 | 92.86 | 19.11 | 88.34 | 90.24
Jemni et al. [59] | cGAN | 15.8M | ✓ | 17.45 | 89.80 | 89.95 | 20.18 | 92.41 | 94.35
Otsu [31] | Thresh | - | ✗ | 13.85 | 77.73 | 77.89 | 9.74 | 51.45 | 53.05
Sauvola [32] | Thresh | - | ✗ | 14.25 | 77.11 | 84.10 | 13.78 | 67.81 | 74.08
DE-GAN [8] | cGAN | 31M | ✗ | 18.74 | $\textbf{97.91}^{\ast}$ | 98.23∗ | 16.16 | 77.59 | 85.74
DocEnTr [9] | Vi-Trans | 68M | ✗ | 19.11 | 92.53 | 95.15 | 19.46 | 90.59 | 93.97
D2BFormer [60] | Vi-Trans | 194M | ✗ | 19.35 | 93.52 | 95.09 | 18.91 | 88.84 | 93.42
DocDiff [10] | DPM | 8.2M | ✗ | - | - | - | 17.92 | 88.11 | 90.43
NAF-DPM | DPM | 9.4M | ✗ | 19.40 | 93.55 | 95.76 | 19.67 | 90.64 | 94.51

We continue the quantitative analysis of our model on the most recent H-DIBCO
2018 [3] and DIBCO 2019 [23] datasets. In H-DIBCO 2018 [3] our network
achieves state-of-the-art performance in terms of pseudo-F-Measure, while the
model introduced by Jemini et al. [59] keeps the best results in PSNR and
F-Measure. It is important to note that their model is designed to restore
handwritten documents and uses additional training data. Despite this, NAF-DPM
can reach similar results without any additional training data, pre-
processing, post-processing stages, or fine-tuning strategies. Results are
shown in Table V.

Regarding the challenging DIBCO 2019 [23], our approach clearly reaches top
performance in all the metrics, beating both general purpose binarization
algorithm [31, 32] and specifically designed algorithm for this kind of
dataset [23, 60]. Results are shown in Table VI.

This study leads us to say that our method turns out to be very robust to
different forms of degradation. In each dataset, it excels in at least one
metric and performs as the first or second best network each time, unlike
other networks, such as DEGAN[8], which turn out to be SOTA for one dataset
(DEGAN-DIBCO2017) but perform worse in other datasets (H-DIBCO2018 and
DIBCO2019). The “predict and refine” strategy is also in this case very
effective and shows clearly the superiority with respect to common and well-
known approaches such as generative adversarial networks (GANs) and vision
transformers.

It is interesting to note the increase in performance achieved by NAF-DPM
against DocDiff [10], which is also based on DPM. The design of dedicated
nonlinear activation-free networks for the initial predictor and the denoiser
along with the use of an ODE fast solver allows to unlock the full potential
of DPMs for these types of tasks. This is confirmed by the clear margin in the
PSNR metrics in both the H-DIBCO 2018 [3] dataset (from 17.92db to 19.60db)
and in DIBCO 2019 [23](from 15.14db to 15.40db).

TABLE VI: Results of image binarization on DIBCO 2019 Database [23]. NAF-DPM outperforms all other methods in all metrics, demonstrating a great power of generalization. In bold the best results, underlined the second best. Method | Model | DIBCO 2019
---|---|---
PSNR($\uparrow$) | F-Measure($\uparrow$) | Fps($\uparrow$)
Otsu [31] | Thresh | 9.08 | 47.83 | 45.59
Sauvola [32] | Thresh | 13.72 | 51.73 | 55.15
Competition Top | CNN | 14.48 | 72.88 | 72.15
DE-GAN [8] | cGAN | 12.29 | 55.98 | 53.44
D2BFormer [60] | Vi-Trans | 15.05 | 67.63 | 66.69
DocDiff [10] | DPM | 15.14 | 73.38 | 75.12
NAF-DPM | DPM | 15.39 | 74.61 | 76.25

To strengthen the power of our approach, we can analyze the results obtained
in relation to the number of parameters of the various networks. In Table V,
we can appreciate that the performance of networks is not correlated with the
increasing number of their parameters. Our approach, together with DocDiff
[10], managed to achieve very good results while keeping the number of
parameters very small. Our network is much more powerful than DE-GAN [8] by
having one-third of the parameters. It manages to exceed the performance of
the best-known vision transformer approaches dedicated to this task having
about one-sixth of the parameters of DocEnTr [9] and even one-twentieth
compared to D2BFormer [60]. Acronyms used in Tables V, VI: Thresh:
Thresholding, CNN: Convolutional Neural Network, cGAN: Conditioned Generative
Adversarial Network, Vi-Trans: Vision Transformer, DPM: Diffusion
Probabilistic Model.

####  IV-C2 Qualitative Evaluation

In Figure 8 there are visual results from 3 different patches of DIBCO2017.
Binarized images are very similar to ground truth. This confirms the high
values of PSNR and F-Measure.

Figure 8: Visual results. Three random extracted patches from DIBCO 2017 [2].
From left to right we find Degraded images, Restored images, and ground truth
images. Reconstructed images are almost identical to ground truth.

We conduct also a qualitative analysis of our network and other different
methods. In Figure 9 we compare an image from DIBCO2018 [3] enhanced by our
network and the same image processed by two basic thresholding techniques
(Otsu [31] and Sauvola [32]), a conditional-GAN (DE-GAN [8]) and a vision
transformer (DocEnTr [9]). It can be seen that dynamic threshold methods fail
to precisely restore all portions of the image. Their small receptive field
(25x25 sliding windows) does not allow them to get a complete view of the
image. DEGAN [8] succeeds in producing a good binarized version; however,
there are some areas where the network demonstrates poor accuracy and poor
generalization. The images enhanced by DocEnTR [9] and by our method are
almost overlapping and very similar to the ground truth.

Figure 9: Qualitative Result offered by different models on sample number 9 of
H-DIBCO 2018 [3].

##  V Conclusion

In this work, we propose NAF-DPM, a framework that combines a convolutional
neural network (CNN) and a diffusion probabilistic model (DPM) following the
strategy “predict and refine” with the final aim of restoring the original
quality of degraded documents. As backbone network for the initial predictor
and the diffusion model, we design an effective variant of a nonlinear
activation-free network (NAFNet). To mitigate the slow sampling rate of
diffusion models we employ an ODE fast solver which is able to converge in a
maximum of 20 iterations. To better retain text characters, we introduced an
additional module based on convolutional recurrent neural networks (CRNN) to
simulate the behavior of an OCR system during the training phase. We test our
approach on two use cases: document deblurring and document binarization. In
both tasks, NAF-DPM produces high-quality samples with a few sampling
iterations beating all previous powerful models on pixel-level and perceptual
similarity metrics. It also significantly reduces character errors when
commercial OCR systems transcribe the enhanced images. Given the results
obtained with NAF-DPM and considering the significant generalization power of
diffusion models, possible future investigations might involve applying the
method to other related tasks and developing a unified end-to-end framework
for all the different document enhancement tasks. In particular, this latter
work is important to reduce the number of different models and to decrease the
training or finetuning phases needed to adapt the model to the various tasks.

## References

  * [1] M. Hradiš, J. Kotera, P. Zemčík, and F. Šroubek, “Convolutional neural networks for direct text deblurring,” in _Proceedings of BMVC 2015_. The British Machine Vision Association and Society for Pattern Recognition, 2015. [Online]. Available: http://www.fit.vutbr.cz/research/view_pub.php?id=10922
  * [2] I. Pratikakis, K. Zagoris, G. Barlas, and B. Gatos, “Icdar2017 competition on document image binarization (dibco 2017),” in _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_ , vol. 01, 2017, pp. 1395–1403.
  * [3] I. Pratikakis, K. Zagori, P. Kaddas, and B. Gatos, “Icfhr 2018 competition on handwritten document image binarization (h-dibco 2018),” in _2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR)_ , 2018, pp. 489–493.
  * [4] Z. Anvari and V. Athitsos, “A survey on deep learning based document image enhancement,” _ArXiv_ , vol. abs/2112.02719, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:244908637
  * [5] Y. Zhou, S. Zuo, Z. Yang, J. He, J. Shi, and R. Zhang, “A review of document image enhancement based on document degradation problem,” _Applied Sciences_ , vol. 13, no. 13, 2023. [Online]. Available: https://www.mdpi.com/2076-3417/13/13/7855
  * [6] A. Sulaiman, K. Omar, and M. F. Nasrudin, “Degraded historical document binarization: A review on issues, challenges, techniques, and future directions,” _Journal of Imaging_ , vol. 5, no. 4, 2019. [Online]. Available: https://www.mdpi.com/2313-433X/5/4/48
  * [7] T. Lelore and F. Bouchara, “Fair: A fast algorithm for document image restoration,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 8, pp. 2039–2048, 2013.
  * [8] S. Lin, Z. He, and L. Sun, “Defect enhancement generative adversarial network for enlarging data set of microcrack defect,” _IEEE Access_ , vol. 7, pp. 148 413–148 423, 2019.
  * [9] M. Souibgui, S. Biswas, S. Jemni, Y. Kessentini, A. Fornes, J. Llados, and U. Pal, “Docentr: An end-to-end document image enhancement transformer,” in _2022 26th International Conference on Pattern Recognition (ICPR)_. Los Alamitos, CA, USA: IEEE Computer Society, aug 2022, pp. 1699–1705. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICPR56361.2022.9956101
  * [10] Z. Yang, B. Liu, Y. Xxiong, L. Yi, G. Wu, X. Tang, Z. Liu, J. Zhou, and X. Zhang, “Docdiff: Document enhancement via residual diffusion models,” in _Proceedings of the 31st ACM International Conference on Multimedia_ , ser. MM ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 2795–2806. [Online]. Available: https://doi.org/10.1145/3581783.3611730
  * [11] X.-J. Mao, C. Shen, and Y. Yang, “Image denoising using very deep fully convolutional encoder-decoder networks with symmetric skip connections,” _ArXiv_ , vol. abs/1603.09056, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:1773169
  * [12] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _CoRR_ , vol. abs/1501.00092, 2015. [Online]. Available: http://arxiv.org/abs/1501.00092
  * [13] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [14] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [15] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of GANs for improved quality, stability, and variation,” in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/forum?id=Hk99zCeAb
  * [16] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, “Generative adversarial nets,” in _Neural Information Processing Systems_ , 2014. [Online]. Available: https://api.semanticscholar.org/CorpusID:261560300
  * [17] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 6840–6851. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf
  * [18] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in _International Conference on Learning Representations_ , 2021. [Online]. Available: https://openreview.net/forum?id=St1giarCHLP
  * [19] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, “Palette: Image-to-image diffusion models,” in _ACM SIGGRAPH 2022 Conference Proceedings_ , ser. SIGGRAPH ’22. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3528233.3530757
  * [20] P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 8780–8794. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/49ad23d1ec9fa4bd8d77d02681df5cfa-Paper.pdf
  * [21] L. Chen, X. Chu, X. Zhang, and J. Sun, “Simple baselines for image restoration,” _arXiv preprint arXiv:2204.04676_ , 2022.
  * [22] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps,” in _Advances in Neural Information Processing Systems_ , A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022. [Online]. Available: https://openreview.net/forum?id=2uAaGwlP_V
  * [23] I. Pratikakis, K. Zagoris, X. Karagiannis, L. Tsochatzidis, T. Mondal, and I. Marthot-Santaniello, “Icdar 2019 competition on document image binarization (dibco 2019),” in _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , 2019, pp. 1547–1556.
  * [24] G. Panci, P. Campisi, S. Colonnese, and G. Scarano, “Multichannel blind image deconvolution using the bussgang algorithm: spatial and multiresolution approaches,” _IEEE Transactions on Image Processing_ , vol. 12, no. 11, pp. 1324–1337, 2003.
  * [25] H. Cho, J. Wang, and S. Lee, “Text image deblurring using text-specific properties,” in _Computer Vision – ECCV 2012_ , vol. 7576, 10 2012, pp. 524–537.
  * [26] J. Pan, Z. Hu, Z. Su, and M.-H. Yang, “Deblurring text images via l0-regularized intensity and gradient prior,” in _2014 IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 2901–2908.
  * [27] O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, “Deblurgan: Blind motion deblurring using conditional adversarial networks,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Los Alamitos, CA, USA: IEEE Computer Society, jun 2018, pp. 8183–8192. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR.2018.00854
  * [28] H. Neji, M. B. Halima, T. M. Hamdani, J. Nogueras-Iso, and A. M. Alimi, “Blur2sharp: A gan-based model for document image deblurring,” _International Journal of Computational Intelligence Systems_ , vol. 14, pp. 1315–1321, 2021. [Online]. Available: https://doi.org/10.2991/ijcis.d.210407.001
  * [29] S. Gonwirat and O. Surinta, “Deblurgan-cnn: Effective image denoising and recognition for noisy handwritten characters,” _IEEE Access_ , vol. 10, pp. 90 133–90 148, 2022.
  * [30] Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6228–6237.
  * [31] N. Otsu, “A threshold selection method from gray-level histograms,” _IEEE Transactions on Systems, Man, and Cybernetics_ , vol. 9, no. 1, pp. 62–66, 1979.
  * [32] J. Sauvola and M. Pietikäinen, “Adaptive document image binarization,” _Pattern Recognition_ , vol. 33, no. 2, pp. 225–236, 2000. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0031320399000552
  * [33] P. V. Bezmaternykh, D. A. Ilin, and D. P. Nikolaev, “U-Net-bin: hacking the document image binarization contest,” _Computer Optics_ , vol. 43, no. 5, pp. 825–832, Oct. 2019.
  * [34] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 4, pp. 4713–4726, 2023.
  * [35] H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen, “Srdiff: Single image super-resolution with diffusion probabilistic models,” _Neurocomputing_ , vol. 479, pp. 47–59, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231222000522
  * [36] J. Whang, M. Delbracio, H. Talebi, C. Saharia, A. G. Dimakis, and P. Milanfar, “Deblurring via stochastic refinement,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 16 293–16 303.
  * [37] S. Shang, Z. Shan, G. Liu, and J. Zhang, “Resdiff: Combining cnn and diffusion model for image super-resolution,” _ArXiv_ , vol. abs/2303.08714, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:257532290
  * [38] Q. Zhang and Y. Chen, “Fast sampling of diffusion models with exponential integrator,” in _The Eleventh International Conference on Learning Representations_ , 2023. [Online]. Available: https://openreview.net/forum?id=Loek7hfb46P
  * [39] S. Xue, M. Yi, W. Luo, S. Zhang, J. Sun, Z. Li, and Z.-M. Ma, “Sa-solver: Stochastic adams solver for fast sampling of diffusion models,” 2023.
  * [40] T. Salimans and J. Ho, “Progressive distillation for fast sampling of diffusion models,” in _International Conference on Learning Representations_ , 2022. [Online]. Available: https://openreview.net/forum?id=TIdIXIpzhoI
  * [41] A. Sauer, D. Lorenz, A. Blattmann, and R. Rombach, “Adversarial diffusion distillation,” 2023.
  * [42] C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans, “On distillation of guided diffusion models,” in _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Los Alamitos, CA, USA: IEEE Computer Society, jun 2023, pp. 14 297–14 306. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR52729.2023.01374
  * [43] D. Kingma, T. Salimans, B. Poole, and J. Ho, “Variational diffusion models,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 21 696–21 707. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/b578f2a52a0229873fefc2a4b06377fa-Paper.pdf
  * [44] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Los Alamitos, CA, USA: IEEE Computer Society, jun 2022, pp. 10 674–10 685. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.01042
  * [45] A. Vahdat, K. Kreis, and J. Kautz, “Score-based generative modeling in latent space,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 11 287–11 302. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/5dca4c6b9e244d24a30b4c45601d9720-Paper.pdf
  * [46] M. Fritsche, S. Gu, and R. Timofte, “Frequency separation for real-world super-resolution,” _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , pp. 3599–3608, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:208158302
  * [47] K.-U. Song, D. Shim, K.-W. Kim, J. young Lee, and Y.-G. Kim, “Fs-ncsr: Increasing diversity of the super-resolution space via frequency separation and noise-conditioned normalizing flow,” _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pp. 967–976, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:248299786
  * [48] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang, “Diffusion models: A comprehensive survey of methods and applications,” _ACM Comput. Surv._ , vol. 56, no. 4, nov 2023. [Online]. Available: https://doi.org/10.1145/3626235
  * [49] Z. Luo, F. K. Gustafsson, Z. Zhao, J. Sjölund, and T. B. Schön, “Refusion: Enabling large-size realistic image restoration with latent-space diffusion models,” in _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2023, pp. 1680–1691.
  * [50] B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, pp. 2298–2304, 2015. [Online]. Available: https://api.semanticscholar.org/CorpusID:24139
  * [51] A. Randika, N. Ray, X. Xiao, and A. Latimer, “Unknown-box approximation to improve optical character recognition performance,” in _Document Analysis and Recognition – ICDAR 2021_ , J. Lladós, D. Lopresti, and S. Uchida, Eds. Cham: Springer International Publishing, 2021, pp. 481–496.
  * [52] A. Horé and D. Ziou, “Image quality metrics: Psnr vs. ssim,” in _2010 20th International Conference on Pattern Recognition_ , 2010, pp. 2366–2369.
  * [53] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 586–595, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:4766599
  * [54] K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, “Image quality assessment: Unifying structure and texture similarity,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, pp. 2567–2581, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:215785896
  * [55] F. Deng, Z. Wu, Z. Lu, and M. Brown, “Binarizationshop: A user-assisted software suite for converting old documents to black-and-white,” in _JCDL’10 - Digital Libraries - 10 Years Past, 10 Years Forward, a 2020 Vision_ , ser. Proceedings of the ACM International Conference on Digital Libraries, 2010, pp. 255–258, copyright: Copyright 2010 Elsevier B.V., All rights reserved.; 10th Annual Joint Conference on Digital Libraries, JCDL 2010 ; Conference date: 21-06-2010 Through 25-06-2010.
  * [56] H. Nafchi, S. Ayatollahi, R. Farrahi Moghaddam, and M. Cheriet, “An efficient ground truthing tool for binarization of historical manuscripts,” 08 2013, pp. 807–811.
  * [57] R. Hedjam, H. Z. Nafchi, R. F. Moghaddam, M. Kalacska, and M. Cheriet, “Icdar 2015 contest on multispectral text extraction (ms-tex 2015),” in _2015 13th International Conference on Document Analysis and Recognition (ICDAR)_ , 2015, pp. 1181–1185.
  * [58] J.-C. Burie, M. Coustaty, S. Hadi, M. W. A. Kesiman, J.-M. Ogier, E. Paulus, K. Sok, I. M. G. Sunarya, and D. Valy, “Icfhr2016 competition on the analysis of handwritten text in images of balinese palm leaf manuscripts,” _2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR)_ , pp. 596–601, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:1165182
  * [59] S. Khamekhem Jemni, M. A. Souibgui, Y. Kessentini, and A. Fornés, “Enhance to read better: A multi-task adversarial network for handwritten document image enhancement,” _Pattern Recognition_ , vol. 123, p. 108370, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0031320321005501
  * [60] M. Yang and S. Xu, “A novel degraded document binarization model through vision transformer network,” _Information Fusion_ , vol. 93, pp. 159–173, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1566253522002597