# Beyond Image Prior: Embedding Noise Prior
into Conditional Denoising Transformer

Yuanfei Huang, , and Hua Huang  The authors are with the School of Artificial
Intelligence, Beijing Normal University, Beijing, 100875, China. E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>(Corresponding author: Hua Huang.)

###### Abstract

Existing learning-based denoising methods typically train models to generalize
the image prior from large-scale datasets, suffering from the variability in
noise distributions encountered in real-world scenarios. In this work, we
propose a new perspective on the denoising challenge by highlighting the
distinct separation between noise and image priors. This insight forms the
basis for our development of conditional optimization framework, designed to
overcome the constraints of traditional denoising framework. To this end, we
introduce a Locally Noise Prior Estimation (LoNPE) algorithm, which accurately
estimates the noise prior directly from a single raw noisy image. This
estimation acts as an explicit prior representation of the camera sensor’s
imaging environment, distinct from the image prior of scenes. Additionally, we
design an auxiliary learnable LoNPE network tailored for practical application
to sRGB noisy images. Leveraging the estimated noise prior, we present a novel
Conditional Denoising Transformer (Condformer), by incorporating the noise
prior into a conditional self-attention mechanism. This integration allows the
Condformer to segment the optimization process into multiple explicit
subspaces, significantly enhancing the model’s generalization and flexibility.
Extensive experimental evaluations on both synthetic and real-world datasets,
demonstrate that the proposed method achieves superior performance over
current state-of-the-art methods. The source code is available at
https://github.com/YuanfeiHuang/Condformer.

###### Index Terms:

Image denoising, vision Transformer, noise modeling, conditional optimization.

##  1 Introduction

Image denoising, a fundamental aspect of low-level vision, is garnering
increased interest due to its significant applications in computational
photography and computer vision. The primary goal of image denoising is to
mitigate the impact of unwanted noise in noisy observations, thereby enhancing
image quality for either aesthetic enhancement or to facilitate subsequent
processing tasks.

In general, natural images embody strong priors for visual perception [1, 2],
such as repetitive textures and continuous edges, which are more readily
inferred than the seemingly random presence of noise. Thus, a prevailing
strategy among many existing learning-based denoising methods [3, 4, 5, 6, 7,
8, 9] involves developing a unified model capable of generalizing from a vast
collection of noisy-clean pairs. This process typically formulates
optimization as:

$\hat{\theta}=\arg\max_{\theta}\mathbb{E}_{x}\log P(y|x;\theta)$ (1)

where $\theta$ represents the model parameters, $x$ and $y$ denote the noisy
observation and clean target, respectively. In this context, the noise map is
treated as an additive mask over the clean image, with the ultimate aim of
deducing the underlying image prior. However, the noise prior, which is
crucial for distinguishing between various noise distributions, is often
overlooked.

(a)

(b)

Figure 1: Illustration of denoising model optimization: (a) unconditional
optimization space with image prior [8, 9]; (b) conditional optimization with
noise prior and image prior in this work.

This framework faces limitations in real-world scenarios due to two primary
challenges:

1) The difficulty and cost associated with gathering large-scale noisy-clean
image datasets. Learning sophisticated image priors necessitates a model with
substantial capacity, which is hindered by the challenges of acquiring clean
images. Clean image acquisition typically requires long exposures in static
scenes [10, 11, 12] or involves complex alignment procedures [10].

2) The inefficiency and incompleteness of an unconditional optimization space.
Conventional learning-based denoising methods, which focus solely on image
priors, are inherently unconditional. However, as illustrated in Fig. 1a,
these methods attempt to learn and generalize the image prior from numerous
noisy-clean samples, resulting in an optimization space that is both overly
broad, encompassing unnecessary scenarios, and simultaneously incomplete,
missing critical outlier cases.

To address these issues, we propose segmenting the singular unconditional
optimization space into multiple subspaces by incorporating reliable noise
priors alongside the complicated image priors, as illustrated in Fig. 1b.This
approach recognizes that a noisy observation is influenced both by the scene
(image prior) and the imaging environment (noise prior), making it logical to
infer the noise prior for optimizing the denoising process. Consequently, the
optimization space for a conditional denoising model should comprise various
independent and complete subspaces, each conditioned on specific priors, and
can be represented as:

$\hat{\theta}=\arg\max_{\theta}\sum_{i=1}^{n}\mathbb{E}_{x_{i}}\log
P(y|x_{i},z_{i};\theta)$ (2)

where $\\{z_{i}\\}^{n}_{i=1}$ represents the noise prior.

Distinct from traditional models that rely on given conditional embeddings
(e.g., cGAN [13], cVAE [14] and DnCNN [15]) or implicit noise prediction
(e.g., VDN [16] and CVF-SID [17]), a conditional denoising model must adeptly
estimate an explicit noise prior from a single noisy observation and
distinctly address the noise and image priors based on their independence. In
essence, as depicted in Fig. 1b, the principle of a conditional denoising
model lies in its ability to navigate the generation of pixels by harnessing
implicit natural image priors to shape the optimization landscape, while also
leveraging explicit sensor noise priors to precisely target the optimization’s
focus.

Building upon this concept, we introduce a novel approach for explicit noise
prior estimation from a single noisy observation, termed Locally Noise Prior
Estimation (LoNPE), and develop a Conditional denoising Transformer
(Condformer) that incorporates this noise prior. This integration allows for
the segmentation of the entire optimization space into distinct, explicit
optimization subspaces. Our main contributions are summarized as follows:

  * •

By rethinking the imaging mechanism in physics, we offer a new perspective on
image denoising, highlighting the independence between noise and image priors.
This distinction is crucial for conditional optimization, particularly within
the context of real-world scenarios.

  * •

We introduce an innovative LoNPE algorithm for estimating noise prior from raw
noisy image. This method effectively captures the characteristics of sensor
noise, providing an explicit prior for conditional optimization. Additionally,
we present a learnable LoNPE network, tailored for practical application with
only single sRGB noisy observation.

  * •

By exploring the noise statistics concealing in the latent space, we propose a
novel Condformer that leverages the estimated noise prior within a conditional
self-attention module. This design represents a pioneering effort to
incorporate prior knowledge into a Transformer-like architecture for
denoising.

  * •

Quantitative and qualitative experiments demonstrate the superior performance
of our LoNPE algorithm and Condformer model across various real and synthetic
noise analysis and image denoising tasks.

The rest of this paper is organized as follows: Section 2 reviews related
work. Section 3 presents our noise prior estimation and conditional image
denoising methods. Qualitative and quantitative experiments are reported and
analyzed in Section 4. Finally, Section 5 concludes our work and discusses the
limitation and future work.

##  2 Related Work

As the core goal of this paper is to explore a conditional denoising
Transformer with explicit noise prior, we next mainly introduce the advances
in the fields of noise modeling, image denoising, and vision Transformer,
respectively.

###  2.1 Noise Modeling

In general, the signal-independent Gaussian distribution is regarded as a
theoretically common hypothesis of noise modeling, and has derived numerous
supervised image denoising methods to handle the widely-used additive white
Gaussian noise (AWGN). However, the real noise model is more sophisticated.
Particularly, due to the characteristics of imaging sensor, the practical
noise could be explicitly modeled on raw sensors, and the corresponding raw
sensor noise commonly consists of the signal-dependent shot noise and the
signal-independent read noise. Typically, the Poisson-Gaussian noise model
[18] was employed to characterize the distribution of this raw sensor noise
and has inspired numerous advances in realistic noise synthesis and real image
denoising [19, 20, 21, 17]. Beyond the hypothesis on building the distribution
of noise, to match the noise model in more complex imaging environment on
various devices, multi-frame calibration [22, 18, 21] and variance-stabilizing
transformations [23, 24] technologies were presented to refine the noise model
for different sensors. Furthermore, some works attempt to employ the
calibrated noise parameter of sensor to explicitly synthesize noisy samples
[21, 25, 26], or guide the network optimization [16, 17, 27, 28] for training
a denoising model. Instead of explicitly modeling noise from a certain
distribution, learnble noise modeling methods recently have been raised with
the development of generative models, such as variational bayes [29],
generative adversarial networks [30, 31], and normalizing flows [32, 33, 34].

###  2.2 Image Denoising

After decades of development, image denoising methods are generally divided
into model-based and learning-based. Model-based denoising methods aim to
model the characteristics of natural images as a regularization prior to
iteratively optimize a well-designed model. The representative regularization
priors include total variation [35], sparsity [36], non-local self-similarity
[37, 38], and external statistical priors [39]. On the other hands, learning-
based methods attempt to reconstruct a clean image from the noisy observation
with an end-to-end learnable model, which is trained from large-scale noisy-
clean pairs. Initially, with the development of convolutional neural networks
(CNN), CNN-based denoising methods [15, 40, 3, 41, 5, 4] have received
significant advances in learning an end-to-end mapping from the noisy
observations to clean targets. In essence, due to the nature of CNN in local
visual perception, the key of these CNN-based denoising methods is to learn
how a pixel is generated from a corrupted one and its neighbors in local
perception region, namely, to learn the image prior in local perceptions, such
as details in texture, smoothness in flat. Nevertheless, some contextual image
priors are difficult to capture in local perceptions, e.g., objects,
structural information, edges and repeated textures. Recently, considering
these image priors in non-local or global visual perception, several
Transformer-based [42, 6, 7], MLP-based [43, 44, 45] and Mamba-based [46, 47]
denoising methods have attracted increasing attentions and achieved remarkably
superior performances against other existing CNN-based methods. In particular,
to capture long-range feature dependencies, pyramid [48], rectangle-
window[49], chaotic-window [50], sparse [51] and anchored-stripe [52] self-
attention mechanisms have been explored. Yet capturing spatial correspondence
commonly causes quadratically increasing computational loads as the resolution
increases, then efficient Transformer-based denoisers recently achieve growing
concerns. Specifically, hierarchical U-shape architecture [8] and channel-wise
self-attention mechanism [9] were proposed to reduce the unbearable
computational loads from increasing spatial resolutions.

Except for the evolution of denoiser architectures, a blind denoising strategy
with stronger generalization is essential for practical applications due to
the unknowability and the diversity of noise in real scene. In the type of
aforementioned supervised learning-based methods, numerous noisy-clean pairs
with various noise levels are employed to train an unified denoiser [15, 53,
9, 54]. Besides, to adapt for real scenes with only noisy observations, self-
supervised denoising methods [2, 17, 55] have been raised by learning implicit
representation from image priors. However, they often fall short in scenarios
requiring high accuracy, complex noise handling, and stable convergence.

###  2.3 Vision Transformer

The self-attention mechanism [42] in Transformers facilitates learning long-
range dependencies, leading to significant success in natural language
processing [42, 56] and computer vision tasks [57, 58]. For high-level vision,
ViT [57] demonstrated the Transformer’s effectiveness in non-local visual
perception and high-accuracy object recognition, sparking the development of
more effective and efficient Transformer architectures for various vision
tasks like object detection [58, 59], semantic segmentation [60, 61], and low-
level vision [6, 9].

Especially in low-level vision, self-attention mechanism was firstly utilized
to transfer relevant textures in reference-based super-resolution [62]. More
generally, IPT [6] later introduced a multi-task image processing model using
standard Transformer architecture with tokenized inputs. By integrating the
advantage of local attention mechanism of CNN and long-range dependency of
Transformer, Swin Transformer [58] was proposed by introducing the shifted
window scheme and was applied into image restoration tasks [7]. To alleviate
the limitation of Swin Transformer in receptive fields, cross aggregation
Transformer (CAT) [49] was proposed by aggregating features cross different
windows to expand the receptive field. Despite their effectiveness, these
Transformers are computationally intensive as calculating the spacial cross-
covariance of large-scale tokens. To address this, Uformer [8] was proposed by
building a hierarchical U-shape architecture with locally-enhanced window
Transformer blocks. Besides, by calculating the channel-wise cross-covariance,
an efficient Restormer [9] was proposed and achieved state-of-the-art
performances in several image restoration tasks.

Nonetheless, these existing Transformers rely on large-scale datasets with
perfect labels and unconditional optimization, which is commonly redundant.
Instead, inspired by the conditional text generation with Transformer [63,
64], we aim to explore a conditional Transformer for image denoising.

Figure 2: Investigation on independence of the image prior and the noise
prior. We select three noisy-clean pairs from SIDD training dataset, which are
captured with different image priors (related to scene and brightness) or
noise priors (related to ISO level and shutter speed). By investigating the
local variance of noisy residues and the local mean of clean image, the
statistical results show that the noise prior depends on only the camera
settings, but little on the image prior.

##  3 Method

In an imaging pipeline with a photosensor, the target imaging scene is
formulated as incident lights hitting the camera sensor array and then
transformed into digital responses for imaging. In this section, we first
introduce the noise formation model in an imaging sensor and generalize the
independence of noise and image priors, then describe the proposed LoNPE
algorithm/network for noise prior estimation and the Condformer architecture
for conditional denoising.

###  3.1 Preliminary on Noise Prior

####  3.1.1 Noise Formation Model

Although the noise in a processed sRGB image is generally complex to
explicitly analyze due to the nonlinearity of image signal processing (ISP),
the raw noise formation model of a digital sensor in camera is well understood
[65]. In particular, the raw noise in a camera primarily consists of the shot
noise during photon-to-electron conversion and the read noise during electron-
to-digital conversion [25].

Specifically, due to the quantum nature of light, the collected noisy
photoelectrons can be modeled as Poisson random variable, which follows

$(\mathbf{L}+\mathbf{N}_{\text{s}})\sim\mathcal{P}(\mathbf{L})$ (3)

where $\mathbf{L}$ and $\mathbf{N}_{\text{s}}$ indicate the incident clean
photoelectron and the shot noise, respectively. $\mathcal{P}(\cdot)$ denotes
the Poisson distribution.

These photoelectrons are subsequently read out as quantizable digital signals,
and commonly attached with the read noises $\mathbf{N}_{\text{r}}$, which can
be approximately modeled as Gaussian random variables

$\mathbf{N}_{\text{r}}\sim\mathcal{N}(\mathbf{0},\sigma^{2}_{\text{r}})$ (4)

where $\mathcal{N}(\cdot)$ denotes the Gaussian distribution.

This Poisson-Gaussian can be further treated as a heteroscedastic Gaussian
distribution [18], and the final raw digital sensor signals can be formulated
as

$\mathbf{I}\sim\mathcal{N}(\mathbf{L},\sigma^{2}_{\text{s}}\cdot\mathbf{L}+\sigma^{2}_{\text{r}})$
(5)

where $\sigma_{\text{s}}$ and $\sigma_{\text{r}}$ indicate the “noise prior”,
which depends on the imaging environments, including the camera sensor, and
photography settings.

In this manner, the noise prior plays a significant role in raw sensor noise
modeling, and is commonly proportional to the noise level. Particularly,
$\mathbf{L}$ indicates the pixel-wise intensity of the target scene
illuminance, indicating the “image prior”, and thereby has nothing with the
noise prior.

####  3.1.2 Independence of Noise Prior and Image Prior

To further illustrate the independence of noise prior and image prior, as
shown in Fig. 2, we investigate several raw noisy-clean pairs from SIDD-Medium
training dataset, where the noisy observations are captured under different
scene illuminances (e.g., scene and brightness) indicating the image prior and
different imaging environments (e.g., ISO level and shutter speed) indicating
the noise prior. As described above, the noise prior should depend on these
imaging environments, and affects the statistical parameter of distribution in
pixel-wise variances.

However, it is infeasible to calculate the pixel-wise variances on a single
image. We randomly sample $1000\times 16\times 16$ local raw noisy-clean patch
pairs, and calculate the local variances $\sigma^{2}$ of the noisy residues
and the local means $\mu$ of the clean patches, to further approximate the
corresponding pixel-wise statistic of noise. As formulated in Eq.(5), the
pixel-wise variance $\sigma^{2}$ of noises should be proportional to the
intensity $\mu$ of illuminances. Consequently, as the statistical results
shown in Fig. 2, its slope and intercept represent $\sigma^{2}_{\text{s}}$ and
$\sigma^{2}_{\text{r}}$, respectively.

From these observations, on a common sence and environmental brightness,
namely with the same image prior, the statistical results of noise show a non-
negligible discrepancy for various imaging environments. Specifically, higher
ISO level indicates larger sensitivity of the sensor, leading more noises
affecting image quality. Faster shutter speed causes lower exposure, and
commonly needs to increase the ISO to compensate for the lack of exposure,
indirectly increasing the image noise. Nonetheless, under a common imaging
environment (e.g., ISO level or shutter speed) with the same noise prior,
different scenes/brightnesses share a statistically similar result of the
noise distribution. Thus, this observation indicates the noise prior is beyond
and independent on the image prior.

Figure 3: Pipeline of LoNPE algorithm. The dynamic range of the input raw
noisy image has been normalized into $[0,1]$.

###  3.2 Locally Noise Prior Estimation

Specifically in practical usage, we can only capture a corrupted noisy
observation and commonly have little additional information on the
environmental illumination, sensor technology, and photography settings.
Therefore, the independence of noise prior and image prior is a critical
principle that we can use to estimate the noise prior from a single noisy
image.

As aforementioned, under a common imaging environment, different scenes should
have statistically similar noise priors, yet the image prior of a scene
generally indicates the features for visual perception, which are commonly
represented as texture or edges. The final image commonly has a specific
statistical distribution, and each pixel can be formulated as

$\mathbf{I}_{i}=\Phi(\\{\mathbf{I}_{j}\\}_{j\in\mathcal{O}(i)})$ (6)

where $\mathcal{I}(\cdot)$ indicates image prior model implemented as an
onefold denoising model, $\mathcal{O}(i)$ denotes the local neighbors of
location $i$.

Due to the intrinsic sophisticated characteristics of image, the statistical
variance of a whole image is less effective to infer neither noise prior nor
image prior. To effectively separate the noise prior and the image prior, we
present a Locally Noise Prior Estimation (LoNPE) algorithm by eliminating the
effect of image prior, such as the sophisticated textures, spatially non-local
structures and edges, and etc.

A primal motivation is to employ the local luminance constancy in a smooth
patch, where

$\mathbf{I}_{i}\simeq\mathbb{E}_{j\in\mathcal{O}(i)}(\mathbf{I}_{j})$ (7)

in other words, the image prior is negligible in a local smooth patch, and is
represented as the mean value of pixels in local region. Thus, on the local
neighbor locations $\mathcal{O}(i)$, the statistical distribution of
$\\{\mathbf{I}_{i}\\}_{i\in\mathcal{O}(i)}$ should be approximately same, and
could be utilized to estimate the noise prior as its independency on the image
prior.

As illustrated in Fig. 3, we firstly preprocess the image to restrict its
theoretical value range into $[0,1]$, then partition it into a group of local
patches, and select the smooth patches to eliminate the interference of image
prior. Next, we calculate these patches’ statistical values of mean and
variance, to finally estimate the noise prior ($\sigma_{\text{s}}$,
$\sigma_{\text{r}}$) with a simple least square optimization solver.

####  3.2.1 LoNPE Algorithm

Due to the variety of sensor bit depth $B$ (e.g., $B=10$ in iPhone 7 camera,
$B=14$ in Canon 80D camera), the theoretical value range of raw digital sensor
image $[0,2^{B}]$ might be different. For generally analyzing, we preprocess
the raw image by normalizing its theoretical value range into $[0,1]$, as

$\mathbf{I}=\mathbf{I}\cdot 2^{-B}$ (8)

Subsequently, the image is firstly partitioned into $n$ patches
$\\{\mathbf{I}_{i}\\}^{n}_{i=1}$ at same size of $\mathcal{O}$. As shown in
Fig. 3, on the assumption of local illuminance constancy, we select $m$
smoother patches $\\{\mathbf{I}_{i}\\}^{m}_{i=1}$ as samples for noise prior
estimation. In detail, on a smooth patch, the local luminance should be
approximated by its statistical mean, as

$L_{i}=\mathbb{E}_{j\in\mathcal{O}(i)}(\mathbf{I}_{j})$ (9)

besides, the statistical variance $\sigma^{2}$ should contain only noise
prior, and is formulated as

$\displaystyle\sigma^{2}_{i}$
$\displaystyle=L_{i}\cdot\sigma_{\text{s}}^{2}+\sigma_{\text{r}}^{2}$ (10)
$\displaystyle=\mathbb{E}_{j\in\mathcal{O}(i)}(\mathbf{I}_{j}-L_{i})^{2}$

Particularly, to eliminate the image prior, these smooth patches are sampled
from the original patch pools $\\{\mathbf{I}_{i}\\}^{n}_{i=1}$ by employing a
local smoothness criterion $\lambda_{\mathcal{S}}$, which is designed to
quantify the local smoothness of an heteroscedastic Gaussian distribution, and
is formulated as

$\lambda_{\mathcal{S}}=\sigma_{i}/\sqrt{L_{i}}$ (11)

that, the lower $\lambda_{\mathcal{S}}$, the higher smoothness in $i$-th local
patch.

Figure 4: Framework of LoNPE network. The backbone of network is same as the
one in DoTNet [66].

Based on the independence of noise prior and image prior, all local patches of
an image should share a common noise prior ($\sigma_{\text{s}}$,
$\sigma_{\text{r}}$). Then, we have

$\displaystyle\begin{bmatrix}\sigma^{2}_{1}\\\ \sigma^{2}_{2}\\\ ...\\\
\sigma^{2}_{m}\end{bmatrix}$ $\displaystyle=\begin{bmatrix}L_{1}\\\ L_{2}\\\
...\\\ L_{m}\end{bmatrix}\cdot\sigma^{2}_{\text{s}}+\sigma^{2}_{\text{r}}$
(12) $\displaystyle=\begin{bmatrix}L_{1}\quad 1\\\ L_{2}\quad 1\\\
...\quad...\\\ L_{m}\quad
1\end{bmatrix}[\sigma^{2}_{\text{s}}\quad\sigma^{2}_{\text{r}}]^{\text{T}}$

if only $\textit{Rank}([\mathbf{L},\ \mathbf{1}])\geq 2$ where
$\mathbf{L}=[L_{1},L_{2},...,L_{m}]^{\text{T}}$, it would be effective to
estimate the noise prior $(\sigma_{\text{s}},\ \sigma_{\text{r}})$ using a
simple least square optimization algorithm.

Note that, LoNPE algorithm estimate the noise prior from a single raw noisy
observation $\mathbf{I}$ (or $\mathbf{I}_{\text{raw}}$) as

$[\sigma_{\text{s}},\sigma_{\text{r}}]=\Psi_{\text{LoNPE}}(\mathbf{I}_{\text{raw}})$
(13)

which strictly follows the statistical characteristics of raw sensor noise
prior, it might be infeasible to directly apply to a sRGB color image due to
the sophisticated ISP procedure.

####  3.2.2 LoNPE Network

Inspired by the concept of noise representation in [66], that the noise level
could be represented as an explicit parameter and is easy to learn by an
external neural network. We build a learnable CNN model to predict the noise
prior
$(\hat{\sigma}_{\text{s}},\hat{\sigma}_{\text{r}})\rightarrow(\sigma_{\text{s}},\sigma_{\text{r}})$
from a single sRGB noisy color image $\mathbf{I}_{\text{rgb}}$, which is
formulated as

$[\hat{\sigma}_{\text{s}},\hat{\sigma}_{\text{r}}]=\Phi_{\text{LoNPE}}(\mathbf{I}_{\text{rgb}})$
(14)

where $\Phi_{\text{LoNPE}}(\cdot)$ denotes the noise prior estimation network,
and called “LoNPE network”. The framework of LoNPE network is shown in Fig. 4,
consisting of a backbone as the DoTNet in [66] for feature extraction and two
fully-connected layers for decision. Due to the physical characteristics of
shot noise (photon-to-electron conversion rate) and read noise (quantization
range of digital signal), the output range of $\sigma_{\text{s}}$ and
$\sigma_{\text{r}}$ are limited to $[0,1]$.

To learn an effective LoNPE network, we need to calculate the groundtruth
noise prior, by applying the LoNPE algorithm $\Phi_{\text{LoNPE}}(\cdot)$ with
numerous training samples of raw noisy observations, e.g., SIDD-Medium raw-
domain dataset. Subsequently, we train the LoNPE network by optimizing the
following objective function,

$\mathcal{L}_{\text{LoNPE}}=\left\|\Phi_{\text{LoNPE}}(\mathbf{I}_{\text{rgb}})-\Psi_{\text{LoNPE}}(\mathbf{I}_{\text{raw}})\right\|_{1}$
(15)

where $\left\|\cdot\right\|_{1}$ represents L1 loss function, minimizing the
mean absolute error (MAE) between the estimated noise prior and the
corresponding groundtruth.

Figure 5: Architecture of Condformer. The information flow is successively
transmitted into the denoiser encoder, the latent module, and the denoiser
decoder, where the latent module stacks groups of CondSA blocks. Particularly,
the noise prior is generated via LoNPE network, and is then embedded as a
conditional vector $\mathbf{z}$ into the query tensor $\mathbf{Q}$ and key
tensor $\mathbf{K}$ in each CondSA block.

###  3.3 Conditional Denoising Transformer

As discussed in Section 1, a conditional denoising model is necessary for
improving the performance by decomposing the optimization space under the
guidance of noise prior. Considering the image prior and noise prior in a
noisy image, an excellent conditional denoising model should be good at
extracting the implicit image prior to learn how to restore the corrupted
pixels, and utilizing the explicit noise prior to precisely control the
intensity of restoration.

Due to the strong capability of Transformer-based denoising models (e.g., CAT
[49], Uformer [8], Restormer [9] and etc), we design a Conditional denoising
Transformer (Condformer) by embedding the noise prior into the self-attention
module. Particularly, as the independence of noise prior and image prior,
guiding the model to learn from image prior and noise prior separately is the
primary principle in designing the Condformer.

####  3.3.1 Embedding noise prior in latent space

Existing image denoising networks typically use a global residual connection
for noise prediction and an U-shape encoder-decoder structure for image prior
representation, suggesting that noise is implicitly concealed in the latent
space as illustrated in Fig. 5. In latent space manipulation [67, 68], the
latent space code from the encoder effectively represents implicit semantic
attributes for domain transfer. Inspired by this and considering the residual
attribute of noise, the latent space code of a noisy image significantly
represents noise statistics.

As mentioned earlier in Section 1, an effective conditional denoising model
should separately consider the image and noise priors, adhering to the
principle of their independence. Specifically as shown in Fig. 5, we firstly
extract the latent space code $\mathbf{X}$ of the noisy image using a denoiser
encoder, and then construct a feature fusion module to embed the noise prior
$(\sigma_{\text{s}},\sigma_{\text{r}})$.

####  3.3.2 Overall pipeline

Considering the efficiency in practical applications, we employ the Restormer
as our denoiser baseline, replacing its self-attention module by a conditional
self-attention module that embeds noise prior in the latent space. Following
[9], given a noisy color observation
$\mathbf{I}_{\text{rgb}}\in\mathbb{R}^{3\times h\times w}$, we construct a
multi-scale hierarchical denoiser encoder which consists of three channel-wise
Transformer blocks to capture cross-covariance across channels, generating the
latent space code
$\mathbf{X}\in\mathbb{R}^{8c\times\frac{h}{8}\times\frac{w}{8}}$, where $c$ is
the number of channels.

On handling the latent space code, we introduce a conditional self-attention
module (CondSA) that embeds the noise prior into the implicit latent code for
conditional optimization of denoising model. The rectified latent code is

$\hat{\mathbf{X}}=\text{CondSA}(\mathbf{X},\mathbf{z})$ (16)

This is then fed into a feed-forward network (FFN) for feature transformation.
In particular, to effectively exploit the noise prior
$(\sigma_{\text{s}},\sigma_{\text{r}})$ estimated by LoNPE algorithm or
network, CondSA encodes this prior into a latent conditional embedding vector
$\mathbf{z}\in\mathbb{R}^{1\times k}$ using a shallow module with fully-
connected layers. Each CondSA has a specific embedding vector to adapt the
intermediate features. The rectified latent space code, equipped with noise
prior, guides the denoiser decoder for specific denoising. In particularly, to
preserve fine structural and textural details in the restored images, we use a
hierarchical skip-connection strategy [9] that integrates low-level features
from the encoder and high-level features from the decoder.

Consequently, as depicted in Fig. 5, the denoised output
$\hat{\mathbf{L}}_{\text{rgb}}$ can be formulated as

$\hat{\mathbf{L}}_{\text{rgb}}=\Phi_{\text{Condformer}}(\mathbf{I}_{\text{rgb}},(\sigma_{\text{s}},\sigma_{\text{r}}))$
(17)

where $\Phi_{\text{Condformer}}(\cdot)$ represents the Condformer model, with
inputs consisting of a noisy observation $\mathbf{I}_{\text{rgb}}$ and the
estimated noise prior $(\sigma_{\text{s}},\sigma_{\text{r}})$ from LoNPE
algorithm $\Phi_{\text{LoNPE}}(\mathbf{I}_{\text{raw}})$ or network
$\Psi_{\text{LoNPE}}(\mathbf{I}_{\text{rgb}})$.

Following the convention of supervised denoising methods, we train the
Condformer by optimizing the pixel-wise objective function as

$\mathcal{L}_{\text{Condformer}}=\left\|\mathbf{L}_{\text{rgb}}-\hat{\mathbf{L}}_{\text{rgb}}\right\|_{1}$
(18)

where $\mathbf{L}_{\text{rgb}}$ denotes the clean target corresponding to the
denoised output.

Subsequently, we introduce the preliminary definition of the self-attention
module in Restormer, and describe the proposed CondSA for embedding the
conditional noise prior.

####  3.3.3 Conditional Self-Attention Mechanism

Aiming to alleviate the high complexity of the conventional self-attention
module when calculating the key-query cross-covariance across spatial
dimensions, the self-attention in Restormer attempts to calculate the key-
query cross-covariance across channels, and is formulated as

$\mathbf{Y}=W^{Y}\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})+\mathbf{X}$
(19)

that,

$\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Softmax}({\mathbf{Q}\mathbf{K}^{\textsf{T}}}/{\alpha})\mathbf{V}$
(20)

where $\mathbf{X}$ and $\mathbf{Y}$ are the input and output features.
$\mathbf{Q}\in\mathbb{R}^{c\times hw}$, $\mathbf{K}\in\mathbb{R}^{c\times hw}$
and $\mathbf{V}\in\mathbb{R}^{c\times hw}$ indicate the query, key and value
matrix obtained by encoding the input feature
$\mathbf{X}\in\mathbb{R}^{c\times h\times w}$ with linear layers $W^{Q}$,
$W^{K}$ and $W^{V}$ respectively, each of which stacks a $1\times 1$
convolution and a $3\times 3$ depth-wise convolution layer. Besides, $\alpha$
is a learnable scaling parameter to control the magnitude of the cross-
covariance of $\mathbf{Q}$ and $\mathbf{K}$ before applying a softmax layer.

By extracting and exploring the local and non-local features, the whole
network indeed aims to employ the image priors of the noisy observation, which
exactly meets the unconditional optimization paradigm in Fig. 1a. According to
the optimization of conditional denoising model in Eq.(2) and Fig. 1b, the
noise prior should be embedded into a conditional self-attention module as

$\mathbf{Y}=W^{Y}\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V},\mathbf{z})+\mathbf{X}$
(21)

where $\mathbf{z}\in\mathbb{R}^{1\times 2k}$ represents the conditional
embedding vector from the noise prior $(\sigma_{\text{s}},\sigma_{\text{r}})$
by repeating $k$ times in channel dimension.

The conditional attention should effectively represent the correlation
between-in the intermediate image features, and the latent correlation between
the intermediate image features and noise prior. Therefore, a fusion module is
necessary for correlation representation of noise and image priors.
Intuitively, the query tensor $\mathbf{Q}$ and the key tensor $\mathbf{K}$
indicates the information for feature retrieval [42]; instead, the value
tensor $\mathbf{V}$ represent the property of the input feature. Therefore, it
is reasonable to embed noise prior into the query/key tensors and generate the
conditional counterparts $\mathbf{Q}^{\prime}$ and $\mathbf{K}^{\prime}$, so
we have

$\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V},\mathbf{z})=\text{Softmax}(\mathbf{Q}^{\prime}{\mathbf{K}^{\prime}}^{\textsf{T}}/{\alpha})\mathbf{V}$
(22)

and

$\mathbf{Q}^{\prime}={\Phi}_{\text{LFM}}(\mathbf{Q},\mathbf{z}),\quad\mathbf{K}^{\prime}={\Phi}_{\text{LFM}}(\mathbf{K},\mathbf{z})$
(23)

where $\Phi_{\text{LFM}}$ represents a linear fusion module (LFM) layer for
feature fusion. In detail, in order to fuse the query/key tensor and the
conditional embedding vector in channel and spatial dimension, we can simply
concatenate them in channel-wise and feed them into a a $1\times 1$
convolution and a $3\times 3$ depth-wise convolution layer as the convention.
In this way, the noise prior would be embedded into each channel and a local
receptive field of the tensors.

Figure 6: Noise prior estimation on synthetic noises with Poisson-Gaussian
distribution. By applying our LoNPE algorithm, we can effectively calculate a
relative accurate noise prior from a single noisy observation.

##  4 Experiments

We first describe the experimental setup and then evaluate the performances of
our LoNPE and Condformer on noise statistics and blind image denoising.
Finally, we perform ablation studies to demonstrate the effectiveness of our
methods.

###  4.1 Experimental Setup

####  4.1.1 Datasets and Metrics

To demonstrate the effectiveness of our LoNPE and Condformer, we conduct
experiments on both synthetic and real datasets. Below are the details of the
synthetic and real datasets, and evaluation metrics.

Synthetic Datasets. Following [53], we adopt several sRGB image datasets for
training the LoNPE and Condformer networks, including the DIV2K and Flicker2K
dataset [69] with 3650 high-quality 2K images, the Berkeley segmentation
dataset (BSD) [70] with 400 images, Waterloo Exploration Database (WED)
dataset [71] with 4744 images. Considering both signal-dependent and signal-
independent noises, we randomly add noises on the clean sRGB image with
Poisson-Gaussian noise model. The noise level are set to
$\sigma_{\text{s}}\sim[0,0.3]$ and $\sigma_{\text{r}}\sim[0,50/255]$, both of
which indicate the groundtruth noise prior to demonstrate the effectiveness of
our LoNPE algorithm. Especially, the Poisson-Gaussian noise will be degraded
to additive Gaussian white noise (AWGN) when $\sigma_{\text{s}}=0$. To
evaluate the performance, we apply our methods on several benchmarks,
including CBSD68 [70], Kodak24 [72] and Urban100 [73].

Real Datasets. Consistent with previous real image denoising work [9], we
adopt the SIDD-Medium dataset [10] for real noise statistics and real image
denoising tasks, which contains 320 noisy-clean image pairs in both raw and
sRGB domains. Particularly, we first apply our LoNPE algorithm on 320 noisy
raw images to estimate their corresponding noise priors, and then utilize them
to help training the LoNPE and Condformer networks on the 320 sRGB noisy-clean
image pairs. For validation, 1024 pairs of noisy-clean sRGB image patches from
SIDD validation dataset [10] are adopted. Besides, evaluation is also
conducted on 1280 noisy $256\times 256$ patches from the SIDD benchmark
dataset [10] and 50 noisy $512\times 512$ images from the DND benchmark
dataset [12].

Evaluation Metrics. Two commonly-used image quality assessment criteria are
adopted to evaluate the performances: Peak Signal-to-Noise Ratio (PSNR) and
Structural SIMilarity (SSIM) [74], and are calculated in the sRGB domain. Note
that, since the clean groundtruthes of SIDD and DND benchmark datasets are
unavailable, we calculate these metrics on only SIDD validation dataset and
obtain the metrics of them from the online servers111SIDD:
https://abdokamel.github.io/sidd/;
DND: https://noise.visinf.tu-darmstadt.de.

####  4.1.2 Implementation Details

Settings on LoNPE Algorithm. As described in Section 3.2.1, to accurately
calculate the noise prior of a noisy observation in raw domain, our LoNPE
algorithm $\Psi_{\text{LoNPE}}$ is applied on top $\frac{m}{n}=10\%$ of the
sampled local patches, which are at size of $\mathcal{O}=16\times 16$ for each
raw image. Given a $h\times w$ raw noisy image, we first sample
$n=\left\lfloor\frac{h}{16}\right\rfloor\times\left\lfloor\frac{w}{16}\right\rfloor$
local patches, and select the top $m$ smooth patches with lower
$\lambda_{\mathcal{S}}$, then calculate its noise prior using Eq.(13).

Settings on LoNPE Network. As described in Section 3.2.2, for practical
applications on estimating the noise prior of a sRGB noisy image, we need to
train an effective LoNPE network $\Phi_{\text{LoNPE}}$ as illustrated in Fig.
4. Similar to [66], for a single noisy image, we randomly sample 8 local
patches of size $32\times 32$ and average the output
$(\hat{\sigma}_{\text{s}},\hat{\sigma}_{\text{r}})$ as the final predicted
noise prior. In the training phase, AdamW optimizer [75] is employed with
cosine annealing [76] learning rate from $10^{-3}$ to $10^{-6}$ during 50K
mini-batch iterations, on minimizing the objective function in Eq.(15), and
the batch size is set to 64.

Figure 7: Statistical experiments on noise prior of raw sensor images from
various scenes in SIDD-Medium training datasets. We can find that different
imaging environments cause various noise priors, e.g., higher ISO level causes
more shot noises and higher shutter speed causes more read noises. Instead,
different target scenes show similar noise priors, indicating the independence
of image prior and noise prior.

Settings on Condformer. Following the settings of Restormer in [9], we build
our Condformer with groups of Transformer block in the encoder or decoder
modules. Yet in the latent module, we stack 8 CondSA blocks with the predicted
noise prior to recify the latent space. In the training phase, we adopt the
AdamW optimizer with $(\beta_{1},\beta_{2})=(0.9,0.999)$ and set weight decay
to $10^{-4}$. Using progressive training strategy proposed by [9], we set the
batch size and patch size pairs to [$(64,128^{2})$, $(16,256^{2})$,
$(8,384^{2})$, $(4,512^{2})$] at training iterations [0k, 150k, 200k, 250k].
We train our Condformer models for total 300k iterations and the initial
learning rate is set to $4\times 10^{-4}$ and gradually reduced to $10^{-6}$
through the cosine annealing. Data augmentation is performed on the training
data through horizontal flip and random rotation of 90, 180, and 270.

Both of the LoNPE and Condformer networks are implemented on PyTorch framework
using NVIDIA A800 GPUs.

###  4.2 Experiments on Noise Statistics

In this section, we conduct several statistical experiments by analyzing the
statistics of synthetic or real noises, to further demonstrate the
effectiveness of our LoNPE algorithm on noise prior representation.

####  4.2.1 On synthetic noises

As illustrated in Section 3.1.2, the noise prior is beyond and independent of
the image prior. From Eq. (5), the noise prior
$(\sigma_{\text{s}},\sigma_{\text{r}})$ affects raw noisy observations but is
implicit in the sRGB color observations due to unknown ISP operations. To
address this, we randomly add noises on the clean sRGB images with Poisson-
Gaussian sampling and employ our LoNPE algorithm on the synthesized noisy
images to estimate the corresponding noise prior.

As shown in Fig. 6, we set three degree of noise priors
$(\sigma_{\text{s}},\sigma_{\text{r}})$ as $(0.05,0.02)$, $(0.10,0.04)$ and
$(0.15,0.08)$, representing mild, moderate and severe corruptions,
respectively. We sample Poisson-Gaussian noises and add them to 100 images
from the Urban100 dataset. By applying LoNPE algorithm, we obtain the
estimated noise prior values of each image with different noise level, and
find that the mean estimation results are close to the groundtruthes with less
discrepancy, particularly for the signal-dependent noise prior
$\sigma_{\text{s}}$. However, there exists a nonnegligible discrepancy in the
signal-independent noise prior $\sigma_{\text{r}}$ estimation, and the
discrepancy increases as the noise level is enlarged. A reason is that, the
image context reflects the signal intensity of the target scene illuminance,
and could interfere the estimation of the signal-independent
$\sigma_{\text{r}}$. This issue is particularly pronounced at the high-
frequency regions, e.g., edges and textures. That is why we need to employ the
local smoothness criterion $\lambda_{\mathcal{S}}$ to eliminate the
interference of image prior.

####  4.2.2 On real noises

As described in Section 3.2, the goal of LoNPE is to calculate the noise prior
from a single raw noisy observation based on the independence of noise prior
and image prior. Thus, the most critical thing for demonstration is the
statistical experiment on the correlations of the estimated noise prior
$(\sigma_{\text{s}},\sigma_{\text{r}})$ and the imaging environments.

TABLE I: Quantitative comparisons of different synthetic image denoising methods on several validation datasets with a fixed $\sigma_{\text{s}}$ and various $\sigma_{\text{r}}\in[15,25,50]/255$. PSNR$\uparrow$ criterion is adopted to evaluate the performances. Method |  Publisher |  Noise |  Params | CBSD68 | Kodak24 | Urban100
---|---|---|---|---|---|---
Model |  (M) |  $\sigma_{\text{s}}$=0 |  $\sigma_{\text{s}}$=0.1 |  $\sigma_{\text{s}}$=0.3 |  $\sigma_{\text{s}}$=0 |  $\sigma_{\text{s}}$=0.1 |  $\sigma_{\text{s}}$=0.3 |  $\sigma_{\text{s}}$=0 |  $\sigma_{\text{s}}$=0.1 |  $\sigma_{\text{s}}$=0.3
Noisy |  - |  - |  - |  20.12 |  17.10 |  13.88 |  20.02 |  16.97 |  13.72 |  20.27 |  17.12 |  13.87
DnCNN [15] |  TIP’17 |  G |  0.7 |  31.00 |  29.10 |  26.30 |  31.89 |  30.12 |  27.27 |  30.46 |  28.70 |  25.64
Restormer [9] |  CVPR’22 |  G |  26.1 |  31.57 |  28.98 |  25.40 |  32.82 |  30.24 |  26.62 |  32.65 |  29.84 |  25.68
VDN [16] |  NeurIPS’19 |  SV-G |  2.0 |  31.21 |  29.33 |  27.22 |  32.28 |  30.49 |  28.33 |  31.46 |  29.61 |  27.05
DRANet [77] |  PR’24 |  SV-G |  1.6 |  31.33 |  29.44 |  27.30 |  32.50 |  30.69 |  28.44 |  31.93 |  30.09 |  27.65
VIRNet [28] |  TPAMI’24 |  SV-G |  10.5 |  31.45 |  29.52 |  27.17 |  32.65 |  30.76 |  28.24 |  32.23 |  30.24 |  27.14
FBI-D [78] |  CVPR’21 |  P-G |  7.5 |  26.59 |  25.55 |  23.30 |  27.84 |  26.69 |  24.11 |  25.60 |  24.56 |  22.32
VBDNet [79] |  PR’23 |  P-G |  8.3 |  25.93 |  25.51 |  24.41 |  27.00 |  26.73 |  25.68 |  26.45 |  25.86 |  24.57
CLIPDenoising [80] |  CVPR’24 |  P-G |  34.5 |  30.52 |  28.18 |  25.19 |  31.31 |  29.02 |  25.92 |  30.04 |  27.79 |  24.48
Restormer† [9] |  CVPR’22 |  P-G |  26.1 |  31.48 |  29.66 |  28.01 |  32.70 |  31.02 |  29.44 |  32.46 |  30.95 |  29.37
MambaIR† [47] |  ECCV’24 |  P-G |  23.2 |  31.41 |  29.60 |  27.94 |  32.65 |  30.99 |  29.41 |  32.42 |  30.91 |  29.32
Condformer |  Ours |  P-G |  27.0 |  31.59 |  29.84 |  28.14 |  32.81 |  31.20 |  29.58 |  32.64 |  31.15 |  29.56
†: Model is re-trained on Poisson-Gaussian noise model with the same settings as our Condformer. |
Figure 8: Visual results of restoring the Poisson-Gaussian noisy images. We
can find that our Condformer can preserve the details and remain less
distortions as compared with other methods, showing higher generalization on
various corruptions with different noise levels.

Based on the independence of image prior and noise prior investigated in Fig.
2, the noise prior is quantifiable using our LoNPE algorithm. To demonstrate
the effectiveness of our LoNPE algorithm, we conduct a statistical study on
the estimated noise priors of SIDD noisy observations with various target
scenes and imaging environments. Specifically, according to the camera
information of the SIDD-Medium raw-domain training dataset, we mainly analyze
the estimated noise prior under various brightness (including “high-light”,
“normal-light” and “low-light”), scenes (including scene “001”, “002”, “007”
and “008”), ISO levels (ranging from 100 to 3200), and shutter speed (ranging
from 1/1600 to 1/60). Subsequently, the statistical results in Fig. 7 can be
summarized into several points:

1) Higher ISO level leads to more shot noise. Under same brightness, the shot
noise prior $\sigma_{\text{s}}$ is proportional to the ISO levels. Since the
ISO settings represent the photo-sensibility of camera sensor, a higher ISO
level amplifies the electrical signal generated by the photons hitting the
sensor. As shot noise follows Poisson distribution, its relative strength
becomes more pronounced as you increase the ISO. The higher the amplification,
the more noticeable the noise will be in the final image. This is crucial in
low-light conditions to brighten the image. Thus, the slope of variance
($\sigma^{2}_{\text{s}}$) in Eq. (5) would be larger as the ISO increased.

2) Higher shutter speed leads to more read noises. Under same brightness, the
read noise prior $\sigma_{\text{r}}$ is proportional to the shutter speed. In
camera sensor, especially those designed for high-speed photography, the
demand to read out quickly can introduce more noises. Fast readout processes
might require higher readout clock speeds and increased electronic noise,
contributing to the overall read noise.

3) Independence of noise and image priors. Empirically, the environmental
brightness is proportional to the intensity of photons and should influence
only shot noise; instead, the ISO and shutter speed indicate the magnification
of electrons from photons and would affect the shot noise and read noise,
respectively. Besides, the captured signal in a digital image is a sum of both
the scene-derived signal and various sensor noises. However, these sensor
noises overlay the signal in a way that their behavior and characteristics do
not alter based on the attributes of the target scene itself. This phenomenon
is well demonstrated by statistical analysis in Fig. 7. We can find that
different scenes (with various image priors) have similar noise prior under
the same imaging environment, which is strong evidence of demonstrating the
independence of noise prior and image prior.

Understanding the independence of noise prior and image prior allows
photographers and engineers to develop better noise removal algorithms and
improve sensor designs. By treating the scene and sensor noise as separate
entities, it becomes easier to process images to enhance the desired signal
intensity while minimizing the impact of sensor noise on the final image.

###  4.3 Experiments on Image Denoising

TABLE II: Quantitative comparisons of different real image denoising methods on several benchmarks. Particularly, the criteria of SIDD and DND benchmarks are obtained from the corresponding online server. Method |  Publisher |  Params$\downarrow$ | SIDD Validation |  SIDD Benchmark† |  DND Benchmark†
---|---|---|---|---|---
(M) |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
Noisy |  - |  - |  23.66 |  0.4848 |  29.56 |  0.3347 |  29.84 |  0.7015
RIDNet [3] |  ICCV’19 |  1.5 |  38.77 |  0.9511 |  38.98 |  0.9076 |  39.24 |  0.9513
VDN [16] |  NeurIPS’19 |  7.8 |  39.36 |  0.9562 |  39.49 |  0.9117 |  39.30 |  0.9493
MIRNet [81] |  ECCV’20 |  31.8 |  39.72 |  0.9586 |  39.80 |  0.9147 |  39.88 |  0.9543
MPRNet [5] |  CVPR’21 |  15.7 |  39.71 |  0.9586 |  39.80 |  0.9149 |  39.82 |  0.9540
Uformer [8] |  CVPR’22 |  50.8 |  39.89 |  0.9594 |  39.97 |  0.9160 |  40.05 |  0.9562
Restormer [9] |  CVPR’22 |  26.1 |  40.02 |  0.9603 |  40.09 |  0.9171 |  40.03 |  0.9564
NAFNet [82] |  ECCV’22 |  29.1 |  39.97 |  0.9599 |  40.04 |  0.9166 |  39.10 |  0.9495
MSANet [83] |  NeurIPS’22 |  8.6 |  39.56 |  0.9575 |  39.70 |  0.9131 |  39.65 |  0.9553
CAT [49] |  NeurIPS’22 |  25.8 |  40.01 |  0.9600 |  40.09 |  0.9167 |  40.05 |  0.9561
MIRNetv2 [4] |  TPAMI’23 |  5.9 |  39.84 |  0.9593 |  39.91 |  0.9154 |  39.86 |  0.9550
ShuffleFormer [50] |  ICML’23 |  50.1 |  40.00 |  0.9603 |  40.08 |  0.9168 |  40.01 |  0.9560
GRL [52] |  CVPR’23 |  19.8 |  39.89 |  0.9595 |  40.01 |  0.9161 |  39.76 |  0.9540
ART [51] |  ICLR’23 |  25.7 |  39.96 |  0.9598 |  40.03 |  0.9164 |  40.05 |  0.9557
VIRNet [28] |  TPAMI’24 |  15.4 |  39.70 |  0.9586 |  39.78 |  0.9148 |  39.77 |  0.9533
MambaIR [47] |  ECCV’24 |  23.2 |  39.89 |  0.9598 |  39.97 |  0.9164 |  39.83 |  0.9542
Condformer |  Ours |  27.0 |  40.21 |  0.9612 |  40.23 |  0.9176 |  40.10 |  0.9562
Figure 9: Qualitative comparisons on SIDD validation datasets. Our Condformer
can preserve more details as accurately guide the model to remove the noises
adaptively, instead other unconditional denoisers prone to handle the high-
frequency details as undesired noises since the unknownability of noise level
in training and testing phases.

####  4.3.1 On synthetic images

In this section, we conduct Poisson-Gaussian image denoising experiments on
synthetic validation datasets. Note that, recent blind denoisers were commonly
trained on noise models as Gaussian (G), spatially-variant Gaussian (SV-G) and
Poisson-Gaussian (P-G) distributions. We formulate the corresponding noises as
follows,

$\displaystyle\mathbf{N}_{\text{G}}$
$\displaystyle\sim\mathcal{N}(\mathbf{0},\sigma^{2}_{\text{r}})$ (24)
$\displaystyle\mathbf{N}_{\text{SV-G}}$
$\displaystyle\sim\mathbf{M}\odot\mathcal{N}(\mathbf{0},\mathbf{1})$
$\displaystyle\mathbf{N}_{\text{P-G}}$
$\displaystyle\sim\mathcal{N}(\mathbf{0},\sigma^{2}_{\text{s}}\cdot\mathbf{L}+\sigma^{2}_{\text{r}})$

obviously, Gaussian noise model is a specific case of P-G noise model when
$\sigma_{\text{s}}=0$, and SV-G noise model is more fine-grain as applying a
spatial map $\mathbf{M}$ to represent the variance of Gaussian, thus P-G noise
model can be regarded as a specific case of SV-G by setting the map relative
to image context $\mathbf{L}$.

In comparison of synthetic image denoising, several current state-of-the-art
blind denoisers are selected, including: 1) Gaussian noise model drived: DnCNN
[15] and Restormer [9]; 2) SV-G noise model drived: VDN [16], DRANet [77] and
VIRNet [28]; 3) P-G noise model drived: FBI-D [78], VBDNet [79], CLIPDenoising
[80] and our Condformer. Moreover, we re-train the Restormer [9] and MambaIR
[47] denoisers on P-G noise model with same settings as our Condformer for
fair comparisons.

As reported in TABLE I, quantitative comparisons on three public validation
datasets with various noise levels show that, our Condformer achieves superior
performances against other methods. On denoising accuracy, our Condformer
obtains higher PSNR values especially when $\sigma_{\text{s}}>0$ and shows
slight disadvantages compared with the Restormer trained on Gaussian noise
model when $\sigma_{\text{s}}=0$. On model generalization, under same
experimental settings, our Condformer achieves excellent performances on all
noise levels, instead Restormer† and MambaIR† show lower generalization on
various noise levels. That is, our Condformer possesses a complete and
conditional optimization space for divide-and-conquer, which is naturally
stronger than any unconditional models. Furthermore, the qualitative
comparisons shown in Fig. 8 demonstrate our Condformer can handle any degree
of noisy corruption, being capable of detail preservation and noise removal
well.

Figure 10: Qualitative comparisons on SIDD benchmark datasets. Especially
under low-light imaging environments, noises with severe corruption prone to
overwhelm the image contexts. In this case, only our Condformer could preserve
more details, e.g., textures, edges and natural structures. Figure 11:
Qualitative comparisons on DND benchmark under normal-light imaging
environment. Our Condformer preserves the edges and natural structures more
distinctly than other denoisers. Figure 12: Qualitative comparisons on DND
benchmark under low-light imaging environment. Although noise interferes the
image context restoration overwhelmingly under low-light environment, our
Condformer remains fewer noises and can restore cleaner flatten regions than
other denoisers.

####  4.3.2 On real images

In this section, we demonstrate the effectiveness of the proposed Condformer
for real image denoising. We compare our Condformer with several state-of-the-
art real image denoising methods on SIDD validation, SIDD benchmark, and DND
benchmark datasets. We select several representative CNN-based denoisers and
current state-of-the-art Transformer-based and Mamba-based denoisers,
including: 1) CNN-based denoiser: RIDNet [3], VDN [16],MIRNet [81], MPRNet
[5], NAFNet [82], MSANet [83], MIRNetv2 [4] and VIRNet [28]; 2) Transformer-
based denoiser: Uformer [8], Restormer [9], CAT [49], ShuffleFormer [50], GRL
[52] and ART [51]; 3) Mamba-based denoiser: MambaIR [47]. As reported in TABLE
II, our Condformer achieves the highest PSNR and SSIM criteria over all the
compared denoising methods. Note that, since the clean groundtruth images of
SIDD and DND benchmarks are inaccessible, we upload the denoised results of
all the considered methods on their online servers for testing.

Particularly, compared with other Transformer-based denoisers, our Condformer
needs relatively lower complexity. For example, on SIDD validation dataset,
our Condformer achieves 0.32dB gain of PSNR over Uformer with about half of
its parameters. Even though Restormer is considered as the baseline of our
Condformer without embedded noise prior in the self-attention module, our
Condformer gets about 0.2dB gain of PSNR over it with similar computational
complexities. In spite of training on a single SIDD-Medium dataset, our
Condformer performs excellently on the out-of-distribution DND benchmark.

Qualitatively on the in-distribution SIDD validation and benchmark datasets,
as shown in Fig. 9 and Fig. 10, our Condformer can not only successfully
remove the noises but also keep the details well, including textures, edges
and natural structures. After conditional optimization with noise prior, our
Condformer could handle each noisy image in a sensor-specific way, where the
denoising intensity is proportional to the corruption degree of the noisy
observation. Especially under low-light imaging environments, noises prone to
overwhelm the image contexts as introducing higher ISO level and indirectly
enlarging the shot noises. Furthermore, Fig. 11 and Fig. 12 shows that our
Condformer has excellent generalization ability as handling the out-of-
distribution DND noisy images well, on both detail preservation and undesired
noise removal. Note that, since the images have low visibility under low-light
imaging environment, we employ image normalization on each denoised result for
better visualization.

TABLE III: Quantitative comparisons of different Transformer-based denoising methods on computational complexity. The FLOPs, Memory and Time are calculated when processing a single $512\times 512\times 3$ sRGB image on a NVIDIA A800 GPU. Method |  Params.$\downarrow$ |  FLOPs$\downarrow$ |  Mem.$\downarrow$ |  Time$\downarrow$
---|---|---|---|---
(M) |  (G) |  (G) |  (s)
Uformer [8] |  50.8 |  343.1 |  2.89 |  0.21
Restormer [9] |  26.1 |  564.0 |  3.81 |  0.36
CAT [49] |  25.8 |  543.5 |  10.83 |  15.76
ShuffleFormer [50] |  50.5 |  344.4 |  2.97 |  0.23
GRL [52] |  19.8 |  5012.3 |  12.38 |  10.45
ART [51] |  25.7 |  542.8 |  3.66 |  0.41
Condformer (Ours) |  27.0 |  565.2 |  3.81 |  0.37

####  4.3.3 On computational complexity

In this section, we mainly conduct a comparison of our Condformer and other
Transformer-based denoising methods on several computational complexity
criteria. In detail, the number of parameters represents the model size for
transmission and storage necessaries, FLOPs and Time indicate the time
complexity of model, and Memory is the memory usage when running model on a
GPU device. The last three criteria are calculated when processing a single
$512\times 512\times 3$ sRGB image on a NVIDIA A800 GPU. To avoid randomness,
the running time is averaged on handling 100 images with torch.cuda.Event
timer. As reported in TABLE III, our Condformer shows relatively trade-off on
the complexities, which is a resource-friendly method.

###  4.4 Model Analysis

In this section, we mainly investigate the effects of our Condformer and LoNPE
modules with experimental analysis.

TABLE IV: Ablation study of Condformer with different priors on SIDD Validation dataset. CondSA |  Prior |  LCM |  Params$\downarrow$ |  FLOPs$\downarrow$ |  Time$\downarrow$ |  PSNR$\uparrow$
---|---|---|---|---|---|---
|  |  |  (M) |  (G) |  (s) |  (dB)
✗ |  - |  - |  26.11 |  563.96 |  0.361 |  39.68
✓ |  $(0,0)$ |  ✗ |  26.13 |  564.04 |  0.363 |  39.68
✓ |  $(\hat{\sigma}_{\text{s}},\hat{\sigma}_{\text{r}})$ |  ✗ |  26.74 |  564.22 |  0.368 |  39.81
✓ |  $(\hat{\sigma}_{\text{s}},\hat{\sigma}_{\text{r}})$ |  ✓ |  27.02 |  565.35 |  0.368 |  39.90
✓ |  $(\sigma_{\text{s}},\sigma_{\text{r}})$ |  ✓ |  26.41 |  565.17 |  1.012 |  39.90

####  4.4.1 Investigation on Condformer

Aiming at guiding the model to learn from image prior and noise prior
separately, the core of our Condformer is embedding the noise prior
effectively in the latent space for conditional optimization. Thus, we conduct
an ablation study of Condformer in TABLE IV, note that all the models in this
section are trained during 100K iterations, where the image size is $3\times
128\times 128$ and batch size is set to 8 for training on a single NVIDIA A800
GPU.

Since the key of our Condformer is constructing CondSA blocks in the latent
module in Restormer, we first train a baseline model which is same sa
Restormer and achieves PSNR of 39.68dB on SIDD validation dataset.
Subsequently, to avoid deviation on the performance as introducing an
auxiliary vector, we firstly build a void CondSA module by embedding zero
vector, and find it gain nothing. However, after embedding the estimated noise
prior $(\hat{\sigma}_{\text{s}},\hat{\sigma}_{\text{r}})$ from LoNPE network
by a naive concatenation with the input tensor of CondSA, the corresponding
model achieves about 0.1 PSNR gain, demonstrating the positive role of noise
prior. Furthermore, to learn an effective representation of the correlation
between-in the intermediate image features and noise prior, a LFM module is
designed for correlation representation of noise prior and image features. The
result shows that an effective fusion module can significantly boost the
contribution of the noise prior. As reported in TABLE IV, the fourth model
achieves higher performances than the afore three models, especially gains
more than 0.2dB of PSNR against the baseline.

Moreover, although the precision of noise prior estimation might affects the
denoising results, the Condformer with estimated noise prior
$(\hat{\sigma}_{\text{s}},\hat{\sigma}_{\text{r}})$ is slightly inferior to
the upper bound (the one with groudtruth
$(\sigma_{\text{s}},\sigma_{\text{r}})$ obtained via LoNPE algorithm), but
costs fewer time complexities. That is because the noise prior is a low-
dimensional vector as easier to learn via a simple network, instead the LoNPE
algorithm needs to calculate the numerical solutions iteratively which is
time-consuming.

TABLE V: Investigation on LoNPE algorithm with different patch-sampling hyperparameters. Particularly, $\mathcal{O}$, $m/n$ and $\lambda_{\mathcal{S}}$ denote the sampling size, ratio and index, respectively. LoNPE Algorithm |  Error$\downarrow$ |  Time(s)$\downarrow$
---|---|---
$\mathcal{O}$ |  $m/n$ |  $\lambda_{\mathcal{S}}$ |  $\Delta(\sigma_{\text{s}})$ |  $\Delta(\sigma_{\text{r}})$ |
$8\times 8$ |  $5$% |  ✗ |  0.094 |  0.056 |  0.31
$8\times 8$ |  $5$% |  ✓ |  0.052 |  0.037 |  0.36
$8\times 8$ |  $10$% |  ✓ |  0.052 |  0.037 |  0.36
$16\times 16$ |  $10$% |  ✓ |  0.029 |  0.021 |  0.62
$16\times 16$ |  $20$% |  ✓ |  0.029 |  0.022 |  1.13
$32\times 32$ |  $10$% |  ✓ |  0.063 |  0.033 |  0.65
LoNPE Network |  0.032 |  0.026 |  $<0.01$

####  4.4.2 Investigation on LoNPE

As described in Section 3.2, our LoNPE algorithm is based on the statistical
analysis of the sampled local smooth patches from a single noisy raw image.
From Eq.(12), the noise prior is estimated on $m$ local patches with
statistical mean $\\{L_{i}\\}^{m}_{i=1}$ and variance
$\\{\sigma^{2}_{i}\\}^{m}_{i=1}$. Particularly, from Eqs.(9)-(10), the
precision of statistical values $L_{i}$ and $\sigma^{2}_{i}$ are highly relied
on the spatial size of samples $\mathcal{O}$, and the precision of parameter
estimation is highly relied on the number of samples $m$ to ensure a large
$\textit{Rank}([\mathbf{L},\mathbf{1}])$.

We conduct an investigation on the effects of sampling size $\mathcal{O}$ and
ratio $m/n$ in TABLE V. Since there is no groundtruth for noise prior in SIDD-
Medium raw-domain training dataset, we synthesize noisy samples by randomly
sampling the noises from a Poisson-Gaussian distribution with
$(\sigma_{\text{s}},\sigma_{\text{r}})=(0.2,0.01)$ and adding it on a clean
raw noisy image. The average error (i.e., mean absolute percentage error
$\Delta(\cdot)$ [84]) and running time are reported in TABLE V, with the
increase of patch size $\mathcal{O}$ and sampling ratio $m/n$, it might
introduce a negative effect of image prior. In a word, a larger patch has more
precise statistical values but might contains more image priors interfering
the noise prior estimation, such as edges and textures. Empirically, we choose
$\mathcal{O}=16\times 16$ and $m/n=10$% which provides more precise estimation
of noise prior.

Besides, to further eliminate the image prior in each patch, we introduce a
local smoothness criterion $\lambda_{\mathcal{S}}$ to filter the smooth
patches from the original patch pools $\\{I_{i}\\}^{n}_{i=1}$, which can
effectively boost the accuracy of noise prior estimation as reported in TABLE
V. To further demonstrate the effectiveness of local smoothness criterion, we
record the sampled local patches in Fig. 13. It is obvious that random
sampling local patches from the raw image inevitably captures the numerous
high-frequency details, which would interfere the estimation of noise prior
because of the nonnegligible deviation on calculating the local luminance in
Eq.(9). Instead, by employing the local smoothness criterion
$\lambda_{\mathcal{S}}$, it is easy to filter the smooth patches from the
original patch pools, which plays significant role on eliminating the image
prior and help estimating noise prior.

Figure 13: Visualization of local patch sampling with $\lambda_{\mathcal{S}}$.
The local smoothness criterion $\lambda_{\mathcal{S}}$ can effectively filter
the smooth patches to eliminate the interference of image prior to noise prior
estimation, instead random sampling might introduce high-frequency patches and
cause inaccuracy of local luminance calculation.

##  5 Conclusion and Discussions

In this paper, by rethinking the real image denoising task and revisiting the
formation model of raw sensor noises, we have generalized a principle of the
independence of image prior and noise prior, which guides an alternative
conditional optimization to tackle the limitations of existing learning-based
unconditional denoising methods. At the algorithmic level, we have presented a
novel Condformer architecture, which effectively embeds the noise prior into
the self-attention module. The noise prior is explicitly estimated using our
LoNPE algorithm or network. Extensive experiments confirm the advantages of
conditional optimization with noise prior, demonstrating that the proposed
LoNPE and Condformer achieve superior performance on both synthetic and real
noise statistics and image denoising tasks, respectively.

Nonetheless, due to the defective sensor and circuits technology, the
formation model of raw sensor noise is actually more sophisticated than the
Poisson-Gaussian noise model. For instance, read noise might follows a heavy-
tailed Cauchy distribution [25] in extremely low-light environments, dark
shading [26] can result from sensor non-uniformity, and more sophisticated
noise models are emerging [85]. Therefore, future work should explore the
estimation of noise prior in these more complex scenarios, aiming to further
enhance noise estimation and denoising performance.

## References

  * [1] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2018, pp. 9446–9454.
  * [2] J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila, “Noise2noise: Learning image restoration without clean data,” in _Int. Conf. Mach. Learn._ , 2018, pp. 2965–2974.
  * [3] S. Anwar and N. Barnes, “Real image denoising with feature attention,” in _IEEE/CVF Int. Conf. Comput. Vis._ , 2019, pp. 3155–3164.
  * [4] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Learning enriched features for fast image restoration and enhancement,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 45, no. 2, pp. 1934–1948, 2022.
  * [5] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2021, pp. 14 821–14 831.
  * [6] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-trained image processing transformer,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2021, pp. 12 299–12 310.
  * [7] J. Liang, J. Cao, G. Sun, K. Zhang, L. V. Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in _IEEE/CVF Int. Conf. Comput. Vis. Worksh._ , 2021.
  * [8] Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li, “Uformer: A general u-shaped transformer for image restoration,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 17 683–17 693.
  * [9] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 5728–5739.
  * [10] A. Abdelhamed, S. Lin, and M. S. Brown, “A high-quality denoising dataset for smartphone cameras,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2018, pp. 1692–1700.
  * [11] J. Anaya and A. Barbu, “Renoir – a dataset for real low-light image noise reduction,” _J. Vis. Commun. Image Represent._ , vol. 51, pp. 144–154, 2018\.
  * [12] T. Plotz and S. Roth, “Benchmarking denoising algorithms with real photographs,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2017, pp. 1586–1595.
  * [13] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [14] K. Sohn, H. Lee, and X. Yan, “Learning structured output representation using deep conditional generative models,” in _Adv. Neural Inform. Process. Syst._ , 2015.
  * [15] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising,” _IEEE Trans. Image Process._ , vol. 26, no. 7, pp. 3142–3155, 2017.
  * [16] Z. Yue, H. Yong, Q. Zhao, D. Meng, and L. Zhang, “Variational denoising network: Toward blind noise modeling and removal,” in _Adv. Neural Inform. Process. Syst._ , 2019.
  * [17] R. Neshatavar, M. Yavartanoo, S. Son, and K. M. Lee, “Cvf-sid: Cyclic multi-variate function for self-supervised image denoising by disentangling noise from image,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 17 583–17 591.
  * [18] A. Foi, M. Trimeche, V. Katkovnik, and K. Egiazarian, “Practical poissonian-gaussian noise modeling and fitting for single-image raw-data,” _IEEE Trans. Image Process._ , vol. 17, no. 10, pp. 1737–1754, 2008.
  * [19] X. Liu, M. Tanaka, and M. Okutomi, “Practical signal-dependent noise parameter estimation from a single noisy image,” _IEEE Trans. Image Process._ , vol. 23, no. 10, pp. 4361–4371, 2014.
  * [20] S. Guo, Z. Yan, K. Zhang, W. Zuo, and L. Zhang, “Toward convolutional blind denoising of real photographs,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 1712–1722.
  * [21] Y. Wang, H. Huang, Q. Xu, J. Liu, Y. Liu1, and J. Wang, “Practical deep raw image denoising on mobile devices,” in _Eur. Conf. Comput. Vis._ , 2020.
  * [22] G. E. Healey and R. Kondepudy, “Radiometric ccd camera calibration and noise estimation,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 16, no. 3, pp. 267–276, 1994.
  * [23] M. Makitalo and A. Foi, “A closed-form approximation of the exact unbiased inverse of the anscombe variance-stabilizing transformation,” _IEEE Trans. Image Process._ , vol. 20, no. 9, pp. 2697–2698, 2011.
  * [24] M. Makitalo and A. Foi, “Optimal inversion of the generalized anscombe transformation for poisson-gaussian noise,” _IEEE Trans. Image Process._ , vol. 22, no. 1, pp. 91–103, 2013.
  * [25] K. Wei, Y. Fu, Y. Zheng, and J. Yang, “Physics-based noise modeling for extreme low-light photography,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 44, no. 11, pp. 8520–8537, 2022.
  * [26] H. Feng, L. Wang, Y. Wang, H. Fan, and H. Huang, “Learnability enhancement for low-light raw image denoising: A data perspective,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 46, no. 1, pp. 370–387, 2024.
  * [27] H. Kim and K. M. Lee, “Nerds: A general framework to train camera denoisers from raw-rgb noisy image pairs,” in _Int. Conf. Learn. Represent._ , 2023\.
  * [28] Z. Yue, H. Yong, Q. Zhao, L. Zhang, D. Meng, and K.-Y. K. Wong, “Deep variational network toward blind image restoration,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2024.
  * [29] D. Zheng, X. Zhang, K. Ma, and C. Bao, “Learn from unpaired data for image restoration: A variational bayes approach,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 45, no. 5, pp. 5889–5903, 2022.
  * [30] K.-C. Chang, R. Wang, H.-J. Lin, Y.-L. Liu, C.-P. Chen, Y.-L. Chang, and H.-T. Chen, “Learning camera-aware noise models,” in _Eur. Conf. Comput. Vis._ , 2020, pp. 343–358.
  * [31] B. Henz, E. S. Gastal, and M. M. Oliveira, “Synthesizing camera noise using generative adversarial networks,” _IEEE Trans. Vis. Comput. Graph._ , vol. 27, no. 3, pp. 2123–2135, 2020.
  * [32] A. Abdelhamed, M. A. Brubaker, and M. S. Brown, “Noise flow: Noise modeling with conditional normalizing flows,” in _IEEE/CVF Int. Conf. Comput. Vis._ , 2019, pp. 3165–3173.
  * [33] A. Maleky, S. Kousha, M. S. Brown, and M. A. Brubaker, “Noise2noiseflow: Realistic camera noise modeling without clean images,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 17 632–17 641.
  * [34] S. Kousha, A. Maleky, M. S. Brown, and M. A. Brubaker, “Modeling srgb camera noise with normalizing flows,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 17 463–17 471.
  * [35] L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” _Physica D: Nonlinear Phenomena_ , vol. 60, no. 1, pp. 259–268, 1992.
  * [36] S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2014, pp. 2862–2869.
  * [37] A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2005, pp. 60–65.
  * [38] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” _IEEE Trans. Image Process._ , vol. 16, no. 8, pp. 2080–2095, 2007.
  * [39] J. Xu, L. Zhang, and D. Zhang, “External prior guided internal prior learning for real-world noisy image denoising,” _IEEE Trans. Image Process._ , vol. 27, no. 6, pp. 2996–3010, 2018.
  * [40] K. Zhang, W. Zuo, and L. Zhang, “Ffdnet: Toward a fast and flexible solution for cnn-based image denoising,” _IEEE Trans. Image Process._ , vol. 27, no. 9, pp. 4608–4622, 2018.
  * [41] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 43, no. 7, pp. 2480–2495, 2021.
  * [42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Adv. Neural Inform. Process. Syst._ , 2017.
  * [43] I. O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit _et al._ , “Mlp-mixer: An all-mlp architecture for vision,” _Adv. Neural Inform. Process. Syst._ , vol. 34, pp. 24 261–24 272, 2021.
  * [44] Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li, “Maxim: Multi-axis mlp for image processing,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2022, pp. 5769–5780.
  * [45] X. Hua, Z. Li, and H. Hong, “An efficient multiscale spatial rearrangement mlp architecture for image restoration,” _IEEE Trans. Image Process._ , vol. 33, pp. 423–438, 2024.
  * [46] Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, and Y. Liu, “Vmamba: Visual state space model,” 2024.
  * [47] H. Guo, J. Li, T. Dai, Z. Ouyang, X. Ren, and S.-T. Xia, “Mambair: A simple baseline for image restoration with state-space model,” _Eur. Conf. Comput. Vis._ , 2024.
  * [48] Y. Mei, Y. Fan, Y. Zhang, J. Yu, Y. Zhou, D. Liu, Y. Fu, T. S. Huang, and H. Shi, “Pyramid attention network for image restoration,” _Int. J. Comput. Vis._ , vol. 131, no. 12, pp. 3207–3225, 2023.
  * [49] C. Zheng, Y. Zhang, J. Gu, Y. Zhang, L. Kong, and X. Yuan, “Cross aggregation transformer for image restoration,” in _Adv. Neural Inform. Process. Syst._ , 2022.
  * [50] J. Xiao, X. Fu, M. Zhou, H. Liu, and Z.-J. Zha, “Random shuffle transformer for image restoration,” in _Int. Conf. Mach. Learn._ , 2023, pp. 38 039–38 058.
  * [51] J. Zhang, Y. Zhang, J. Gu, Y. Zhang, L. Kong, and X. Yuan, “Accurate image restoration with attention retractable transformer,” in _Int. Conf. Learn. Represent._ , 2023.
  * [52] Y. Li, Y. Fan, X. Xiang, D. Demandolx, R. Ranjan, R. Timofte, and L. Van Gool, “Efficient and explicit modelling of image hierarchies for image restoration,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2023, pp. 18 278–18 289.
  * [53] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, “Plug-and-play image restoration with deep denoiser prior,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 44, no. 10, pp. 6360–6376, 2021.
  * [54] Y. Cui, W. Ren, X. Cao, and A. Knoll, “Image restoration via frequency selection,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 46, no. 2, pp. 1093–1108, 2024.
  * [55] S. Chen, J. Zhang, Z. Yu, and T. Huang, “Exploring efficient asymmetric blind-spots for self-supervised denoising in real-world scenarios,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2024, pp. 2814–2823.
  * [56] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” in _Adv. Neural Inform. Process. Syst._ , 2020, pp. 1877–1901.
  * [57] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _Int. Conf. Learn. Represent._ , 2021.
  * [58] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _IEEE/CVF Int. Conf. Comput. Vis._ , 2021.
  * [59] Y. Li, H. Mao, R. Girshick, and K. He, “Exploring plain vision transformer backbones for object detection,” in _Eur. Conf. Comput. Vis._ , 2022, pp. 280–296.
  * [60] R. Strudel, R. Garcia, I. Laptev, and C. Schmid, “Segmenter: Transformer for semantic segmentation,” in _IEEE/CVF Int. Conf. Comput. Vis._ , 2021, pp. 7262–7272.
  * [61] B. Zhang, Z. Tian, Q. Tang, X. Chu, X. Wei, C. Shen _et al._ , “Segvit: Semantic segmentation with plain vision transformers,” _Adv. Neural Inform. Process. Syst._ , vol. 35, pp. 4971–4982, 2022.
  * [62] F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo, “Learning texture transformer network for image super-resolution,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 5791–5800.
  * [63] N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher, “Ctrl: A conditional transformer language model for controllable generation,” _arXiv preprint arXiv:1909.05858_ , 2019.
  * [64] E. Hosseini-Asl, B. McCann, C.-S. Wu, S. Yavuz, and R. Socher, “A simple language model for task-oriented dialogue,” in _Adv. Neural Inform. Process. Syst._ , 2020.
  * [65] T. Brooks, B. Mildenhall, T. Xue, J. Chen, D. Sharlet, and J. T. Barron, “Unprocessing images for learned raw denoising,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 11 036–11 045.
  * [66] Y. Huang, J. Li, Y. Hu, X. Gao, and H. Huang, “Transitional learning: Exploring the transition states of degradation for blind super-resolution,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 45, no. 5, pp. 6495–6510, 2023.
  * [67] G. Yang, N. Fei, M. Ding, G. Liu, Z. Lu, and T. Xiang, “L2m-gan: Learning to manipulate latent space semantics for facial attribute editing,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2021, pp. 2951–2960.
  * [68] V. T. Hu, W. Zhang, M. Tang, P. Mettes, D. Zhao, and C. Snoek, “Latent space editing in transformer-based flow matching,” in _AAAI Conf. Artif. Intell._ , 2024, pp. 2247–2255.
  * [69] E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog. Worksh._ , 2017, pp. 126–135.
  * [70] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in _IEEE/CVF Int. Conf. Comput. Vis._ , 2001, pp. 416–423.
  * [71] K. Ma, Z. Duanmu, Q. Wu, Z. Wang, H. Yong, H. Li, and L. Zhang, “Waterloo exploration database: New challenges for image quality assessment models,” _IEEE Trans. Image Process._ , vol. 26, no. 2, pp. 1004–1016, 2016.
  * [72] R. Franzen, “Kodak lossless true color image suite,” _source: http://r0k. us/graphics/kodak_ , vol. 4, no. 2, 1999.
  * [73] J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2015, pp. 5197–5206.
  * [74] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment:from error visibility to structural similarity,” _IEEE Trans. Image Process._ , vol. 13, no. 4, pp. 600–612, 2004.
  * [75] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _Int. Conf. Learn. Represent._ , 2019.
  * [76] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” in _Int. Conf. Learn. Represent._ , 2017.
  * [77] W. Wu, S. Liu, Y. Xia, and Y. Zhang, “Dual residual attention network for image denoising,” _Pattern Recogn._ , vol. 149, p. 110291, 2024.
  * [78] J. Byun, S. Cha, and T. Moon, “Fbi-denoiser: Fast blind image denoiser for poisson-gaussian noise,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2021, pp. 5768–5777.
  * [79] H. Liang, R. Liu, Z. Wang, J. Ma, and X. Tian, “Variational bayesian deep network for blind poisson denoising,” _Pattern Recogn._ , vol. 143, p. 109810, 2023.
  * [80] J. Cheng, D. Liang, and S. Tan, “Transfer clip for generalizable image denoising,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2024, p. 25974\.
  * [81] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Learning enriched features for real image restoration and enhancement,” in _Eur. Conf. Comput. Vis._ , 2020, pp. 492–511.
  * [82] L. Chen, X. Chu, X. Zhang, and J. Sun, “Simple baselines for image restoration,” in _Eur. Conf. Comput. Vis._ , 2022, pp. 17–33.
  * [83] Y. Gou, P. Hu, J. Lv, J. T. Zhou, and X. Peng, “Multi-scale adaptive network for single image denoising,” in _Adv. Neural Inform. Process. Syst._ , 2022\.
  * [84] A. de Myttenaere, B. Golden, B. Le Grand, and F. Rossi, “Mean absolute percentage error for regression models,” _Neurocomputing_ , vol. 192, pp. 38–48, 2016.
  * [85] Y. Cao, M. Liu, S. Liu, X. Wang, L. Lei, and W. Zuo, “Physics-guided iso-dependent sensor noise modeling for extreme low-light photography,” in _IEEE/CVF Conf. Comput. Vis. Pattern Recog._ , 2023, pp. 5744–5753.

|  Yuanfei Huang (Member, IEEE) received the Ph.D. degree from the School of
Electronic Engineering, Xidian University, Xi’an, China, in 2021. He is
currently a Lecturer with the School of Artificial Intelligence, Beijing
Normal University, Beijing, China. His research interests include image and
video processing, computer vision, and machine learning. In these areas, he
has published technical articles in referred journals and proceedings
including IEEE TPAMI / TIP, etc.
---|---
|  Hua Huang (Senior Member, IEEE) received his B.S. and Ph.D. degrees from
Xi’an Jiaotong University, Xian, China, in 1996 and 2006, respectively. He is
currently a Professor with School of Artificial Intelligence, Beijing Normal
University, Beijing, China. His current research interests include image and
video processing, computational photography, and computer graphics. He
received the Best Paper Award of ICML2020 / EURASIP2020 / PRCV2019 /
ChinaMM2017.
---|---