# Variational Deep Image Denoising

Jae Woong Soh
Department of ECE, INMC
Seoul National University
<EMAIL_ADDRESS>Nam Ik Cho
Department of ECE, INMC
Seoul National University
<EMAIL_ADDRESS>
###### Abstract

Convolutional neural networks (CNNs) have shown outstanding performance on
image denoising with the help of large-scale datasets. Earlier methods naïvely
trained a single CNN with many pairs of clean-noisy images. However, the
conditional distribution of the clean image given a noisy one is too
complicated and diverse, so that a single CNN cannot well learn such
distributions. Therefore, there have also been some methods that exploit
additional noise level parameters or train a separate CNN for a specific noise
level parameter. These methods separate the original problem into easier sub-
problems and thus have shown improved performance than the naïvely trained
CNN. In this step, we raise two questions. The first one is whether it is an
optimal approach to relate the conditional distribution only to noise level
parameters. The second is what if we do not have noise level information, such
as in a real-world scenario. To answer the questions and provide a better
solution, we propose a novel Bayesian framework based on the variational
approximation of objective functions. This enables us to separate the
complicated target distribution into simpler sub-distributions. Eventually,
the denoising CNN can conquer noise from each sub-distribution, which is
generally an easier problem than the original. Experiments show that the
proposed method provides remarkable performance on additive white Gaussian
noise (AWGN) and real-noise denoising while requiring fewer parameters than
recent state-of-the-art denoisers.

##  1 Introduction

Image denoising has been an important task due to inevitable noise corruption
in the image acquisition process. It aims to reconstruct the underlying clean
image from a noisy one. The observed noise from an imaging device is generally
the accumulation of multiple noises from the sources such as capturing sensors
and on-device image processing pipelines. Since the noise generation process
is too complicated to be accurately modeled, the noise is usually assumed to
be an AWGN based on the central limit theorem.

Recently, CNNs have shown great success in removing AWGN from images [48, 8,
29, 49, 26, 34], largely surpassing traditional methods such as total
variation, K-SVD denoising, NLM, BM3D, WNNM, etc. [36, 10, 6, 9, 15, 43].
However, earlier CNN-based methods were mostly non-blind ones, which need
separate models trained (fitted) to different noise levels. When the input
image bears a noise with a different level from the trained one, a severe
performance drop occurs due to domain discrepancy between the distributions of
training and test images. This phenomenon degrades its reliability and limits
practical applications.

A naïve approach to alleviating this problem is to compose training data
consisting of images having a wide range of noise levels and train a single-
model blind denoiser that can cope with broad noise levels [48, 38, 24].
However, the blind denoiser is generally not as good as the non-blind one that
is well-fitted to the given noise level because learning a conditional mapping
between the clean and noisy images with diverse distribution is more difficult
than the learning of simpler distribution. Another approach is to develop
flexible networks that can deal with multiple noise levels by exploiting
additional information [49, 21]. In this case, a noise level estimator is
additionally needed along with the denoising CNN, and hence we will refer to
this approach as a two-stage system. One of its drawbacks is that they do not
work properly when the noise level estimator fails to provide accurate
information.

So far, it has been hypothesized that dividing a complex distribution into
simpler sub-distributions will make a CNN easy to learn the overall task,
eventually bringing performance gain. Also, providing additional information
to a CNN is regarded as to ease the target distribution into simpler ones. For
AWGN removal, the noise level has been provided as additional prior
information without a doubt, which enables to separate the overall problem
into sub-problems corresponding to specific noise levels. However, there can
be a better way to divide the distribution, for example, by reflecting the
image semantics [40].

Meanwhile, the distribution of noise from imaging devices referred to as
“real-noise,” largely deviates from the i.i.d. Gaussian distribution. Hence,
denoisers trained with Gaussian noise do not perform well for the real-noises,
and thus they have limitations in practical, real-world situations. Therefore,
very recently, the real-noise denoising task attracted researchers, and
several works have been proposed [16, 5, 4, 44, 45, 22]. Dealing with real-
noise is a more challenging task than the AWGN in two aspects.

First, it is hard to build a paired clean-noisy dataset for real-noise
denoising. Since the distribution of real-noise has more complicated features
than the Gaussian, _e.g._ , multi-modal, signal-dependent, spatially variant,
etc., accurate modeling for real-world noises has also been a longstanding
problem [12, 11, 28, 32, 47, 42]. Specifically, heteroscedastic or Poisson
mixture models have been considered for reflecting some of these features [12,
11, 28, 47]. Recently, some researchers have taken many pairs of clean and
real-noisy images with careful image acquisition settings [2, 33]. These
methods can alleviate the data scarcity problem, but such acquisition
processes are costly and labor-intensive. Some other works have modeled the
noise corruption process based on the generative adversarial network (GAN)
[7], normalizing flows [1], or prior knowledge on camera pipeline and noise
properties [16, 5, 42].

Second, learning such a complex distribution may be a burdensome task to a
single CNN. There have been some researches to address the previous issue,
i.e., precise real-noise modeling or acquiring well-registered real noisy-
clean image pairs, but very few researchers considered how to relieve the
hardness of training or make the best use of the dataset. Notably, most
previous real-noise denoisers correspond to the category of naïve blind
denoisers.

To address the above issues, we propose a new method that can handle blind
scenarios, including synthetic AWGN and real-world noise, namely Variational
Deep Image Denoiser (VDID). Our approach to solving the denoising problem is
“divide-and-conquer.” We split the original objective into simpler sub-
problems, which eventually ease the overall task. For the Gaussian denoising,
we raise doubt about conventional approaches that find different denoisers or
control the features according to the noise level only. Instead, we seek an
optimal data-driven prior or criterion to bring out the best performance with
a single CNN. For the real-noise denoising, we tackle the problem differently
from the traditional methods. Instead of looking for a good model for the
real-noise distribution, we relax the original problem to simpler ones by
dividing the underlying complex posterior distribution into sub-modal
distributions. Specifically, we formulate our objective in terms of maximum a
posterior (MAP) inference and present an approximated form of the objective by
introducing a latent variable based on variational Bayes. By doing so, the
network learns its latent space, which represents the sub-distributions of the
noisy images. In other words, we divide the original problem into several sub-
problems and solve each case separately. In particular, we introduce a latent
representation of noisy images and exploit their representations as an
additional prior to handle them differently. Furthermore, our method is
trained in an end-to-end scheme without any additional noise information.

In summary, our contributions are as follows.

  * •

We present a novel CNN-based blind image denoiser, which is trained in an end-
to-end scheme.

  * •

To the best of our knowledge, we first tackle the image denoising as to
relaxing the original problem into easier ones.

  * •

Based on the variational approximation, we reformulate our target problem to
include the auto-encoding term, which incorporates underlying noisy image
distribution.

  * •

Based on the latent space implying the noisy image manifolds, our VDID can
focus on simpler sub-distributions of the original problem.

  * •

From the extensive experiments, we have shown that the proposed method
achieves state-of-the-art performances while requiring fewer parameters.

##  2 Related Work

Based on the assumption of image noise distribution as pixel-wise i.i.d.
Gaussian with a standard deviation of $\sigma_{N}$, AWGN denoising has been a
longstanding problem. Recently, CNN-based denoisers have also been actively
studied [48, 8, 29, 49, 21, 26, 34], and noise removal in real-world images
attracted researchers due to its practical importance [7, 16, 5, 1, 45, 42,
33, 2, 4, 22]. From the viewpoint of objectives, we divide former related
approaches into three categories: Specific Non-blind Model, Naïve Blind Model,
and Two-Stage Blind Model. For the entire paper, we will denote the noisy
image as $\mathbf{y}$, the clean image $\mathbf{x}$, and the underlying data
distribution $p_{data}$.

###  2.1 Specific Non-blind Model

Most CNN-based AWGN denoisers such as DnCNN-S, RED, and NLRN [48, 29, 26]
adopt “specific non-blind model,” which is a separate network trained for a
specific noise level. Its objective can be expressed as

$\hat{\theta_{i}}=\arg\max_{\theta_{i}}\mathbb{E}_{p_{data}(\mathbf{x},\mathbf{y},\sigma_{N})}[\log
p_{\theta_{i}}(\mathbf{x}|\mathbf{y},\sigma_{N}^{i})],$ (1)

where $\theta_{i}$ denotes the parameters of the $i$-th network. Specifically,
most previous works have taken one of two choices for the noise levels:
$\sigma_{N}^{i}\in\\{15,25,50\\}$ or $\\{10,30,50,70\\}$. At the inference,
the noise level information of a test image is required for obtaining a
desirable output. Also, a number of networks should be prepared in the bag,
requiring a large memory space.

###  2.2 Naïve Blind Model

Most blind denoisers are in this category of naïve blind model, totally
relying on the representation power of CNN. Especially, since exploiting the
information such as noise level is very difficult in the case of real-noises,
it would be an appropriate approach to adopt the naïve blind model. The
objective of this approach is expressed as

$\hat{\theta}=\arg\max_{\theta}\mathbb{E}_{p_{data}(\mathbf{x},\mathbf{y})}[\log
p_{\theta}(\mathbf{x}|\mathbf{y})],$ (2)

for training a single network parameterized by $\theta$ to capture the
conditional distribution. In general, naïve blind models show worse
performance than the specific models [48, 13], since the distribution
$p(\mathbf{x}|\mathbf{y})$ is more complicated than
$p(\mathbf{x}|\mathbf{y},\sigma_{N})$. Specifically, DnCNN-B, UNLNet, GCBD,
and RIDNet [48, 24, 7, 4] come into this category.

###  2.3 Two-Stage Blind Model

Some recent methods adopt the two-stage blind model, where noise level
parameters $\mathbf{c}$ are first estimated and then fed to the denoising
network. Its objective is

$\displaystyle\hat{\phi}=$
$\displaystyle\arg\max_{\phi}\mathbb{E}_{p_{data}(\mathbf{y},\mathbf{c})}[\log
p_{\phi}(\mathbf{c}|\mathbf{y})],$ (3) $\displaystyle\hat{\theta}=$
$\displaystyle\arg\max_{\theta}\mathbb{E}_{p_{data}(\mathbf{x},\mathbf{y},\mathbf{c})}[\log
p_{\theta}(\mathbf{x}|\mathbf{y},\mathbf{c})]],$ (4)

where $\phi$ and $\theta$ denote the parameters of noise estimator and
denoiser, respectively. For the Gaussian noise, $\mathbf{c}$ is selected as
the standard deviation of Gaussian distribution [49, 21], and for the real-
noise, more complicated parameters are selected [16, 44, 22]. Notably, prior
information on noise is additionally required for this setting.

##  3 Variational Deep Image Denoising

This section presents our objective in terms of MAP inference and then
reformulates the problem to tractable sub-problems.

###  3.1 Problem Statement

Given a noisy image $\mathbf{y}$, the objective is to find a latent clean
image $\mathbf{x}$. The MAP inference for this problem is

$\mathbf{\hat{x}}=\arg\max_{\mathbf{x}}\log p(\mathbf{x}|\mathbf{y}).$ (5)

Most of the traditional approaches divide the posterior term into likelihood
and prior terms as

$\displaystyle\arg\max_{\mathbf{x}}\log p(\mathbf{x}|\mathbf{y})$
$\displaystyle=\arg\max_{\mathbf{x}}\log p(\mathbf{y}|\mathbf{x})+\log
p(\mathbf{x})$
$\displaystyle=\arg\min_{\mathbf{x}}\frac{1}{2\sigma^{2}}||\mathbf{y}-\mathbf{x}||_{2}^{2}+\Phi(\mathbf{x}),$
(6)

and solve it with a well-designed prior $\Phi(\mathbf{x})$ under the i.i.d.
Gaussian assumption. However, we do not follow this approach since we use a
data-driven discriminative learning scheme. Also, it needs to be noted that
the traditional approaches require a complex data likelihood term, which is
hard to approximate the likelihood of real noisy images.

We first introduce a new latent random variable $\mathbf{c}$, which implies
suitable information both for denoising (task-relevant information) and
properties of clean/noisy image (domain-relevant information). Then, we bring
an inference problem of the posterior $p(\mathbf{c}|\mathbf{x},\mathbf{y})$,
in which the latent $\mathbf{c}$ includes both the domain- and task-relevant
information learned from clean and noisy images. However, this inference
problem is intractable. Also, our other objective is to infer $\mathbf{x}$,
which cannot be observed during the inference. In summary, our interested
inference problems are $\log p(\mathbf{c}|\mathbf{x},\mathbf{y})$ and $\log
p(\mathbf{x}|\mathbf{y})$, which are intractable or unobservable.

###  3.2 Proposed Variational Lower Bound

To approximate the posterior $p(\mathbf{c}|\mathbf{x},\mathbf{y})$, we
introduce a tractable probability distribution $q(\mathbf{c}|\mathbf{y})$.
Then, the joint probability distribution $\log p(\mathbf{x},\mathbf{y})$ can
be reformulated as

$\displaystyle\log p(\mathbf{x},\mathbf{y})=$
$\displaystyle\mathbb{E}_{\mathbf{c}\sim q(\mathbf{c}|\mathbf{y})}[\log
p(\mathbf{x}|\mathbf{y},\mathbf{c})]$
$\displaystyle+D_{KL}(q(\mathbf{c}|\mathbf{y})||p(\mathbf{c}|\mathbf{x},\mathbf{y}))$
$\displaystyle-D_{KL}(q(\mathbf{c}|\mathbf{y})||p(\mathbf{c}))$
$\displaystyle+\mathbb{E}_{\mathbf{c}\sim q(\mathbf{c}|\mathbf{y})}[\log
p(\mathbf{y}|\mathbf{c})],$ (7)

with some prior distribution $p(\mathbf{c})$. To approximate the intractable
KL divergence term between $q(\mathbf{c}|\mathbf{y})$ and
$p(\mathbf{c}|\mathbf{x},\mathbf{y})$, we introduce a _variational lower
bound_ $\mathcal{L}$.

Definition 1 _Variational lower bound_ $\mathcal{L}$ is defined as

$\displaystyle\mathcal{L}=$ $\displaystyle\mathbb{E}_{\mathbf{c}\sim
q(\mathbf{c}|\mathbf{y})}[\log p(\mathbf{x}|\mathbf{y},\mathbf{c})]$
$\displaystyle-
D_{KL}(q(\mathbf{c}|\mathbf{y})||p(\mathbf{c}))+\mathbb{E}_{\mathbf{c}\sim
q(\mathbf{c}|\mathbf{y})}[\log p(\mathbf{y}|\mathbf{c})].$ (8)

Theorem 1 Given a noisy image $\mathbf{y}$ and its underlying clean image
$\mathbf{x}$, the joint log-distribution $\log p(\mathbf{x},\mathbf{y})$ can
be reformulated including variational lower bound $\mathcal{L}$ as

$\log
p(\mathbf{x},\mathbf{y})=\mathcal{L}+D_{KL}(q(\mathbf{c}|\mathbf{y})||p(\mathbf{c}|\mathbf{x},\mathbf{y})).$
(9)

Then,

$\log p(\mathbf{x},\mathbf{y})\geq\mathcal{L}.$ (10)

(The overall derivation and proof can be found in _appendix_.)

Definition 2 We define a log-posterior $q(\mathbf{x}|\mathbf{y})$ which
approximates the original posterior $p(\mathbf{x}|\mathbf{y})$ as

$\displaystyle q(\mathbf{x}|\mathbf{y})$
$\displaystyle=\int_{\mathbf{c}}q(\mathbf{x},\mathbf{c}|\mathbf{y})d\mathbf{c}$
$\displaystyle=\int_{\mathbf{c}}p(\mathbf{x}|\mathbf{y},\mathbf{c})q(\mathbf{c}|\mathbf{y})d\mathbf{c}.$
(11)

Then, the MAP inference given $\mathbf{y}$ can be done by
$\arg\max_{\mathbf{x}}\log q(\mathbf{x}|\mathbf{y})$.

Through our reformulation, maximizing the joint probability, and the objective
to minimize the KL divergence between our variational distribution and the
posterior on $\mathbf{c}$, are simultaneously approximated as to maximize our
_variational lower bound_. Notably, the first term of $\mathcal{L}$ is the
only term relevant to the relation between $\mathbf{x}$ and $\mathbf{y}$,
responsible for denoising or reconstruction. The other terms are
regularization terms, which impose constraints on the latent variable
$\mathbf{c}$. The second term is the KL divergence, which constrains the
latent distribution, and the third term is the auto-encoder reconstruction
term of noisy images.

As neural networks are experts in inference, in an amortized way [35, 46], we
employ three CNNs parameterized by $\theta$, $\phi_{E}$, and $\phi_{D}$ for
the variational inference. Specifically, our final objective is

$\displaystyle\arg\max_{\theta,\phi_{E},\phi_{D}}$
$\displaystyle\mathbb{E}_{p_{data}(\mathbf{x},\mathbf{y})}[\mathbb{E}_{\mathbf{c}\sim
q_{\phi_{E}}(\mathbf{c}|\mathbf{y})}[\log
p_{\theta}(\mathbf{x}|\mathbf{y},\mathbf{c})]$ $\displaystyle-
D_{KL}(q_{\phi_{E}}(\mathbf{c}|\mathbf{y})||p(\mathbf{c}))$
$\displaystyle+\mathbb{E}_{\mathbf{c}\sim
q_{\phi_{E}}(\mathbf{c}|\mathbf{y})}[\log
p_{\phi_{D}}(\mathbf{y}|\mathbf{c})]],$ (12)

with the underlying empirical data distribution
$p_{data}(\mathbf{x},\mathbf{y})$. By introducing such regularizations, our
denoiser can approximately solve a MAP problem according to the latent
$\mathbf{c}$, where $\mathbf{c}$ is the variable involved in the noisy image
generation process. In other words, our denoiser divides the problem according
to the latent $\mathbf{c}$, where $\mathbf{c}$ should “imply” the noisy image
manifold.

###  3.3 Network Architecture

Figure 1: The overall architecture of proposed VDID, where $k$, $s$, and $n$
denote kernel size, stride, and the number of filters, respectively.

The overall network architecture is shown in Figure 1. The denoiser takes a
noisy image $\mathbf{y}$ with the latent variable $\mathbf{c}$ concatenated
along the channel axis to infer the clean image $\mathbf{x}$. The denoiser is
fully convolutional, thus highly scalable. For the denoiser, the residual
block (ResBlock) is adopted as the basic building block [17, 25]. Precisely,
the same residual block of [25] is adopted, which consists of $3\times 3$
convolution layers of $64$ filters followed by the rectified linear unit
(ReLU) and another convolution layer (Conv-ReLU-Conv). Then, the input is
added to the output of the convolution layer, which forms the skip-connection.
The $N$ number of residual blocks and one convolution layer compose the
residual-in-residual block (RIRBlock) [50]. The denoiser consists of $D$
RIRBlocks with some convolution layers and a long skip-connection, as shown in
Figure 1. The last convolution layer infers the residual image (noise) instead
of the clean image itself, according to [48].

The encoder and decoder networks are simple feedforward convolutional networks
without skip-connection. The encoder decreases the feature map’s spatial size
twice (one-fourth of its height and width), and the output $\mathbf{c}$ has
four channels. For the differentiable Monte Carlo, we adopt the
reparameterization trick [23, 18] as

$\epsilon\sim\mathcal{N}(0,I),\quad\mathbf{c}=\epsilon\odot\sigma+\mu,$ (13)

where $\odot$ denotes Hadamard product, and $\mu$ and $\sigma$ are the
encoder’s outputs. The decoder network has symmetrical architecture as the
encoder. The details of the architectures are presented in _appendix_.

###  3.4 Corresponding Loss Terms

#### First Term

For the first term of our objective, which is the denoising term, we adopt
mean absolute error (MAE) between the ground-truth clean image and the
inferred output [25] to minimize the distortion. We denote the corresponding
loss term as $\mathcal{L}_{denoise}$:

$\mathcal{L}_{denoise}=||\mathbf{x}-\hat{\mathbf{x}}||_{1},$ (14)

where $\hat{\mathbf{x}}$ denotes the output of the denoiser.

#### Second Term

The KL divergence between the prior distribution and the posterior can be
calculated analytically. We set the prior $p(\mathbf{c})$ as Gaussian
distribution with zero mean and unit covariance. Thus, the KL divergence term
is

$D_{KL}(q(\mathbf{c}|\mathbf{y})||p(\mathbf{c}))=D_{KL}(\mathcal{N}(\mu,\Sigma)||\mathcal{N}(0,I)).$
(15)

#### Third Term

As the third term, which is the auto-encoder reconstruction term, we first
minimize MAE between the noisy input image and the decoder’s output. However,
using only pixel-wise loss strictly assumes $p(\mathbf{y}|\mathbf{c})$ to be a
family of the probability distribution of Laplacian or Gaussian. To relax and
better learn the noisy image distribution, we adopt additional adversarial
loss [14]. Specifically, we adopt non-saturating GAN loss [14], corresponding
to minimizing Jensen-Shannon divergence between $p_{data}(\mathbf{y})$ and
$p_{\phi_{D}}(\mathbf{y}|\mathbf{c})$. For the AWGN removal, we add our known
prior that the latent space should include the noise level information. Hence,
we add a noise level estimation loss. Eventually, the latent $\mathbf{c}$
works as a prior for the denoiser, which contains an abstract of noisy image
distribution.

The corresponding loss term $\mathcal{L}_{recon}$ is described as

$\mathcal{L}_{recon}=||\mathbf{y}-\hat{\mathbf{y}}||_{1}+\lambda_{1}\mathcal{L}_{adv}+\lambda_{2}||\sigma_{N}-EST(\mathbf{c})||_{1}$
(16)

where $\hat{\mathbf{y}}$ denotes the output of the decoder, and $EST(\cdot)$
is a simple two-layer CNN (conv-relu-conv).

#### Overall Loss

The overall loss is the sum of three terms, with the normalized KL divergence
multiplied by $\beta$ [18],

$\arg\min_{\theta,\phi_{E},\phi_{D}}\mathbb{E}_{p_{data}(\mathbf{x},\mathbf{y})}[\mathcal{L}_{denoise}+\beta
D_{KL}+\mathcal{L}_{recon}].$ (17)

###  3.5 Discussions

#### Probabilistic View

Let us assume the denoising loss term as L2-norm for simplicity, which assumes
i.i.d. Gaussian distribution as target probabilistic family (L1-norm is
associated with Laplacian distribution). We refer a neural network
parameterized by $\theta$ as $f_{\theta}(\cdot)$.

The naïve blind model models
$p_{\theta}(\mathbf{x}|\mathbf{y})=\mathcal{N}(f_{\theta}(\mathbf{y}),I/2)$,
and learning the data distribution under this family hinders expressiveness.
Since the posterior distribution $p(\mathbf{x}|\mathbf{y})$ including diverse
degradation is too complex to be captured by a single Gaussian, its
performance cannot be expected as much as the specific model with simpler
$p(\mathbf{x}|\mathbf{y})$ with a single degradation. On the other hand, our
framework models
$p(\mathbf{x}|\mathbf{y},\mathbf{c})=\mathcal{N}(f_{\theta}(\mathbf{y},\mathbf{c}),I/2)$,
and it is still the Gaussian form. But it learns different mean values with
respect to $\mathbf{c}$, which grants more representation power by learning
multiple Gaussians in accordance with $\mathbf{c}$. Then, the marginal
posterior has more representative power
$p(\mathbf{x}|\mathbf{y})=\int_{\mathbf{c}}p(\mathbf{x}|\mathbf{y},\mathbf{c}),p(\mathbf{c}|\mathbf{y})d\mathbf{c}$.
Though, our inference approximates the marginal posterior through Monte-Carlo
using only one sample of $\mathbf{c}$.

The two-stage model can be considered a special case of our method where
$q(\mathbf{c}|\mathbf{y})$ is chosen as deterministic. In this case, the $c$
is determined based on the prior knowledge and carefully modeled by
“understanding the data.” Then, the point estimate of
$\mathbf{\hat{c}}=\arg\max_{\mathbf{c}}q(\mathbf{c}|\mathbf{y})$ is used for
the second step inference [37]. Note that this bi-level optimization scheme
would be sub-optimal to the task objective compared to the joint optimization.
Unlike the two-stage model, our method is more Bayesian and implicitly learns
$\mathbf{c}$, which is enforced to contain the degradation information along
with the original image content information. In other words, our method
conducts Bayesian inference whereas the two-stage model conducts deterministic
estimates, and also the additional information $\mathbf{c}$ is learned by
enforcing the network to “understand the data.”

#### Connection to Blind AWGN Denoiser

For the AWGN, where $q(\mathbf{c}|\mathbf{y})$ is chosen as deterministic, the
proposed method can be regarded as a two-stage blind denoiser, _i.e._ , a
noise level estimator (corresponding to the encoder) and a flexible denoiser
(corresponding to the denoiser) that works for a range of noise levels. Hence,
a two-stage blind denoiser can be considered a special case of our approach.
In this case, the $\mathbf{c}$ is carefully determined by “understanding the
data.” For AWGN, it is chosen as a standard deviation of Gaussian
distribution, based on the prior knowledge. The point estimate of
$\mathbf{\hat{c}}=\arg\max_{\mathbf{c}}q(\mathbf{c}|\mathbf{y})$ is used for
the second step inference [37]. Note that this bi-level optimization scheme
would be sub-optimal to the task objective, compared to the joint
optimization. On the other hand, our method implicitly learns $\mathbf{c}$,
which enforces the network to “understand the data.”

#### Discussions on Loss Terms

When we use only the first term, it is just a naïve approach to solve a blind
denoising problem, totally relying on the discriminative power of CNN. In this
case, the objective is

$\arg\max_{\theta,\phi_{E}}\mathbb{E}_{p_{data}(\mathbf{x},\mathbf{y})}[\mathbb{E}_{\mathbf{c}\sim
q_{\phi_{E}}(\mathbf{c}|\mathbf{y})}[\log
p_{\theta}(\mathbf{x}|\mathbf{y},\mathbf{c})]].$ (18)

It is notable that no matter what latent distribution
$q_{\phi_{E}}(\mathbf{c}|\mathbf{y})$ we choose, this criteria is maximized if
for each $\mathbf{c}$, $\mathbb{E}_{p_{data}(\mathbf{x},\mathbf{y})}[\log
p_{\theta}(\mathbf{x}|\mathbf{y},\mathbf{c})]$ is maximized. In other words,
there is a trivial solution “independent” to $\mathbf{c}$, if our model has
the optimal parameter satisfying
$\theta^{*}=\arg\max_{\theta}\mathbb{E}_{p_{data}(\mathbf{x},\mathbf{y})}[\log
p_{\theta}(\mathbf{x}|\mathbf{y})]$. In this case, $\mathbf{c}$ collapses and
the original problem cannot be divided into sub-problems. The second term
gives regularization constraints, where the KL divergence term forces
disentanglement of $\mathbf{c}$, giving “discriminative power” over observed
noisy images $\mathbf{y}$. The third term further gives constraints on the
latent variable $\mathbf{c}$. As the auto-encoder reconstruction term forces
the reconstruction from $\mathbf{c}$ to $\mathbf{y}$, this term forces
$\mathbf{c}$ to include the information on a noisy image.

Table 1: The average PSNR on AWGN denoising. The best results are highlighted
in red and the second best in blue.

Noise level | Dataset | CBM3D [9] | RED [29] | CDnCNN [48] | FFDNet [49] | UNLNet [24] | VDN [44] | VDID (Ours)
---|---|---|---|---|---|---|---|---
$\sigma_{N}=10$ | CBSD68 | 35.91 | 33.89 | 36.13 | 36.14 | 36.20 | 36.29 | 36.34
| Kodak24 | 36.43 | 34.73 | 36.46 | 36.69 | - | 36.85 | 37.02
| Urban100 | 36.00 | 34.42 | 34.61 | 35.78 | - | 35.97 | 36.30
$\sigma_{N}=30$ | CBSD68 | 29.73 | 28.45 | 30.34 | 30.32 | 30.21 | 30.64 | 30.64
| Kodak24 | 30.75 | 29.53 | 31.17 | 31.27 | 31.18 | 31.67 | 31.74
| Urban100 | 30.36 | 28.84 | 30.00 | 30.53 | 30.41 | 31.14 | 31.41
$\sigma_{N}=50$ | CBSD68 | 27.38 | 26.34 | 27.95 | 27.97 | 27.85 | 28.33 | 28.33
| Kodak24 | 28.46 | 27.42 | 28.83 | 28.98 | 28.86 | 29.44 | 29.49
| Urban100 | 27.94 | 26.25 | 27.59 | 28.05 | 27.95 | 28.86 | 29.10
$\sigma_{N}=70$ | CBSD68 | 26.00 | 25.09 | 25.66 | 26.55 | - | 26.93 | 26.94
| Kodak24 | 27.09 | 26.16 | 26.36 | 27.56 | - | 28.05 | 28.10
| Urban100 | 26.31 | 24.58 | 25.24 | 26.40 | - | 27.31 | 27.55
|  |  |  |  |  |  |  |

##  4 Experimental Results

We perform denoising experiments on Gaussian and real noises. For our VDID, we
set $N=5$ and $D=5$, which amounts to about $2.2$ M parameters, including the
denoiser and the encoder. All the results are evaluated in sRGB space and
demonstrated with PSNR and SSIM [41].

###  4.1 Implementation Details

For the training, we extract patches with the size of $96\times 96$ from
training images for AWGN and $256\times 256$ for real-noise. We adopt Adam
optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. For the loss term, we
set $\beta=0.01,\lambda_{1}=0.001$, and $\lambda_{2}=1$ for AWGN,
$\lambda_{2}=0$ for real-noise. For data augmentation, a random flip and
$90\degree$ rotations of the patches are applied. The initial learning rate is
$2\times 10^{-4}$ and decayed half in every $100,000$ iterations, until it
reaches $2\times 10^{-5}$. The batch size is set to $32$ for AWGN and $4$ for
real-noise.

###  4.2 Results on AWGN Removal

For training, we use DIV2K [3] training dataset which includes $800$ high-
resolution images, and add synthetic Gaussian noise with noise level
$\sigma_{N}\in[5,70]$. The performance is evaluated with three color-image
datasets: CBSD68 [30], Kodak24, and Urban100 [19] with noise levels
$\sigma_{N}=10,~{}30,~{}50,~{}70$. We compare our method with several AWGN
denoising algorithms: CBM3D [9], RED [29], CDnCNN [48], FFDNet [49], UNLNet
[24], and VDN [44]. The results are presented in Table 1.

Note that CBM3D [9], RED [29], and FFDNet [49] are non-blind methods, whereas
the rest are blind ones. In most cases, VDN [44] and our VDID show the best
PSNR results, but we note that our VDID needs a smaller number of parameters
(2.2 M) compared to VDN [44] (7.8 M). In conclusion, the results show that our
VDID surpasses other methods considering the tradeoff between the performance
and the number of parameters.

###  4.3 Results on Real-Noise Removal

We use training images of Smartphone Image Denoising Dataset (SIDD) [2], which
is a collection of pairs of noisy and clean images from five smartphone
cameras. It consists of $320$ image pairs for training. To augment more
datasets of synthesized images, we also used DIV2K [3] training images, which
includes $800$ high-resolution images. To generate noisy images, we adopt the
noise synthesis process of CBDNet [16]. We compare with several image
denoising methods: BM3D [9], WNNM [15], DnCNN [48], TNRD [8], FFDNet [49],
GCBD [7], CBDNet [16], RIDNet [4], VDN [44], and AINDNet [22]. For the
evaluation, we use two widely-used real image denoising benchmarks.

  * •

SIDD: SIDD provides $1,280$ small patches for validation and $1,280$ for test
benchmark, which are visually similar to training images. Ground-truth patches
for the validation set are provided, but not for the test set.

  * •

DND: Darmstadt Noise Dataset (DND) consists of $50$ images with real-noise
from $50$ scenes from four consumer cameras. Then, the images are further
cropped by the provider, which results in $1,000$ small patches with a size of
$512\times 512$.

Table 2: Results on SIDD [2] benchmark. The best results are highlighted in
bold.

Method | Blind/Non-blind | Parameters | PSNR | SSIM
---|---|---|---|---
BM3D [9] | Non-blind | - | 25.65 | 0.685
WNNM [15] | Non-blind | - | 25.78 | 0.809
DnCNN [48] | Non-blind | 668 K | 23.66 | 0.583
TNRD [8] | Non-blind | 27 K | 24.73 | 0.643
CBDNet [16] | Blind | 4.4 M | 33.28 | 0.868
RIDNet [4] | Blind | 1.5 M | 38.71 | 0.914
VDN [44] | Blind | 7.8 M | 39.26 | 0.955
AINDNet+TF [22] | Blind | 13.7 M | 38.95 | 0.952
VDID (Ours) | Blind | 2.2 M | 39.25 | 0.955
VDID+ (Ours) | Blind | 2.2 M | 39.33 | 0.956
|  |  |  |

Table 3: Results on DND [33] benchmark. The best results are highlighted in
bold.

Method | Blind/Non-blind | Parameters | PSNR | SSIM
---|---|---|---|---
BM3D [9] | Non-blind | - | 34.51 | 0.8507
WNNM [15] | Non-blind | - | 34.67 | 0.8646
DnCNN+ [48] | Non-blind | 668 K | 37.90 | 0.9430
FFDNet+ [49] | Non-blind | 825 K | 37.61 | 0.9415
GCBD [7] | Blind | 561 K | 35.58 | 0.9217
CBDNet [16] | Blind | 4.4 M | 38.06 | 0.9421
RIDNet [4] | Blind | 1.5 M | 39.26 | 0.9528
VDN [44] | Blind | 7.8 M | 39.38 | 0.9518
AINDNet(S) [22] | Blind | 13.7 M | 39.53 | 0.9561
VDID (Ours) | Blind | 2.2 M | 39.63 | 0.9528
VDID+ (Ours) | Blind | 2.2 M | 39.69 | 0.9532
|  |  |  |

Overall quantitative comparisons on two benchmarks are listed in Table 2 and
3. It is observed that our method shows better results than others, on both
benchmarks in terms of PSNR and SSIM [41]. Note that we also demonstrate
results with self-ensemble [39] based on geometric transformation, which is
denoted with ‘+’ sign.

For a fair comparison, we also denote the number of parameters for the CNN-
based methods. Since our VDID solves simpler sub-problems conditioned on the
latent variable $\mathbf{c}$, it requires a smaller network than other
denoisers based on naïve blind setting. In other words, the problem given to
other methods is more complicated due to the difficulty of ill-posed real
image denoising. Thus, our method achieves state-of-the-art performances in
real-world image denoising while requiring fewer parameters than others.

###  4.4 Visualized Results

For qualitative evaluation, we present a visual comparison in Figure 2. It
shows that denoisers trained with AWGN, such as DnCNN [48] and FFDNet [49],
fail to remove the noise or tend to over-smooth the result, suffering from the
discrepancy between the target noise distributions. On the other hand, methods
for real-noise removal show more plausible results. However, most methods tend
to over-smooth the patterns as shown in the green box, whereas our VDID shows
better-restored line patterns. Also, our method shows clearer results in the
text region of the red box. Through the overall visualization, our method
shows visually pleasing results on real-world images. More visualized
comparisons are presented in _appendix_.

Noisy
30.42/0.5997

DnCNN [48]
38.19/0.9051

FFDNet [49]
38.32/0.9083

CBDNet [16]
38.74/0.9132

RIDNet [4]
39.55/0.9228

VDN [44]
39.87/0.9277

AINDNet [22]
39.00/0.9148

VDID+ (Ours)
39.98/0.9285

Figure 2: Visualized examples from DND benchmark [33] with PSNR/SSIM.

(a) Noisy image $\sigma_{N}=30$
19.72 dB

(b) Restoration of (a)
27.20 dB

(c) Restoration with
$\mathbf{c}$ from $\sigma_{N}=10$
23.95 dB

(d) Restoration with $\mathbf{c}$ from $\sigma_{N}=50$
24.93 dB

(e) Restoration with
$\mathbf{c}$ from flipped input
26.74 dB

Figure 3: Denoising results depending on $\mathbf{c}$. We feed a noisy image
(a) to the denoising network while changing the input to the encoder that
extracts $\mathbf{c}$. (b) Result of VDID with the same input to the denoiser
and encoder. (c) Result when the encoder is given the input with
$\sigma_{N}=10$ so that $\mathbf{c}$ is the latent variable correponding to
$\sigma_{N}=10$. (d) $\mathbf{c}$ correponds to $\sigma_{N}=50$. (e) The
encoder is given the flipped patch with $\sigma_{N}=30$. Also see the
visualized $\mathbf{c}$ in Fig. 4, and text (Section 5) for details.

(a) Encoder input with $\sigma_{N}=30$

(b) Flipped input, $\sigma_{N}=30$

(c) Input with $\sigma_{N}=10$

(d) Input with $\sigma_{N}=50$

(e) Flat input, $\sigma_{N}=10$

(f) Flat input, $\sigma_{N}=30$

(g) Flat input, $\sigma_{N}=50$

Figure 4: Visualization of $\mathbf{c}$ for the image in Figure 3 and for a
flat image with varying $\sigma_{N}$. (a) $\mathbf{c}$ from the encoder, when
its input is Fig. 3(a), (b) when the input is the flip of Fig. 3(a), (c) when
the input is the same but $\sigma_{N}=10$, and (d) $\sigma_{N}=50$. (e)
$\mathbf{c}$ from the encoder given a flat patch with $\sigma_{N}=10$, (f)
$\sigma_{N}=30$, and (g) $\sigma_{N}=50$. See text (Section 5) for details.

##  5 Analysis

To investigate the role of our latent variable $\mathbf{c}$, we conduct
several experiments exhibited in Figure 3 and Figure 4.

#### Latent Variable Manipulation

In our VDID, the denoiser and encoder should be given the same noisy image so
that the encoder extracts appropriate $\mathbf{c}$ that bears information
needed for the denoiser. That is, when the same image with $\sigma_{N}=30$
(Figure 3(a)) is fed to the denoiser and encoder, the VDID performs the best
as in Figure 3(b). If we feed the same input ($\sigma_{N}=30$) to the denoiser
while feeding input with $\sigma_{N}=10$ to the encoder, then the encoder will
extract $\mathbf{c}$ corresponding to $\sigma_{N}=10$, and the performance is
degraded as in Figure 3(c). Similarly, feeding the image with $\sigma_{N}=50$
also lowers the PSNR (Figure 3(d)). To see that the performance is also
affected by the change of image contents, we feed a flipped image of Figure
3(a) to the encoder so that the $\mathbf{c}$ does not match with denoiser
input. Although the noise level is the same, the PSNR is lowered as in Figure
3(e), showing that $\mathbf{c}$ delivers image information as well as noise
information. This is better illustrated in Figure 4 showing the visualized
$\mathbf{c}$, which is also explained in the next paragraph.

#### Latent Variable Visualization

To further investigate the latent space, we visualize the channel-wise average
of $\mathbf{c}$ by feeding various inputs, as in Figure 4. The first row shows
$\mathbf{c}$ extracted from the encoder, when the input image is Figure 3 with
$\sigma_{N}=30$, flipped image with $\sigma_{N}=30$, and the images with
$\sigma_{N}=10$ and $50$. The second row shows $\mathbf{c}$ for the flat image
with $\sigma_{N}=10$, $30$, and $50$ for the reference. By comparing the first
and second rows, it can be seen that $\mathbf{c}$ bears image contents.
Specifically, the flat image generates almost flat $\mathbf{c}$ with the
changing magnitudes according to $\sigma_{N}$, validating that $\mathbf{c}$
delivers noise information. When the input has both texture and flat areas
(the first row), then $\mathbf{c}$ shows the magnitudes changing with respect
to contents and also noise variance. Comparing Figure 4(a) and the second-row
images, it is interesting to see that texture area (tree) pretends to have
lower $\sigma_{N}$ (yellow as Figure 4(e)), and flat areas (building) pretends
to be higher as Figure 4(g). This means that the flat area is more strongly
filtered than the textured.

The visualizations also validate why we obtain the result of Figure 3(e), when
we feed a flipped image to the encoder. Precisely, the tree area is over-
smoothed and the building area is less filtered because $\mathbf{c}$ is
conversely delivered. Likewise, providing noise-only information (flat
$\mathbf{c}$) also lowers the performance as in Figs. 3(c) and (d), which also
imply that our learned $\mathbf{c}$ bears more information than the noise
variance, for the successful denoising.

#### Summary of Latent Variable Analysis

In summary, we note that using only noise variance information is not enough
for the successful denoising, which was a common approach in previous works
[48, 49, 21, 37]. Instead, the optimal method would be to use both noise and
content information. In our method, the encoder learns such information and
delivers it to the denoiser as the latent variable $\mathbf{c}$, which
eventually boosts the performance.

#### Ablation Study

As we propose to use additional loss terms, we provide ablation results on
loss terms in Table 4. With only denoising loss term $\mathcal{L}_{denoise}$,
which corresponds to naïve blind denoising, the performance is inferior to
other methods. Without the adversarial loss $\mathcal{L}_{adv}$ for
approximating the likelihood, it shows a slightly inferior result compared to
using total loss because using only the pixel-wise distance strictly assumes a
probability distribution family. Interestingly, during our research, we found
that without the adversarial loss, the encoder mainly encodes the content
information such as color, while the noise level information is rarely
embedded. By using all loss terms, the proposed method guarantees the
performance that surpasses others.

Table 4: Ablation study on CBSD68 with $\sigma_{N}=30$. Loss term | PSNR
---|---
$\mathcal{L}_{denoise}$ | 30.47
$\mathcal{L}_{denoise}+\beta D_{KL}+\mathcal{L}_{recon}$ w/o $\mathcal{L}_{adv}$ | 30.56
$\mathcal{L}_{denoise}+\beta D_{KL}+\mathcal{L}_{recon}$ w/ $\mathcal{L}_{adv}$ | 30.64

##  6 Conclusion

In this paper, we have presented a novel variational approach for image
denoising. Concretely, we reformulated the log-posterior of the denoising
problem and introduced a _variational lower bound_ to approximate the original
MAP objective. With our _variational lower bound_ , the original problem can
be divided into separate sub-problems, which eventually relaxes the given
problem. Moreover, our _variational lower bound_ incorporates both denoising
objective and noisy image generative models. Hence, we can further exploit the
embedded information of complicated noisy image manifold. We have also
presented three parameterized CNNs for the inference problem and shown that
our method achieves state-of-the-art performance in removing both Gaussian and
real-world noises while requiring fewer parameters. The code is publicly
available at https://github.com/JWSoh/VDIR.

## References

  * [1] Abdelrahman Abdelhamed, Marcus A Brubaker, and Michael S Brown.  Noise flow: Noise modeling with conditional normalizing flows.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3165–3173, 2019.
  * [2] Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown.  A high-quality denoising dataset for smartphone cameras.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1692–1700, 2018.
  * [3] Eirikur Agustsson and Radu Timofte.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 126–135, 2017.
  * [4] Saeed Anwar and Nick Barnes.  Real image denoising with feature attention.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3155–3164, 2019.
  * [5] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron.  Unprocessing images for learned raw denoising.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11036–11045, 2019.
  * [6] Antoni Buades, Bartomeu Coll, and J-M Morel.  A non-local algorithm for image denoising.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 2, pages 60–65. IEEE, 2005.
  * [7] Jingwen Chen, Jiawei Chen, Hongyang Chao, and Ming Yang.  Image blind denoising with generative adversarial network based noise modeling.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3155–3164, 2018.
  * [8] Yunjin Chen and Thomas Pock.  Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration.  IEEE transactions on pattern analysis and machine intelligence, 39(6):1256–1272, 2016.
  * [9] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Image denoising by sparse 3-d transform-domain collaborative filtering.  IEEE Transactions on image processing, 16(8):2080–2095, 2007.
  * [10] Michael Elad and Michal Aharon.  Image denoising via sparse and redundant representations over learned dictionaries.  IEEE Transactions on Image processing, 15(12):3736–3745, 2006.
  * [11] Alessandro Foi.  Clipped noisy images: Heteroskedastic modeling and practical denoising.  Signal Processing, 89(12):2609–2629, 2009.
  * [12] Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian.  Practical poissonian-gaussian noise modeling and fitting for single-image raw-data.  IEEE Transactions on Image Processing, 17(10):1737–1754, 2008.
  * [13] Abhiram Gnansambandam and Stanley H Chan.  One size fits all: Can we train one denoiser for all noise levels?  ICLR, 2020.
  * [14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [15] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng.  Weighted nuclear norm minimization with application to image denoising.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2862–2869, 2014.
  * [16] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Toward convolutional blind denoising of real photographs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1712–1722, 2019.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  ICLR, 2(5):6, 2017.
  * [19] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja.  Single image super-resolution from transformed self-exemplars.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5197–5206, 2015.
  * [20] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [21] Yoonsik Kim, Jae Woong Soh, and Nam Ik Cho.  Adaptively tuning a convolutional neural network by gate process for image denoising.  IEEE Access, 7:63447–63456, 2019.
  * [22] Yoonsik Kim, Jae Woong Soh, Gu Yong Park, and Nam Ik Cho.  Transfer learning from synthetic to real-noise denoising with adaptive instance normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3482–3492, 2020.
  * [23] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  ICLR, 2014.
  * [24] Stamatios Lefkimmiatis.  Universal denoising networks: a novel cnn architecture for image denoising.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3204–3213, 2018.
  * [25] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136–144, 2017.
  * [26] Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang.  Non-local recurrent network for image restoration.  In Advances in Neural Information Processing Systems, pages 1673–1682, 2018.
  * [27] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(Nov):2579–2605, 2008.
  * [28] Markku Makitalo and Alessandro Foi.  Optimal inversion of the generalized anscombe transformation for poisson-gaussian noise.  IEEE transactions on image processing, 22(1):91–103, 2012.
  * [29] Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang.  Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections.  In Advances in neural information processing systems, pages 2802–2810, 2016.
  * [30] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416–423. IEEE, 2001.
  * [31] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In ICLR, 2018.
  * [32] Seonghyeon Nam, Youngbae Hwang, Yasuyuki Matsushita, and Seon Joo Kim.  A holistic approach to cross-channel image noise modeling and its application to image denoising.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1683–1691, 2016.
  * [33] Tobias Plotz and Stefan Roth.  Benchmarking denoising algorithms with real photographs.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [34] Tobias Plötz and Stefan Roth.  Neural nearest neighbors networks.  In Advances in Neural Information Processing Systems, pages 1087–1098, 2018.
  * [35] Daniel Ritchie, Paul Horsfall, and Noah D Goodman.  Deep amortized inference for probabilistic programs.  arXiv preprint arXiv:1610.05735, 2016.
  * [36] Leonid I Rudin, Stanley Osher, and Emad Fatemi.  Nonlinear total variation based noise removal algorithms.  Physica D: nonlinear phenomena, 60(1-4):259–268, 1992.
  * [37] Jae Woong Soh and Nam Ik Cho.  Deep universal blind image denoising.  In International Conference on Pattern Recognition, 2020.
  * [38] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu.  Memnet: A persistent memory network for image restoration.  In Proceedings of the IEEE international conference on computer vision, pages 4539–4547, 2017.
  * [39] Radu Timofte, Rasmus Rothe, and Luc Van Gool.  Seven ways to improve example-based single image super resolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1865–1873, 2016.
  * [40] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Recovering realistic texture in image super-resolution by deep spatial feature transform.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 606–615, 2018.
  * [41] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [42] Kaixuan Wei, Ying Fu, Jiaolong Yang, and Hua Huang.  A physics-based noise formation model for extreme low-light raw denoising.  arXiv preprint arXiv:2003.12751, 2020.
  * [43] Jun Xu, Lei Zhang, and David Zhang.  A trilateral weighted sparse coding scheme for real-world image denoising.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 20–36, 2018.
  * [44] Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang.  Variational denoising network: Toward blind noise modeling and removal.  In Advances in Neural Information Processing Systems, pages 1688–1699, 2019.
  * [45] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  Cycleisp: Real image restoration via improved data synthesis.  arXiv preprint arXiv:2003.07761, 2020.
  * [46] Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt.  Advances in variational inference.  IEEE transactions on pattern analysis and machine intelligence, 41(8):2008–2026, 2018.
  * [47] Jiachao Zhang and Keigo Hirakawa.  Improved denoising via poisson mixture modeling of image sensor noise.  IEEE Transactions on Image Processing, 26(4):1565–1578, 2017.
  * [48] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  IEEE Transactions on Image Processing, 26(7):3142–3155, 2017.
  * [49] Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Ffdnet: Toward a fast and flexible solution for cnn-based image denoising.  IEEE Transactions on Image Processing, 27(9):4608–4622, 2018.
  * [50] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 286–301, 2018.

## Appendix

## A. Derivation and Proof of the Variational Lower Bound

Given a noisy image $\mathbf{y}$ and its underlying clean image $\mathbf{x}$,
the joint distribution can be reformulated as follows. We introduce a
tractable distribution $q(\mathbf{c}|\mathbf{y})$ and a latent variable
$\mathbf{c}$.

$\displaystyle\log p(\mathbf{x},\mathbf{y})=$
$\displaystyle\int_{c}q(\mathbf{c}|\mathbf{y})\log
p(\mathbf{x},\mathbf{y})d\mathbf{c}$ $\displaystyle=$
$\displaystyle\int_{c}q(\mathbf{c}|\mathbf{y})\log\frac{p(\mathbf{x}|\mathbf{y},\mathbf{c})p(\mathbf{y},\mathbf{c})}{p(\mathbf{c}|\mathbf{x},\mathbf{y})}d\mathbf{c}$
$\displaystyle=$
$\displaystyle\int_{c}q(\mathbf{c}|\mathbf{y})\log\frac{p(\mathbf{x}|\mathbf{y},\mathbf{c})p(\mathbf{y}|\mathbf{c})p(\mathbf{c})}{p(\mathbf{c}|\mathbf{x},\mathbf{y})}d\mathbf{c}$
$\displaystyle=$ $\displaystyle\int_{c}q(\mathbf{c}|\mathbf{y})[\log
p(\mathbf{x}|\mathbf{y},\mathbf{c})+\log\frac{q(\mathbf{c}|\mathbf{y})}{p(\mathbf{c}|\mathbf{x},\mathbf{y})}+\log\frac{p(\mathbf{c})}{q(\mathbf{c}|\mathbf{y})}+\log
p(\mathbf{y}|\mathbf{c})]d\mathbf{c}$ $\displaystyle=$
$\displaystyle\mathbb{E}_{\mathbf{c}\sim q(\mathbf{c}|\mathbf{y})}[\log
p(\mathbf{x}|\mathbf{y},\mathbf{c})]+D_{KL}(q(\mathbf{c}|\mathbf{y})||p(\mathbf{c}|\mathbf{x},\mathbf{y}))$
$\displaystyle\qquad\qquad-
D_{KL}(q(\mathbf{c}|\mathbf{y})||p(\mathbf{c}))+\mathbb{E}_{\mathbf{c}\sim
q(\mathbf{c}|\mathbf{y})}[\log p(\mathbf{y}|\mathbf{c})].$

The _variational lower bound_ is defined as

$\mathcal{L}=\mathbb{E}_{\mathbf{c}\sim q(\mathbf{c}|\mathbf{y})}[\log
p(\mathbf{x}|\mathbf{y},\mathbf{c})]-D_{KL}(q(\mathbf{c}|\mathbf{y})||p(\mathbf{c}))+\mathbb{E}_{\mathbf{c}\sim
q(\mathbf{c}|\mathbf{y})}[\log p(\mathbf{y}|\mathbf{c})],$ (19)

then,

$\log
p(\mathbf{x},\mathbf{y})=\mathcal{L}+D_{KL}(q(\mathbf{c}|\mathbf{y})||p(\mathbf{c}|\mathbf{x},\mathbf{y})).$
(20)

Since the KL divergence is non-negative, the following inequality holds.

$\log p(\mathbf{x},\mathbf{y})\geq\mathcal{L}.$ (21)

## B. The Auto-Encoder & Discriminator Architecture

Tables 5 and 6 show the network architectures of the encoder and decoder,
respectively. The number of parameters of the encoder is $154$ K and the
decoder is $114$ K. Also, Table 7 describes the discriminator architecture. We
adopt patch discriminator [20], and spectral normalization [31] is used for
all convolution layers in the discriminator. The notations are as follows.

  * •

$H,W,C$: Height, width, and the number of channels of the input image.

  * •

Conv2d(K, S): 2d convolution with kernel size K and stride S.

  * •

MaxPool(K, S): Max pooling operation with kernel size K and strides S.

  * •

NN Upsampling: Nearest neighbor upsampling.

Table 5: The network architecture of the encoder. Module | Layers | Input size | Output size
---|---|---|---
_Block 1_ | Conv2d(3, 1) | $H\times W\times C$ | $H\times W\times 64$
| MaxPool(2, 2), ReLU | $H\times W\times 64$ | $\frac{H}{2}\times\frac{W}{2}\times 64$
_Block 2_ | Conv2d(3, 1), ReLU | $\frac{H}{2}\times\frac{W}{2}\times 64$ | $\frac{H}{2}\times\frac{W}{2}\times 64$
| Conv2d(3, 1) | $\frac{H}{2}\times\frac{W}{2}\times 64$ | $\frac{H}{2}\times\frac{W}{2}\times 64$
| MaxPool(2, 2), ReLU | $\frac{H}{2}\times\frac{W}{2}\times 64$ | $\frac{H}{4}\times\frac{W}{4}\times 64$
_Block 3_ | Conv2d(3, 1), ReLU | $\frac{H}{4}\times\frac{W}{4}\times 64$ | $\frac{H}{4}\times\frac{W}{4}\times 64$
(a) | Conv2d(3, 1) | $\frac{H}{4}\times\frac{W}{4}\times 64$ | $\frac{H}{4}\times\frac{W}{4}\times 64$
_$\mu$_ , input (a) | Conv2d(3, 1) | $\frac{H}{4}\times\frac{W}{4}\times 64$ | $\frac{H}{4}\times\frac{W}{4}\times 4$
_$\log\sigma^{2}$_ , input (a) | Conv2d(3, 1) | $\frac{H}{4}\times\frac{W}{4}\times 64$ | $\frac{H}{4}\times\frac{W}{4}\times 4$
|  |  |
Table 6: The network architecture of the decoder. Module | Layers | Input size | Output size
---|---|---|---
_Block 1_ | Conv2d(3, 1) | $\frac{H}{4}\times\frac{W}{4}\times 4$ | $\frac{H}{4}\times\frac{W}{4}\times 64$
| NN Upsampling ($\times 2$), ReLU | $\frac{H}{4}\times\frac{W}{4}\times 64$ | $\frac{H}{2}\times\frac{W}{2}\times 64$
_Block 2_ | Conv2d(3, 1), ReLU | $\frac{H}{2}\times\frac{W}{2}\times 64$ | $\frac{H}{2}\times\frac{W}{2}\times 64$
| Conv2d(3, 1) | $\frac{H}{2}\times\frac{W}{2}\times 4$ | $\frac{H}{2}\times\frac{W}{2}\times 64$
| NN Upsampling ($\times 2$), ReLU | $\frac{H}{2}\times\frac{W}{2}\times 64$ | $H\times W\times 64$
_Block 3_ | Conv2d(3, 1), ReLU | $H\times W\times 64$ | $H\times W\times 64$
_Output_ | Conv2d(3, 1) | $H\times W\times 64$ | $H\times W\times C$
|  |  |
Table 7: The network architecture of the discriminator. Module | Layers | Input size | Output size
---|---|---|---
_Conv 1_1_ | Conv2d(3, 1), leakyReLU | $H\times W\times C$ | $H\times W\times 64$
_Conv 1_2_ | Conv2d(3, 2), leakyReLU | $H\times W\times 64$ | $\frac{H}{2}\times\frac{W}{2}\times 64$
_Conv 2_1_ | Conv2d(3, 1), leakyReLU | $\frac{H}{2}\times\frac{W}{2}\times 64$ | $\frac{H}{2}\times\frac{W}{2}\times 128$
_Conv 2_2_ | Conv2d(3, 2), leakyReLU | $\frac{H}{2}\times\frac{W}{2}\times 128$ | $\frac{H}{4}\times\frac{W}{4}\times 128$
_Conv 3_1_ | Conv2d(3, 1), leakyReLU | $\frac{H}{4}\times\frac{W}{4}\times 128$ | $\frac{H}{4}\times\frac{W}{4}\times 256$
_Conv 3_2_ | Conv2d(3, 2), leakyReLU | $\frac{H}{4}\times\frac{W}{4}\times 256$ | $\frac{H}{8}\times\frac{W}{8}\times 256$
_Conv 4_1_ | Conv2d(3, 1), leakyReLU | $\frac{H}{8}\times\frac{W}{8}\times 256$ | $\frac{H}{8}\times\frac{W}{8}\times 512$
_Conv 4_2_ | Conv2d(3, 2), leakyReLU | $\frac{H}{8}\times\frac{W}{8}\times 512$ | $\frac{H}{16}\times\frac{W}{16}\times 512$
_Conv 5_1_ | Conv2d(3, 1), leakyReLU | $\frac{H}{16}\times\frac{W}{16}\times 512$ | $\frac{H}{16}\times\frac{W}{16}\times 512$
_Conv 5_2_ | Conv2d(3, 2), leakyReLU | $\frac{H}{16}\times\frac{W}{16}\times 512$ | $\frac{H}{32}\times\frac{W}{32}\times 512$
_logits_ | Conv2d(3, 1) | $\frac{H}{32}\times\frac{W}{32}\times 512$ | $\frac{H}{32}\times\frac{W}{32}\times 1$
|  |  |

## C. More Visualized Results

We present additional visualized results in Figures 5, 6, 7, and 8.

Noisy

CBDNet [16] / 4.4 M

RIDNet [4] / 1.5 M

VDN [44] / 7.8 M

VDID (Ours) / 2.2 M

Figure 5: Visualized examples from SIDD validation set [2] with the number of
parameters.

Noisy

CBDNet [16] / 4.4 M

RIDNet [4] / 1.5 M

VDN [44] / 7.8 M

VDID (Ours) / 2.2 M

Figure 6: Visualized examples from SIDD validation set [2] with the number of
parameters.

Noisy
26.90/0.7527

DnCNN+ [48]
33.29/0.9271

FFDNet+ [49]
33.05/0.9231

CBDNet [16]
33.62/0.9295

RIDNet [4]
34.09/0.9382

VDN [44]
33.89/0.9376

AINDNet(S) [22]
34.28/0.9414

VDID+ (Ours)
34.63/0.9438

Figure 7: Visualized examples from DND benchmark [33] with PSNR/SSIM.

GT
-

Noisy
15.11 dB

CBM3D [9]
25.50 dB

DnCNN [48]
26.19 dB

FFDNet [49]
26.28 dB

RED [29]
24.29 dB

UNLNet [24]
26.26 dB

VDN [44]
26.78 dB

VDID (Ours)
26.89 dB

Figure 8: Visualized examples on Gaussian noise with $\sigma_{N}=50$ with
PSNR.

## D. More Analysis on Latent Space

#### t-SNE Visualization

For further analysis, we demonstrate t-SNE [27] visualization of $\mathbf{c}$
in Figure 9 to investigate the latent space. In particular, $\mathbf{c}$ is
average-pooled to generate global abstract of a patch as
$AvgPool(\mathbf{c})\in\mathbb{R}^{4}$. We first present results on Gaussian
noise with DIV2K validation set [3], where $1,000$ patches are randomly
sampled from DIV2K validation set, and Gaussian noises are added with noise
levels $10$ to $55$ at $5$ intervals, in Figure 9(a), 9(b).

As shown, the latent space well represents the noisy image manifold based on
their contents and noise distribution. Interestingly, the latent embedding
represents the content information and varies continuously as shown in Figure
9(a), in that similar color and luminance brought similar $\mathbf{c}$. The
latent space also contains noise information as shown in Figure 9(b).
Specifically, the patches with similar noise levels are closely located and
clustered. In conclusion, the latent space, which is suitable for denoising
tasks, contains not only noise level information but also global content
information.

Moreover, we also present t-SNE visualization of the real-noise denoising in
Figure 9(c), 9(d). We extracted patches from three datasets: SIDD [2]
validation set, DND [33], and DIV2K validation set [3] with synthetic noise
following [16].

As shown, the latent embedding is highly correlated to the content information
such as colors and intensities. It might be connected to the common knowledge
that the noise from the real-world is signal-dependent. Interestingly, based
on Figure 9(d), the latent code captures different characteristics between the
noise from SIDD[2] and DND[33], despite we did not inject any supervision
about the dataset. Concretely, they are separately clustered, and we might
infer that there exists a domain gap between them. In other words, our latent
variable sees the difference in noise distribution of smartphone cameras and
commercial cameras. Rather, the synthetic noise based on [16] may better mimic
the characteristics of the noise from commercial cameras compared to SIDD [2]
based on our observation.

(a) Patches (AWGN)

(b) Noise levels (AWGN)

(c) Patches (Real)

(d) Datasets (Real)

Figure 9: t-SNE visualizations on Gaussian (a-b) and real noise (c-d). (a)
t-SNE visualization with patches. (b) Corresponding noise levels presented
with different colors. (c) t-SNE visualization with patches. (d) Corresponding
datasets.