# Denoising single images by feature ensemble revisited

Masud An Nur Islam Fahim
Chosun University
Gwangju, South Korea
<EMAIL_ADDRESS>
Nazmus Saqib
Chosun University
Gwangju, South Korea
<EMAIL_ADDRESS>
Shafkat Khan Siam
Chosun University
Gwangju, South Korea
<EMAIL_ADDRESS>
Ho Yub Jung
Chosun University
Gwangju, South Korea
<EMAIL_ADDRESS>

###### Abstract

Image denoising is still a challenging issue in many computer vision sub-
domains. Recent studies show that significant improvements are made possible
in a supervised setting. However, few challenges, such as spatial fidelity and
cartoon-like smoothing remain unresolved or decisively overlooked. Our study
proposes a simple yet efficient architecture for the denoising problem that
addresses the aforementioned issues. The proposed architecture revisits the
concept of modular concatenation instead of long and deeper cascaded
connections, to recover a cleaner approximation of the given image. We find
that different modules can capture versatile representations, and concatenated
representation creates a richer subspace for low-level image restoration. The
proposed architecture’s number of parameters remains smaller than the number
for most of the previous networks and still achieves significant improvements
over the current state-of-the-art networks.

##  1 Introduction

Image denoising is a classic problem in the low-level vision domain. For a
given image $\mathcal{X}$, it goes through the following mapping to create its
noisy counterpart.

$\mathcal{Y}=\mathcal{X}+\mathcal{N}$ (1)

Here, $\mathcal{Y}$ is the noisy observation, where $\mathcal{N}$ is the
additive noise on a clean image $\mathcal{X}$. Denoising is an ill-posed
problem, with no direct means to separate the source image and corresponding
noise. Hence, researchers follow the best possible approximation of
$\mathcal{X}$ from $\mathcal{Y}$ with corresponding algorithmic strategies.

Typical methods without machine learning involve employing efficient filtering
techniques such as NLM[1], BM3D[2], median[3], Weiner[4], etc. Due to their
limited generalization capability, additional knowledge-based priors or matrix
properties were integrated into the denoising strategies. However, despite
certain improvements with prior-based methods, many concerns remain
unresolved, such as holistic fidelity or the choice of priors.

(a) Noisy

(b) Ground Truth

(c) DEAMNet[5]

(d) Proposed

Figure 1:  Demonstration of our contribution. An image from the BSD68 dataset
with AWGN noise $\sigma$ = 50. Here, the first column shows the ground truth,
followed by the inference from the DEAMNet[5] and the final column shows the
proposed result. From a side-by-side comparison, our reconstruction restores a
closer approximation of the ground truth than the referred study with finer
details within both foreground and background patches.

Convolution neural network (CNN) denoising methods later offer an
unprecedented improvement over the previous strategies through their
customized learning setup. Usually, CNN methods offer better performance
through brute force learning [6], tricky training strategy [7], or inverting
image properties [8] by various proposals. We observed gradual improvements
over the years for denoising solutions. However, these methods with pure brute
force mapping sometimes face fidelity issues within challenging noisy images.
Furthermore, due to the lack of generalization properties, the methods provide
reconstructed images that often result in cartoonized smoothing.

In contrast, the proposed approach rebuilds a previous ensemble-oriented
denoising network that can successfully estimate cleaner image with less
cartoon-like smoothing. For the design of the proposed denoising network, we
carefully maximized detail restoration by providing a variety of low-level
ensemble features while keeping the network relatively shallow to prevent an
oversized receptive field. In summary, our study has the following
contributions:

  * •

We propose a shallow ensemble approach through feature concatenation to create
a large array of feature combinations for low-level image recovery.

  * •

Due to the ensemble of multiple modules, our model successfully returns fine
details compared to previous data-driven studies.

  * •

The parameter space is relatively small compared to the contemporary methods
with computationally fast inference time.

  * •

Finally, the proposed study shows better performance with a different range of
synthetic noise and real noise without the cartoonization effect. See fig. 1.

##  2 Related Work

###  2.1 Filtering based schemes

Traditional filtering approaches aim for a handcrafted filters for noise and
image separation. These studies [4, 3] utilize low-pass filtering methods to
extract the clean images from the noisy images. The iterative filtering
approach adopts progressive reduction for image restoration[9]. Additionally,
several methods use nonlocal similar patches for noise reduction based on the
similarity between the counterpart patches in the same image. For example,
NLM[1] and BM3D[2] assume redundancy within patches from a given image for
noise reduction. Nonetheless, these methods usually produce flat
approximations, as the given image severely degrades the noisy image quality
with a heavy noise presence.

###  2.2 Prior based schemes

Another group of studies focuses on selecting priors for the model that
produce clean images when optimized. These methods reformulate the denoising
problem as a maximum a posteriori (MAP)-based optimization problem, where the
image prior regulates the performance of the objective function. For example,
the studies [10, 11] assume sparsity as the prior for their optimization
process. The primary intuition is to represent each patch separately through
the help of a function. Xu et al. [12] performed real-world image denoising by
proposing a trilateral weighted sparse coding scheme. Other studies [12, 13,
14, 15] focus on rank properties to minimize their objective function.
Weighted nuclear norm minimization (WNNM) [12] calculates the nuclear norm
through a low-rank matrix approximation for image denoising. Additionally,
there are several complex model-based derivations using graph-based
regularizers for noise reduction. However, their performance degrades
monotononically for noisier areas, and recovering the detailed information is
sometime difficult [16, 17]. Additionally, these methods generally output
significantly varying results depending on their prior parameters and the
respective target noise levels.

###  2.3 Learning based schemes

Due to the availability of paired data and the current success of CNN modules,
data-driven schemes have achieved significant improvement in separating clean
images from noisy images. Recent CNN studies, such as DnCNN[18] and IrCNN[19],
utilized the residual connection for the estimating noise removal map before
inference. Both of them evaluated the clean image without taking any priors
regarding the structure or noise. They achieved enhanced performance by using
a noncomplex architecture with repeated convolutional, batch normalization,
and ReLU activation function blocks. However, these methods can fail to
recover some of the detailed texture structure in the presence of heavy noise
area.

Trainable Nonlinear Reaction-Diffusion (TRND) [20] uses prior in their neural
network and extends the non-linear diffusion algorithm for noise reduction.
However, the methods suffer from computational complexity due to requiring a
vast number of parameters. Similarly, the nonlocal color net [21] utilizes the
nonlocal similarity priors for the image denoising operation. Although priors
mostly aid in denoising, there are some cases where the adaptation of the
priors degrades the denoising performance. Very recently, DEAMNet[5] surpasses
the previous state-of-the-art results by using adaptive consistency prior.

With the success of the DnCNN[18], two similar networks called “Formatting
net" and “DiffResNet" are proposed with different loss layers [6]. Later, Bae
et al. [22] proposed a residual learning strategy based on the improved
performance of a learning algorithm using manifold simplification, providing
significantly better performance. After that, Anwar et al.[23] proposed a
cascaded CNN architecture with feature attention using self-ensemble to boost
the performance.

Few recent approaches [24, 25] follow the blind denoising strategy. CBDNet
[25] proposed a blind denoising network consisting of two submodules: noise
estimation and noise removal by incorporating multiple losses. However, their
performance may be limited by manual intervention requirements, and a slightly
lower performance on real-world noisy images. In comparison, FFDNet [26]
achieves enhanced results by proposing the nonblind Gaussian denoising
network. Consequently, RIDNet [6] utilizes perceptual loss with $\ell_{2}$
apart from the DnCNN architectures for noise removal and achieves significant
success by introducing single-stage attention denoising architecture from real
and synthetic noises. Recently, Liu et al. [8] introduced GradNet by
revisiting the image gradient theory of neural networks.

##  3 Methodology

###  3.1 Baseline supervised architecture

Recently, supervised model-based denoising methods embedded a similar baseline
formation in their proposals [6, 27]. In brief, it is possible to
compartmentalize the baseline architecture in fig. 2 into three distinct
modules: initial feature learner, large intermediate processor, and final
reconstruction part. Typically, the primary module consists of a single layer
that serves the purpose of the initial feature learner or initial noise
estimator. For any noisy image $I_{n}$, the representation of the initial
block $E_{p}$ is as follows:

$E_{p}=\beta(I_{n})$ (2)

where $\beta$ is the initial convolutional layer for basic feature training.
Followed that, we see the main restoration part of the given network through
the help of an intermediate processor. Typically, this intermediate layer is a
very long cascaded connection of the unique feature extractor units. From time
to time, we often observe the presence of long residual-dense or residual-
attention blocks as the backbone of such setup.

Figure 2:  This figure shows the general baseline architecture for the
denoising model, which usually consists of the more prolonged feature
extraction phase with cascaded modules, which begin right after the initial
feature collection and end with the final residual aggregation.

Now, if the intermediate block is $\mathcal{M}$; the cascaded representation
of the intermediate processing stage is as follows:

$E_{i}=\mathcal{M}_{j}(\mathcal{M}_{i-1}(.....(E_{p})..))$ (3)

where $\mathcal{M}_{i}$ represents the $i^{th}$ instance of the learning stage
of the intermediate block and $E_{i}$ is the corresponding outcome of the
intermediate layer.

The final reconstruction module operates through a residual connection
followed by a consecutive final convolution. If $\mathcal{R}$ is the final
reconstruction stage before the output, then the recovered image $I_{r}$ is a
combination of $E_{i}$, $E_{p}$, and $I_{n}$.

$I_{r}=\mathcal{R}(E_{i},E_{p},I_{n})=f_{N}(I_{n})$ (4)

Here, $f_{N}$ denotes the overall neural network, $I_{n}$ is the noisy input
image, and $I_{r}$ is the recovered image. A typical choice of the cost
function for this task involves $\ell_{1}$ or $\ell_{2}$ loss. There are other
customized loss functions available, such as weighted-augmentation of
different loss functions that integrate spatial properties or relevant
regularization [6]. In general, the network is optimized by minimizing the
difference from clean images.

$\zeta(\theta)=\frac{1}{N}\sum_{i=1}^{N}||f_{N}(\theta,{I_{n}^{i}})-I_{c}^{i}||_{1}$
(5)

Here, $\theta$ is the learnable parameter, $I_{n}^{i}$ is the noisy image, and
$I_{c}^{i}$ is the corresponding clean image. Most of the baseline network
parameters are placed in the intermediate learning block.

###  3.2 Proposed architecture

In contrast, we design our network to allocate more resources to the
concatenated learned features. Instead of developing a basic learning block
for long cascading connections, we go for depth by proposing various
individual feature learning blocks. The proposed network is focused on
delivering richer and diverse low level features. To further reduce
complexity, we avoid using attention operation, which is typically more
expensive. More details are provided in fig. 3 and the following subsection.

####  3.2.1 Initial feature block

Three consecutive convolution layers are used to extract initial features for
the network. The layers are equal in-depth, but their kernel sizes are in
descending order. The input image goes through the $5\times 5$ convolution
operation at the first layer, followed by $3\times 3$ convolution, and ends at
$1\times 1$ pixelwise operation. A larger kernel size makes use of a larger
neighborhood of input features and estimates the representations on a larger
receptive fields. By limiting the kernel size and the number of layers, the
network learns to focus on the smaller receptive fields and disregards broader
view, which we argue to be less meaningful in low-level vision tasks such as
denoising. Therefore, the purpose of the primary layer is to project the
representation for the denoising features from a smaller receptive field into
individual responses which can be further diversified in the next four block
modules in fig. 3.

###  3.3 Four modules for feature refinement

Before presenting the four modules for feature refinement, we will cover the
convolution, activation functions, and residual connections used in the
modules. Even though the attention mechanism is a common choice to learn
richer representations, we can still find a similar or better result without
it in this study. Additionally, our selection of residual blocks are no longer
than six consecutive connections. The fundamental operations for our modules
are introduced below.

Figure 3:  In the above figure, we present the overall diagram of the proposed
architecture for image denoising. Our pipeline first extracts the initial
feature using consecutive convolution operation, followed by the four modules
for feature refinements. These modules are standing upon the customized
convolution and residual setup with supportive activation functions. After
refinement, we concatenate all the refined feature maps into a single layer,
followed by a final dilated convolution to make the inference.

Convolution. In the internal convolution operation, our choice of kernels
varies from $1\times 1$ to $7\times 7$. Due to such a range, our network is
naturally focused on both smaller and larger receptive fields.

Activation functions. Recent advancements in nonlinear activation functions
have shown that better performance is achievable through the interconnected
operation of different activation representations that are compacted into a
single function. Hence, we choose the SWISH[28] and MISH[29] activations in
addition to the ReLU operations. As a result, our network learns from diverse
representations obtained from various parallel activated functions.

Residual connections. It will become redundant to mention the efficacy of the
residual connections in the vision tasks. In the literature, we can see that
the customization of residual connections varies within the task. In the
original ResNet paper [30], the authors included batch-normalization between
the convolution layer, followed by the ReLU layer. In our study, we use the
convolution layers, which are separated by the ReLU layer. This choice of the
ReLU sandwich residual connection is prevalent in regression tasks [31].

We will focus on the major processing modules below with the description of
the utilized blocks. We propose four processing modules that perform the
refinement operations on the initial features. The following subsections
covers descriptions and the basic reasoning behind the proposed architecture.

####  3.3.1 Residual Feature Aggregation Module

In our residual feature aggregation module, we use the aforementioned residual
blocks as our underlying design mechanism. In the construction of this module,
we took inspiration from the traditional pyramid feature extraction[32] and
aggregation, which has been very influential in computer vision. A typical
pyramid setup is motivated by the needs for multiscale feature aggregation,
which, in essence, utilizes low-frequency information along with high
frequency features. However, subsequent downsampling process is a lossy
operation by nature. To mitigate information loss for low-frequency features,
we choose to employ the concurrent residual blocks on the same initial
features through three different kernel sizes. Naturally, our kernel choice
ranges from $1\times 1$ to $5\times 5$. See fig. 4. Hence, a larger kernel
allows us to learn the features from larger area of image, while the $1\times
1$ kernel operation allows us to maintain the initial receptive field and make
use of more high-frequency information. We aggregate the response from all
three residual block to learn the overall multiscale impact of the initial
features. Finally, a typical $3\times 3$ convolution with standard depth gives
us the $n$ number of diverse representations from this module. As a result,
our model can learn the important multiscale features without going through a
pooling operation.

Figure 4: Residual feature Aggregation Module

####  3.3.2 Multi-Activation Feature Ensemble

Activation functions are unavoidable components for neural network
construction that aid the learning operation by projecting the impactful
information to the next layer. Hence, widely different nonlinear functions are
available as activation functions in all sorts of neural networks for various
utilities. ReLU is the most widely used activation, which at heart is a
‘positive pass’ filter. However, in some cases, zero-out negatives and a
discontinuity in the gradient are argued to be unhelpful in the optimization
process. To address some of its weaknesses, SWISH[28] and MISH[29] were
proposed with smooth gradients while maintaining a similar positive pass shape
of ReLU. A recent experiment [29] shows that these activation functions
provide a smoother loss landscape than the ReLU.

Figure 5: Multi-Activation Feature Ensemble Module

Nonetheless, we incorporate all three activation functions. See fig. 5. SWISH,
MISH, and ReLU activation are applied to the initial features, followed by a
convolution layer. The subsequent responses are concatenated into a single
tensor to learn from the integrated representation of varying activation
functions. No further kernels and residual blocks are utilized for this
module. The initial feature results of these modules are ensembled with the
responses of the other three modules, but the multi-activation are also
integrated into the multi-activated cascaded aggregation module described in
the next subsection.

####  3.3.3 Multi-activated Cascaded Aggregation

In this module, both shallow and relatively deeper layer features are
concatenated. Typically, a deep consecutive convolution operation is
formulated after the initial feature extraction, and the conventional thinking
is to build a deeper network for complex problems. However, we add a single
convolution layer feature to complement the deeper layer features because we
believe that shallower interpretation might be more appropriate for low-level
vision problems. See fig. 6.

Figure 6: Residual feature Aggregation Module

For a single convolution path, a $3\times 3$ kernel size is chosen with the
same depth as the initial features. For the deeper path, five consecutive
convolution layers with different kernel sizes were used. The activation
functions between the layers are ReLUs, however, for both paths, multi-
activation feature ensemble are implemented as described earlier. Both the
shallow and deeper responses are concatenated followed by another convolution
layer.

####  3.3.4 Densely Residual Feature Extraction

The densely residual operation has shown great promise in both regression and
classification tasks [33]. Dense residual connections are an efficient way to
emphasize hierarchical representation. For this reason, we design a densely
residual module to aggregate features for the network. The proposed design in
fig. 7 also utilizes the concatenation between the final and previous
aggregation in support of total hierarchy concentration. A final convolution
is added to combine the three concatenated features from the densely residual
layers.

Figure 7: Densely residual feature extraction module

After collecting and concatenating the individual responses from each of the
four modules, the responses are merged by the final convolution layer with a
dilation rate of $2$, see the overall process in fig. 3. This layer’s output
contains the most refined representation for the restored image. The restored
image is fed into a simple loss function consisting of $\ell_{1}$ and
$\ell_{SSIM}$.

###  3.4 Loss Function

We use two typical loss functions $\ell_{1}$ and $\ell_{SSIM}$ to update the
parameter space. The total loss function is a simple addition of the two.

$\ell_{total}=\ell_{1}+\ell_{SSIM}.$ (6)

$\ell_{1}$ measures the distance between the ground truth clean image and the
restored image as shown in next equation.

$\ell_{1}=\frac{1}{n}\sum_{j=1}^{n}|\gamma_{g}-\gamma_{p}|.$ (7)

Here, $\gamma_{g}$ is the ground truth clean image and $\gamma_{p}$ is the
restored prediction image. The secondary component is the loss function from
SSIM, which is another widely used similarity measure for images.

$\ell_{SSIM}=\frac{1}{n}\sum_{j=1}^{n}1-SSIM(\gamma_{g},\gamma_{p})$ (8)

##  4 Experimental Results

This section describes the overall performance of our method on both real and
synthetic noisy images.

(a) Fireman

(b) Noisy

(c) GT

(d) BM3D[2]

(e) WNNM[12]

(f) DnCNN[18]

(g) FFDNet[26]

(h) IrCNN[19]

(i) VDNet[34]

(j) ADNet[35]

(k) RIDNet[6]

(l) DeamNet[5]

(m) Proposed

Figure 8: Visual quality comparison for “Fireman" from the BSD68 dataset with
AWGN noise level $\sigma=50$ (for best view, zoom-in is recommended).

(a) Model

(b) Noisy

(c) GT

(d) BM3D[2]

(e) WNNM[12]

(f) DnCNN[18]

(g) FFDNet[26]

(h) IrCNN[19]

(i) VDNet[34]

(j) ADNet[35]

(k) RIDNet[6]

(l) DeamNet[5]

(m) Proposed

Figure 9: Visual quality comparison for “Model in black dress" from the
Kodak24 dataset with AWGN noise level $\sigma=50$ (for best view, zoom-in is
recommended)..

###  4.1 Network Implementation and Training Set

For the proposed study, we utilized a TensorFlow framework with NVIDIA GPU
support. Most of the convolutional layers in our network are $3\times 3$
kernels, apart from the specific cases where $1\times 1$, $5\times 5$, and
$7\times 7$ kernels in addition to the $3\times 3$ kernels were used. For the
training phase, we used He et. al [36] for initialization and the Adam
optimizer with the learning rate $10^{-4}$, a typical default in many vision
studies.

For the training, the DIV2K dataset was used. To enable diversity in the data
flow, the typical rotation, blurring, contrast stretching, and inverse
augmentation techniques were implemented. The training images were cropped
into smaller patches. The noisy input images were created by perturbing the
clean patches by AWGN noise with $15$, $25$, and $50$ standard deviations.

###  4.2 Testing set

We use the BSD68, Kodak24, and Urban100 datasets for the inference comparison,
where clean observations are available and noisy versions are created through
the same artificial noise augmentation. The results are summarized in table 1.

The DND, SIDD, and RN15 datasets are used to evaluate the proposed approach on
images with natural noise. A brief description of the real-world noisy image
dataset and the evaluation procedures are described below.

  * •

DND: DND[37] is a real-world image dataset consisting of $50$ real-world noisy
images. However, near noise-free counterparts are unavailable to the public.
The corresponding server provides the PSNR/SSIM results for the uploaded
denoised images.

  * •

SIDD: SIDD[38] is another real-world noisy image dataset that provides $320$
pairs of noisy images and near noise-free counterparts for training. This
dataset follows similar evaluation process as DND dataset.

  * •

RN15: RN15[24] dataset provides $15$ real-world noisy images. Due to the
unavailability of the ground truths, we only present the visual result of this
dataset.

(a) Interior

(b) Noisy

(c) GT

(d) BM3D[2]

(e) WNNM[12]

(f) DnCNN[18]

(g) FFDNet[26]

(h) IrCNN[19]

(i) VDNet[34]

(j) ADNet[35]

(k) RIDNet[6]

(l) DeamNet[5]

(m) Proposed

Figure 10: Visual quality comparison for “Interior" from the Urban100 dataset
with AWGN noise level $\sigma=50$ (for best view, zoom-in is recommended)..

###  4.3 Denoising on synthetic noisy images

For evaluation purposes, we have considered previous state-of-the-art studies
within various contexts. The evaluation procedure includes two filtering
methods BM3D[2], WNNM[12] and several convolutional networks including
DnCNN[18], FFDNet[26], IrCNN[19], ADNet[35], RIDNet[6], VDN[34], and
DEAMNet[5].

Table 1 shows the average PSNR/SSIM scores for the quantitative comparison.
From the average PSNR and SSIM score, the proposed study surpasses the
previous studies with a considerable margin. We have adopted three widely used
datasets BSD68, Kodak24, and Urban100 with three different AWGN noise levels
$15$, $50$, and $50$.

For visual comparison, 8, 9, and 10 from BSD68, Kodak24, and Urban100 are
presented respectively with the noise level of $50$. Figure 8 shows the
“fireman" picture from the BSD68 dataset. The differences in the restoration
are shown in detail with more controlled smoothing. From figure 9, we see that
the proposed approach avoids image cartoonization and preserves details while
restoring clean details. The proposed study manages to restore the structural
continuity compared to other methods while preserving appropriate color and
contrast of the image. The last visual comparison for the synthetic noisy
image is the “Interior" picture from the Urban100 dataset, shown in fig. 10.
For better illustration of differences, a zoomed image of the interior wall of
the place is shown, where the proposed method manages to preserve the brick
separating lines more clearly.

###  4.4 Denoising on real-world noisy images

The results for real-world noisy image restoration are presented in Table 2.
Natural noise removal is challenging because the convoluted noises are not
signal independent and vary within the spatial neighborhood.

(a) Noisy

(b) VDN[34]

(c) BM3D[2]

(d) DEAMNet[5]

(e) Proposed

Figure 12: Visual quality comparison for the SIDD dataset with real noises
(for best view, zoom-in is recommended).

(a) Star

(b) Noisy

(c) VDN[34]

(d) RIDNet[6]

(e) DeamNet[5]

(f) Proposed

Figure 13: Visual quality comparison for “Star” from the DnD dataset with real
noises (for best view, zoom-in is recommended)

We have chosen three real noisy image datasets, SIDD benchmark [38], DnD
benchmark[37], and RN15 [24], to analyze the generalization capability of our
proposed method. For SIDD and DnD benchmarks, the clean counterpart images are
not openly distributed. Hence, the presented PSNR/SSIM in table 2 is obtained
by uploading the results into the corresponding server. For the RN15 dataset,
there are no benchmark utility. Table 2 represents comparative performance for
both SIDD and DnD benchmark. Among the existing methods, VDN[34] and
DEAMNet[5] performs well. However, our method achieves better result among the
existing methods for both the real and synthetic noises.

(a) Noisy

(b) VDN[34]

(c) RIDNet[6]

(d) DeamNet[5]

(e) Proposed

Figure 14: Visual quality comparison for “Dog" from the RN15 dataset. RN15
dataset is set of real noise images without the clean image counterparts (for
best view, zoom-in is recommended).

To demonstrate our performance with real images, we have also provided some
visual comparisons in figures 12, 13, and 14 on the SIDD, DnD, and RN15
datasets, respectively. For visual comparison on real noisy images, we have
included the recent VDN[34], RIDNet[6], and DEAMNet[5]. The visual comparison
shows that our method tends to avoid cartoonization while effectively removing
noise, suppressing artifacts, and preserving object edges. Overall, the
qualitative and quantitative comparison displays effective performance in all
fronts.

Table 1: Quantitative comparison results of the competing methods with AWGN
noise level $\sigma=$15, 25, 50 on kodak24, BSD68, and Urban100. Top results
are in bold, and second best result are underlined.

Method | Metrics |  |  $\sigma=$ 15 |  |  |  $\sigma=$ 25 |  |  |  $\sigma=$ 50 |
---|---|---|---|---|---|---|---|---|---|---
| BSD68 | Kodak24 | Urban100 | BSD68 | Kodak24 | Urban100 | BSD68 | Kodak24 | Urban100
BM3D[2] | PSNR | 32.37 | 31.07 | 32.35 | 29.97 | 28.57 | 29.70 | 26.72 | 25.62 | 25.95
SSIM | 0.8952 | 0.8717 | 0.9220 | 0.8504 | 0.8013 | 0.8777 | 0.7676 | 0.6864 | 0.7791
WNNM[12] | PSNR | 32.70 | 31.37 | 32.97 | 30.28 | 28.83 | 30.39 | 27.05 | 25.87 | 26.83
SSIM | 0.8982 | 0.8766 | 0.9271 | 0.8577 | 0.8087 | 0.8885 | 0.7775 | 0.6982 | 0.8047
DnCNN[18] | PSNR | 32.86 | 31.73 | 31.86 | 30.06 | 28.92 | 29.25 | 27.18 | 26.23 | 26.28
SSIM | 0.9031 | 0.8907 | 0.9255 | 0.8622 | 0.8278 | 0.8797 | 0.7829 | 0.7189 | 0.7874
FFDNet[26] | PSNR | 32.75 | 31.63 | 32.43 | 30.43 | 29.19 | 29.92 | 27.32 | 26.29 | 26.28
SSIM | 0.9027 | 0.8902 | 0.9273 | 0.8634 | 0.8289 | 0.8886 | 0.7903 | 0.7245 | 0.8057
IrCNN[19] | PSNR | 32.67 | 33.60 | 31.85 | 29.96 | 30.98 | 28.92 | 26.59 | 27.66 | 25.21
SSIM | 0.9318 | 0.9247 | 0.9493 | 0.8859 | 0.8799 | 0.9101 | 0.7899 | 0.7914 | 0.8168
ADNet[35] | PSNR | 32.98 | 31.74 | 32.87 | 30.58 | 29.25 | 30.24 | 27.37 | 26.29 | 26.64
SSIM | 0.9050 | 0.8916 | 0.9308 | 0.8654 | 0.8294 | 0.8923 | 0.7908 | 0.7216 | 0.8073
RIDNet[6] | PSNR | 32.91 | 31.81 | 33.11 | 30.60 | 29.34 | 30.49 | 27.43 | 26.40 | 26.73
SSIM | 0.9059 | 0.8934 | 0.9339 | 0.8672 | 0.8331 | 0.8975 | 0.7932 | 0.7267 | 0.8132
VDN[34] | PSNR | 33.90 | 34.81 | 33.41 | 31.25 | 32.38 | 30.83 | 28.19 | 29.19 | 28.43
SSIM | 0.9243 | 0.9251 | 0.9339 | 0.8713 | 0.8842 | 0.8361 | 0.8014 | 0.7213 | 0.8212
DEAMNet[5] | PSNR | 33.19 | 31.91 | 33.37 | 30.81 | 29.44 | 30.85 | 27.74 | 26.54 | 27.53
SSIM | 0.9097 | 0.8957 | 0.9372 | 0.8717 | 0.8373 | 0.9048 | 0.8057 | 0.7368 | 0.8373
Proposed | PSNR | 33.85 | 32.90 | 33.97 | 31.32 | 30.67 | 31.52 | 29.02 | 28.12 | 28.25
SSIM | 0.9603 | 0.9517 | 0.9621 | 0.9150 | 0.9246 | 0.9241 | 0.8831 | 0.8782 | 0.8755

Table 2: Real-image denoising results of several existing methods on SIDD and
DnD dataset. Top results are in bold, and second best results are underlined.

Method | Metrics | BM3D | DnCNN | FFDNet | VDN | RIDNet | DEAMNet | Proposed
---|---|---|---|---|---|---|---|---
SIDD[38] | PSNR | 25.65 | 23.66 | 29.30 | 39.26 | 37.87 | 39.35 | 39.55
SSIM | 0.685 | 0.583 | 0.694 | 0.944 | 0.943 | 0.955 | 0.964
DnD[37] | PSNR | 34.51 | 32.43 | 37.61 | 39.38 | 39.25 | 39.63 | 39.76
SSIM | 0.8507 | 0.7900 | 0.9115 | 0.9518 | 0.9528 | 0.9531 | 0.9617

Table 3: Running time (in seconds) and parameter comparison

Method | Size $256^{2}$ | Size $512^{2}$ | Size $1024^{2}$ | parameters
---|---|---|---|---
BM3D[2] | 0.76 | 3.12 | 12.82 | -
WNNM[12] | 210.26 | 858.04 | 3603.68 | -
DnCNN[18] | 0.01 | 0.05 | 0.16 | 558k
IrCNN[19] | 0.012 | 0.038 | 0.146 | -
FFDNet[26] | 0.01 | 0.05 | 0.11 | 490k
AINDNet[27] | 0.05 | 0.03 | 0.80 | 13764k
ADNet[35] | 0.02 | 0.06 | 0.20 | 519k
VDN[34] | 0.04 | 0.07 | 0.19 | 7817k
RIDNet[6] | 0.07 | 0.21 | 0.84 | 1499k
DEAMNet[5] | 0.05 | 0.19 | 0.73 | 2225k
Proposed | 0.031 | 0.11 | 0.42 | 846k

###  4.5 Computational Complexity

This section provides a comparison of computational complexity through table
3. The table represents the average running times for the three different
image sizes $256\times 256,512\times 512$, and $1024\times 1024$. In addition,
we present the parameter counts of the compared methods. Apart from BM3D[2],
we report model-specific computation time. In this comparison, we have
considered, BM3D[2], DnCNN[18], WNNM[12], IrCNN[19], FFDNet[26], AINDNet[27],
ADNet[35], VDN[34], RIDNet[6], and DEAMNet[5]. In Table 3, our method’s
computation time is only slightly longer than earlier DnCNN, IrCNN, FFDNet,
ADNet and VDN. In terms of parameter counting, the proposed study is
significantly smaller than the recent RIDNet[6], AINDNet[27], VDN[34], and
DEAMNet[5].

##  5 Conclusion

In this paper, the basic strategy for the lowlevel denoising problem is to
gather a variety of lowlevel features while keeping the interpretation simple
by implementing relatively shallow layers. We argue that for lowlevel vision
tasks, the principle of Occam’s razor is more appropriate, and accordingly, we
have designed a network that focused on gathering a variety of lowlevel
evidence rather than providing an deep explanation of the evidence. Thus, we
have revisited the feature ensemble approach for the image denoising problem.
Our study offers a new model which concatenates different modules for creating
large and varying feature maps. To enhance the performance of our network, we
utilized different kernel sizes, residual and densely-residual connections,
and avoided deep unimodule cascaded aggregation. We carefully designed four
different modules for our study, where each helps to restore different spatial
properties. Finally, we have validated our network with natural and synthetic
noisy images. Extensive comparisons show the overall efficiency of the
proposed study.

## References

  * [1] A. Buades, B. Coll, and J.-M. Morel.  A non-local algorithm for image denoising.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 2, pages 60–65 vol. 2, 2005.
  * [2] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Image denoising by sparse 3-d transform-domain collaborative filtering.  IEEE Transactions on Image Processing, 16(8):2080–2095, 2007.
  * [3] Tao Chen, Kai-Kuang Ma, and Li-Hui Chen.  Tri-state median filter for image denoising.  IEEE Transactions on Image Processing, 8(12):1834–1838, 1999.
  * [4] Jingdong Chen, J. Benesty, Yiteng Huang, and S. Doclo.  New insights into the noise reduction wiener filter.  IEEE Transactions on Audio, Speech, and Language Processing, 14(4):1218–1234, 2006.
  * [5] Chao Ren, Xiaohai He, Chuncheng Wang, and Zhibo Zhao.  Adaptive consistency prior based deep network for image denoising.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8596–8606, June 2021.
  * [6] Saeed Anwar and Nick Barnes.  Real image denoising with feature attention.  IEEE International Conference on Computer Vision (ICCV-Oral), 2019\.
  * [7] Jiahong Ouyang, Ehsan Adeli, Kilian M Pohl, Qingyu Zhao, and Greg Zaharchuk.  Representation disentanglement for multi-modal mr analysis.  arXiv preprint arXiv:2102.11456, 2021.
  * [8] Peixia Li, Boyu Chen, Wanli Ouyang, Dong Wang, Xiaoyun Yang, and Huchuan Lu.  Gradnet: Gradient-guided network for visual object tracking.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6162–6171, 2019.
  * [9] Claude Knaus and Matthias Zwicker.  Progressive image denoising.  IEEE Transactions on Image Processing, 23(7):3114–3125, 2014.
  * [10] Qi Xie, Qian Zhao, Deyu Meng, Zongben Xu, Shuhang Gu, Wangmeng Zuo, and Lei Zhang.  Multispectral images denoising by intrinsic tensor sparsity regularization.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1692–1700, 2016.
  * [11] Michael Elad and Michal Aharon.  Image denoising via sparse and redundant representations over learned dictionaries.  IEEE Transactions on Image Processing, 15(12):3736–3745, 2006.
  * [12] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng.  Weighted nuclear norm minimization with application to image denoising.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 2862–2869, 2014.
  * [13] Yuan Xie, Shuhang Gu, Yan Liu, Wangmeng Zuo, Wensheng Zhang, and Lei Zhang.  Weighted schatten p-norm minimization for image denoising and background subtraction.  IEEE Transactions on Image Processing, 25, 08 2016.
  * [14] Ting Xie, Shutao Li, and Bin Sun.  Hyperspectral images denoising via nonconvex regularized low-rank and sparse matrix decomposition.  IEEE Transactions on Image Processing, PP:1–1, 07 2019.
  * [15] Jun Xu, Lei Zhang, David Zhang, and Xiangchu Feng.  Multi-channel weighted nuclear norm minimization for real color image denoising.  In Proceedings of the IEEE international conference on computer vision, pages 1096–1104, 2017.
  * [16] Jiahao Pang and Gene Cheung.  Graph laplacian regularization for image denoising: Analysis in the continuous domain.  IEEE Transactions on Image Processing, 26(4):1770–1785, 2017.
  * [17] Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia.  Image smoothing via l0 gradient minimization.  ACM Transactions on Graphics (SIGGRAPH Asia), 2011.
  * [18] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  IEEE transactions on image processing, 26(7):3142–3155, 2017.
  * [19] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang.  Learning deep cnn denoiser prior for image restoration.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3929–3938, 2017.
  * [20] Yunjin Chen and Thomas Pock.  Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration.  IEEE transactions on pattern analysis and machine intelligence, 39(6):1256–1272, 2016.
  * [21] Stamatios Lefkimmiatis.  Non-local color image denoising with convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3587–3596, 2017.
  * [22] Woong Bae, Jaejun Yoo, and Jong Chul Ye.  Beyond deep residual learning for image restoration: Persistent homology-guided manifold simplification.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 145–153, 2017.
  * [23] Saeed Anwar, Cong Phouc Huynh, and Fatih Porikli.  Chaining identity mapping modules for image denoising.  arXiv preprint arXiv:1712.02933, 2017.
  * [24] M. Lebrun, M. Colom, and J. M. Morel.  The noise clinic: A universal blind denoising algorithm.  In 2014 IEEE International Conference on Image Processing (ICIP), pages 2674–2678, 2014.
  * [25] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Toward convolutional blind denoising of real photographs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1712–1722, 2019.
  * [26] Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Ffdnet: Toward a fast and flexible solution for cnn-based image denoising.  IEEE Transactions on Image Processing, 27(9):4608–4622, 2018.
  * [27] Yoonsik Kim, Jae Woong Soh, Gu Yong Park, and Nam Ik Cho.  Transfer learning from synthetic to real-noise denoising with adaptive instance normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3482–3492, 2020.
  * [28] Prajit Ramachandran, Barret Zoph, and Quoc V. Le.  Swish: a self-gated activation function.  arXiv: Neural and Evolutionary Computing, 2017.
  * [29] Diganta Misra.  Mish: A self regularized non-monotonic activation function.  arXiv preprint arXiv:1908.08681, 2019.
  * [30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [31] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136–144, 2017.
  * [32] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [33] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual dense network for image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472–2481, 2018.
  * [34] Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang.  Variational denoising network: Toward blind noise modeling and removal.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 1690–1701. Curran Associates, Inc., 2019.
  * [35] Chunwei Tian, Yong Xu, Zuoyong Li, Wangmeng Zuo, Lunke Fei, and Hong Liu.  Attention-guided cnn for image denoising.  Neural Networks, 124:177–129, 2020.
  * [36] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  * [37] Tobias Plotz and Stefan Roth.  Benchmarking denoising algorithms with real photographs.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1586–1595, 2017.
  * [38] Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown.  A high-quality denoising dataset for smartphone cameras.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1692–1700, 2018.