11institutetext: Hasso Plattner Institute, University of Potsdam, Germany
11email<EMAIL_ADDRESS>
https://hpi.de/ 22institutetext: Center for AI, Robert Bosch, Germany
22email<EMAIL_ADDRESS>
https://www.bosch-ai.com/

# Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to
Probe the Boundaries of Stable Diffusion Generated Data

Leonhard Hennicke 11 0009-0009-9928-4579 Christian Medeiros Adriano 11
0000-0003-2588-9937 Holger Giese 11 0000-0002-4723-730X
Jan Mathias Koehler∗, 22 0009-0009-5944-4849 Lukas Schott∗, 22

###### Abstract

Generative foundation models like Stable Diffusion comprise a diverse spectrum
of knowledge in computer vision with the potential for transfer learning,
e.g., via generating data to train student models for downstream tasks. This
could circumvent the necessity of collecting labeled real-world data, thereby
presenting a form of data-free knowledge distillation. However, the resultant
student models show a significant drop in accuracy compared to models trained
on real data. We investigate possible causes for this drop and focus on the
role of the different layers of the student model. By training these layers
using either real or synthetic data, we reveal that the drop mainly stems from
the model’s final layers. Further, we briefly investigate other factors, such
as differences in data-normalization between synthetic and real, the impact of
data augmentations, texture vs. shape learning, and assuming oracle prompts.
While we find that some of those factors can have an impact, they are not
sufficient to close the gap towards real data. Building upon our insights that
mainly later layers are responsible for the drop, we investigate the data-
efficiency of fine-tuning a synthetically trained model with real data applied
to only those last layers. Our results suggest an improved trade-off between
the amount of real training data used and the model’s accuracy. Our findings
contribute to the understanding of the gap between synthetic and real data and
indicate solutions to mitigate the scarcity of labeled real data.

###### Keywords:

Data-free Knowledge Distillation Synthetic Data Foundation Models Stable
Diffusion.

##  1 Introduction

Deep learning has revolutionized computer vision tasks, but training neural
networks in various settings incorporates challenges, like the need for large
amounts of labeled training data [3] and the computational resources required
for deployment [35, 53, 31]. While there are approaches mitigating these
challenges; they often lack specificity, e.g., when using general publicly
available datasets such as ImageNet [12]; or diversity, e.g., when generating
data with GANs [18]. Foundation models [6] trained on large-scale diverse
internet data offer a potential solution by reducing the need for task-
specific training data. However, these models are still large and require
expensive hardware for efficient inference. Knowledge distillation [24] is an
approach to transfer the knowledge from foundation models to smaller student
models. From the variety of different approaches to knowledge distillation
[19], we focus on the concept of data-free knowledge distillation [28].

More concretely, we leverage a generative foundation model to create task-
specific training images for our student model. We base our paper on the
pipeline of Fake It Till You Make It (FITYMI) [41] to train models on image
data synthesized by StableDiffusion [36]. Surprisingly, while synthetic images
are humanly almost indistinguishable from real ones, there is a significant
gap in performance (accuracy) when training neural networks only on synthetic
data. We aim to gain deeper insights into the approximately $23$ percentage
point (pp) gap in accuracy between training fully on synthetic data [41] vs.
real data for ImageNet-100.

Our contributions are threefold: (1) we find that mostly the final
synthetically trained layers are responsible for the drop in accuracy from
synthetic to real, e.g., within a ResNet-50 the training of all but the last
two layers can be done with only synthetic data, resulting in a minor accuracy
drop; (2) we show that one could pre-train all but the last two layers merely
with synthetic data and fine-tune the remaining layers with a fraction of real
data, e.g., using 1/8 of real data on a pre-trained model drops performance by
7pp compared to 20pp when omitting pre-training on synthesized data; (3) we
show that other factors like data-augmentations, oracle prompts, differences
in data normalization, and texture vs. shape, while potentially helpful in
reducing the gap, still do not appear to be sufficient to close the gap
towards real data.

##  2 Preliminaries

Definition 1 - Foundation Models are deep neural networks (DNN) models with
very large numbers of trainable parameters, which, thus, require orders of
magnitude more data than traditional DNN models. Besides their prediction
power, foundation models [6] are more adaptable to downstream tasks.

Definition 2 - Stable Diffusion a.k.a. latent diffusion model (LDM) is derived
from a type of generative model [36] that employs equally weighted denoising
autoencoders to predict the denoised version of input data, with an objective
function minimizing the difference between the added and the predicted noise.

$\mathcal{L}_{LDM}=\mathbb{E}_{\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0,1),t}\left[||\epsilon-\epsilon_{\theta}(z_{t},t,\tau_{\theta}(y))||_{2}^{2}\right]$
(1)

where $\tau_{\theta}$ is the specialized encoder for the additional value
guiding the generation process, which is trained jointly with
$\epsilon_{\theta}$ and $y$ is the additional value. By shifting computations
to a perceptually analogous but lower-dimensional domain through an auto-
encoder, Stable Diffusion reduces the computational complexity because it can
focus on the semantically significant aspects of data.

Definition 3 - Knowledge Distillation consists of transferring knowledge from
a larger teacher model to a smaller student model while maintaining
performance and reducing computational costs [24]. Initially, the teacher
model is trained on a large dataset to achieve high accuracy. Then, the
student model learns to mimic the teacher’s behavior using both the original
dataset and the teacher’s soft targets, which contain probabilistic
distribution insights. A Kullback-Leibler divergence loss function guides the
student model to align its output distribution with that of the teacher model.

Definition 4 - Data-Free Knowledge Distillation precludes the access to the
original training data for transferring knowledge from a teacher model to a
student [28]. Instead, one leverages either the architecture or the learned
parameters of the teacher model to distill knowledge to the student. This
process often involves methods such as feature matching, activation mimicking,
or utilizing generative models to generate synthetic data for distillation
(our approach).

##  3 Related Work

###  3.1 Data-Free Knowledge Distillation

Current works exhibit a significant gap on ImageNet-1k [12] top-1 accuracy if
only trained on synthetic data compared to real data. For instance, StableRep
[47] has $34.9\%$ zero-shot accuracy compared to $73.7\%$ linear probing with
a ViT-B/16 [15]. As a point of reference, we note that CLIP, which has been
trained on real data [32], has $68.6\%$ zero-shot accuracy on a ViT-B/16 and
$59.6\%$ on a ResNet50 [21]. In contrast, SynthCLIP [20] only provides a zero-
shot accuracy of $30.5\%$ with a ViT-B/16 backbone. Lastly, also [41] only
achieves $42.9\%$ with a ResNet50. From our perspective, this gap in accuracy
is quite puzzling, as for humans, synthetic images from models like
StableDiffusion look almost indistinguishable from real ones.

We base our paper on the pipeline of Fake It Till You Make It (FITYMI) [41].
Analogously, to their approach training models on image data generated by
StableDiffusion [36]. We adapt the FITYMI setup for our experiments to make
our results directly comparable. We re-compute our replication of the paper’s
best-performing model for ImageNet-100 in all our result tables. Note that due
to random seeds, we observe minor fluctuations ($<1\%$ compared to the results
in Table 1 in [41] with a prompt scheme consisting of class and definition
$p_{c}=$"$c,d_{c}$", and guidance scale $=2$). Furthermore, we focus on using
the ImageNet-100 dataset instead of ImageNet-1K due to resource constraints,
similar to most experiments in FITYMI. Most importantly, this reduced dataset
still exhibits a fairly low accuracy of just $28.4\%$ if it is trained in a
naive training fashion, e.g., without augmentations. To investigate parts of
this gap, the FITYMI paper already covers some ablations on ImageNet-100 like
variations of prompts, augmentations, and the guidance scale that each
significantly impact the performance. Especially, introducing augmentations
improves the baseline performance by $14.8$ pp. Similarly, tuning the guidance
scale improves the performance by $20$ pp. Note that both factors have been
tuned in isolation and the benefits might be less than their sum if combined.
While the aforementioned tricks increase the performance, a large portion of
the gap remains, e.g., a vanilla training on the real ImageNet-100 data leads
to $87.4\%$ top-1 accuracy compared to the best data-free variant from [41]
achieving $64.8\%$. Thus, we extend the ablations by testing out further
factors and investigating where in the network discrepancies between real and
synthetic data originate.

Similarly, to FITYMI, StableRep [47] presents a pipeline learning
representations solely based on synthetic images. They use captions from the
datasets CC3M [44], CC12M [9] or RedCaps [13] respectively to generate image
data with Stable Diffusion v1.5 and then train the model on the generated data
in an unsupervised manner to learn the representations used for linear probing
and zero-shot classification. While the synthetic ImageNet-1K only includes
less than 1.3 million samples, StableRep uses 10 million and 11.6 mio. samples
from CC12M and RedCaps respectively. For CC3M only 2.7 mio. samples are used.
Also, StableRep trains a ViT-B/16 ($\sim 86$ mio. trainable params.) for 35
epochs, whereas we train a ResNet50 ($\sim 25$ mio. trainable params.) for 100
epochs.

Lastly, Azizi et al. [2] use Imagen [40] instead of StableDiffusion to
generate synthetic training data for image classification. They achieve a
top-1 accuracy of 69.24% on ImageNet using a ResNet50 trained for 90 epochs;
however, they achieve this by fine-tuning Imagen on ImageNet data, resulting
in their approach not being completely data-free.

###  3.2 Transfer Learning

##### Transfer Learning - TL.

Reusing knowledge from pre-trained models via TL requires that data from pre-
trained tasks (source) and new ones (target) have some overlap. Unfortunately,
this assumption is unrealistic in domains subject to data sparsity [8]. To
bridge this gap between source-target labels, one can combine similarity-
guided training and adaptive adversarial noise selection [50]. Another
approach is to apply self-supervision methods, for instance, Yamaguchi et al.
[52] investigated a two-stage process that (1) synthesizes target samples by
conditioning on source generative model and, then, (2) labels these new
samples via self-supervised learning methods. Albeit TL has shown to
outperform knowledge distillation [52], one still has to cope with the threat
of "negative transfer", i.e., learning wrong concepts or forgetting correct
ones [54].

##### Catastrophic forgetting - CF.

Re-training models needs to cope with the fundamental phenomenon of
catastrophic forgetting [29], i.e., neural networks when trained on different
tasks tend to "forget" (lose accuracy) in previously learned tasks. This
phenomenon is not limited to sequential learning setups (e.g., continual
learning, reinforcement learning, domain adaptation). Fine-tuning of
foundation and generative models are also affected [51], for instance, in
generative adversarial networks - GANs trained continuously [49]. Some of the
mitigation to CF involve momentum and gradient penalty [46] or changes in the
discriminator component of the GAN [27]. Meanwhile, in GANs continuously
trained in the context of data-free knowledge distillation (DFKD) [10],
mitigation of CF by preserving memory buffers for past samples [5],
remembering previous distributions for generative replay [4], and keeping an
exponential moving average to minimize the impact of abrupt shifts in the
distribution [14]. Our approach is complementary to these approaches, as we
combine an architecture-based approach (fixing parameters, i.e., layers) with
synthetic or real data for pre-training and then fine-tuning. Catastrophic
forgetting could happen in various steps of the pipeline, for instance, the
synthetic images cause the student model to forget features that were
successfully learned by the teacher model. While freezing layers might
mitigate this loss, it might still happen in the retrained layers, because one
cannot guarantee that all fine-tuned knowledge is new knowledge.

##  4 Experiments

###  4.1 Experimental Setup

The premise of our experiments is strongly inspired by the findings of the
paper Fake It Till You Make It (FITYMI) [41]. To make our results comparable
with the ones from FITYMI, all of our experiments were run using the same
setup that was used by the authors, i.e., training a randomly initialized
ResNet-50 [21] for 100 epochs using SGD [39] with a momentum of 0.9 and DINO
augmentations with one global and eight local crops, unless stated otherwise.
We linearly increased the learning rate up to 0.1 for the first 10 epochs and
then decayed it on a cosine schedule. In addition, temperature scaling with
L2-normalized weights was used on the final classification layer. Training a
model using this setup takes over 56 single GPU hours, but it can be
parallelized. Our implementation is based on the implementation of [42] \-
which is publicly available - adapted for our classification task. The dataset
we use is ImageNet-100 [48], a subset of ImageNet-1K [12]. This is the same
subset of classes that is used in FITYMI. We use either the original data from
ImageNet-100 or a synthetic version of this dataset with 1300 images generated
per class. The synthetic images were generated using Stable Diffusion 1.4
unless stated otherwise. This was done using 50 diffusion steps and a guidance
scale [25] of 2.0. The images were generated sized 512 x 384 and using the
WordNet [30] based definition prompts from FITYMI (i.e., "<class name>,
<WordNet definition of class name>"). Generating a full ImageNet-100 dataset
with this setup takes over 73 single-GPU hours. For our experiments, we report
the mean and standard deviation on real ImageNet-100 validation data of the
last 5 epochs for top-1 and top-5 accuracy.

(a) Real Data Transfer Learning.

(b) Synthetic Data Transfer Learning.

Figure 1: This figure illustrates our transfer learning setup for N = 2. N
indicates the number of consecutive layers that are transferred from a model
that was trained on the respective first dataset, starting from the first
layer.

###  4.2 Layer Importance Experiments

It is a widely accepted intuition that CNNs (such as the ResNet-50 we are
using) learn representations of more abstract features in the later layers of
the model while the earlier layers learn representations of features at a
lower abstraction layer [26]. Based on this intuition, we devised a series of
experiments using transfer learning to narrow down which features are missing
from the synthetic data. In these experiments, we pre-train a CNN on either
synthetic or real data and then freeze the first N layers. Next, we
reinitialize the remaining layers and retrain them on the respective other
dataset. As we gradually increase N, we expect to see changes in the accuracy
of the resulting model, that relate to the quality of features at gradually
higher levels of abstraction that are present in the data on which the model
has been pre-trained. When using the term layer in the context of our
experiments, we refer to the bottleneck blocks in the ResNet-50 architecture.
The only exceptions to that are the first layer, which is the singular
convolutional block that comes before the bottleneck blocks, and the last
(18th) layer, which consists of the final (linear) classification layer and
the pooling layer that precedes it. Running this setup for N = 17 results in a
setup equivalent to linear probing, while N = 18 would result in only
evaluating a completely pre-trained model with all parameters frozen.

####  4.2.1 Real Data Transfer Learning

In our first series of experiments, the first N layers are transferred from a
model pre-trained on real data with frozen parameters, while only training the
remaining layers of the model on synthetic data. We evaluate on real data. We
illustrate this setup in Figure 1(a) for $N=2$.

(a) Real Data Transfer Learning.

(b) Synthetic Data Transfer Learning.

Figure 2: Results of the layer importance experiments using transfer learning
from real and synthetic data respectively for N = 1 to N = 17. We also plot
the results for our baseline models trained on synthetic and real data only,
as these present the two extremes of this experiment setup and therefore
provide an approximate lower and upper bound.

The results of this experiment are presented in Figure 2(a). If the
representations learned by the different layers of the model were to all
transfer equally well to real data, we would expect the accuracy to gradually
rise as we replace more and more of the layers with the ones that were pre-
trained on real data and fewer layers are trained on synthetic data. However,
this is not the case, suggesting that the quality of the representations that
the student model learns from the synthetic data varies depending on their
level of abstraction.

As we can see in Figure 2(a), the top-1 accuracy increases by at most 2.0pp
from the baseline respectively as we add the pre-trained layers 1 to 14 and
even decreases for some of the in-between layers. In contrast to that, when
adding the pre-trained layers 15 to 17, we can see progressively larger
increases111Here we refer to the increases relative to the model with only the
previous pre-trained layers added respectively. in top-1 accuracy of 18.4pp in
total, which is visible by a steep increase for the layers 15 to 17. Using the
representations learned from real data by the first 14 layers makes little
difference over using the ones learned from synthetic data compared to the
differences using the representations learned by the later layers.

####  4.2.2 Synthetic Data Transfer Learning

To further support our claim, we present a second series of experiments
similar to the ones in the previous section. Now, the first N layers are
transferred from a model pre-trained on synthetic data with frozen parameters,
while the remaining layers of the model are reinitialized and trained on real
data (setup illustrated in Figure 1(b) for N = 2). Evaluation is always
performed on real validation data. Similarly to our first series of
experiments, if the representations learned from the synthetic data by the
different layers of the model were all transferred equally well to real data,
we would expect the accuracy to gradually decrease as we replace more and more
of the layers with the ones that were pre-trained on synthetic data and fewer
layers trained on real data are used. The results of this experiment are
presented in Figure 2(b) where we plot the top-1 accuracy against the number
of layers transferred. Supporting our previous findings, we can see that the
accuracy varies only little when pre-training layers 1 to 16 with synthetic
data. Specifically, the top-1 accuracy only decreases by 3.5 pp, as the layers
1 to 16 are replaced by the ones pre-trained on synthetic data. In fact, the
top-1 accuracy even increases up to 1.0pp as some of the in-between pre-
trained layers are added.

A larger decrease of 11.3pp in top-1 accuracy is visible as the 17th layer
(the second to last layer) is replaced by one that is pre-trained on synthetic
data - noticeable as a steep drop in Figure 2(b). The decrease in accuracy is
accompanied by an increase in training loss. This suggests that the decrease
in accuracy is probably not due to the model overfitting, but rather due to
the features learned from synthetic data not transferring well to the real
evaluation data.

###  4.3 Data-Reduction Experiments

Based on the results of our layer importance experiments, we suspect that a
model with only the last layers trained on real data (and the earlier layers
were already pre-trained on synthetic data) would require less data to achieve
a similar accuracy as a baseline model trained fully on real data. To test
this hypothesis, we pre-train the first 16 layers of our model on synthetic
data, i.e., $N=16$, and then reinitialize and retrain the remaining layers on
a randomly drawn, reduced number of samples from the real ImageNet-100
dataset, effectively reducing the amount of real data necessary for training.
Assuming our intuition is correct, we should observe that the accuracy remains
within a similar range as the baseline model accuracy, as we decrease the
amount of real training data used.

The results of these experiments are presented in Figure 3. We decrease the
amount of real training data by halving it for each consecutive experiment.
Even though we reduce the real training exponentially, the top-1 accuracy only
decreases by 4.2pp after the third iteration (when using 1/8th of the entire
real training data).

(a) Linear Scale (Read Right to Left)

(b) Logarithmic Scale (Read Left to Right)

Figure 3: Data Reduction Experiments The first 16 layers are trained on
synthetic data and frozen. The last two layers are fine-tuned with a different
number of real data samples. The fitted curves are derived from the plotted
top-1 accuracy on the respective scales via curve fitting, using least squares
polynomial fitting on a 1st-degree polynomial ($f(x)=-2.23log(x)+85.61$).

The fitted curve suggests that the top-1 accuracy decreases logarithmically,
as the amount of real training data is reduced. Additionally, we show in
Figure 4 a series of reductions in real training data for a randomly
initialized model, the drop in accuracy is steeper than for the models with
frozen layers pre-trained on synthetic data, suggesting that synthetic pre-
training can be applied to mitigate the lowered accuracy caused by a lack of
real data.

(a) Linear Scale (Read Right to Left)

(b) Logarithmic Scale (Read Left to Right)

Figure 4: Training on a reduced real dataset Top-1 accuracy of reducing the
amount of real training data with (blue, same as Figure 3) and without
(orange) synthetic data transfer learning.

###  4.4 Other Potential Causes of the Accuracy Gap

##### Normalization

We investigate if differences in first and second-order statistics (dataset
mean and variance) of the synthetic data that is used during training, and the
real data that is used during evaluation, cause the drop in performance. We
tested two interventions during the training of the student model. 1) We
normalize the input using the actual (channel) mean and standard deviation of
the respective dataset instead of the default ImageNet values during both
training and evaluation. 2) We set the batch norm layers to train mode during
evaluation, causing them to keep updating their running mean and std. dev.
instead of using the ones learned during training. We observe a slight
improvement in accuracy for these interventions on both real and synthetic
data (see Table 1). However, the gap might not to be explainable by exact
normalization.

Table 1: Normalization Experiments. Exact normalization: image normalization using the actual (channel) mean / std. dev. of (R) or (S) data instead of ImageNet values during training and evaluation. Batch norm: batch normalization layers keep updating running batch mean and std. dev. during evaluation. Experiment |  Top-1 Acc. (%) |  Top-5 Acc. (%) |  Train Loss
---|---|---|---
Real Data Only (R) |  87.8 ±0.1 |  97.1 ±0.1 |  0.742 ±0.002
\+ Exact Input Normalization |  88.5 ±0.1 |  97.5 ±0.1 |  0.728 ±0.003
\+ Batch Norm in Train Mode |  88.5 ±0.1 |  97.3 ±0.1 |  0.726 ±0.003
Synth. Data Only (S) |  64.8 ±0.1 |  87.6 ±0.1 |  0.696 ±0.003
\+ Exact Input Normalization |  65.9 ±0.1 |  87.6 ±0.2 |  0.694 ±0.004
\+ Batch Norm in Train Mode |  65.2 ±0.2 |  87.3 ±0.2 |  0.700 ±0.003

##### Data Augmentation

As shown in FITYMI [41], the type of augmentations can make a difference for
models trained on synthetic data, more so than when training on real data.
Based on this, we suspected that the gap could be bridged through data
augmentation. We test several sophisticated augmentation pipelines, both for
models trained on real and on synthetic data. However, the accuracy gap still
remains and we could also not replicate the effect of these augmentations
w.r.t. leading to a larger improvement when training on synthetic data (as
shown in FITYMI) compared to our baseline using no data augmentations. As
shown in Table 2, we still observe this effect when comparing the specific
augmentations used in FITYMI, i.e., the ones from the PyTorch example [1] and
the ones from DINO [7]. However, the effect is less pronounced in our results
than in the ones shown in FITYMI. The difference is likely due to a lower
guidance scale being used for the specific experiment in FITYMI. This suggests
that the increased improvisation of Stable Diffusion (due to the lower
guidance scale) has a positive effect on the generated data that is similar to
the augmentations.

Table 2: Augmentation Experiments. The models were trained using different augmentation techniques, as specified in the Experiment column. To accurately portray the impact of the different augmentations, the baseline was trained using the FITYMI setup, as in our other experiments, but without the DINO augmentations. Hence, in this table, only the runs labeled "DINO" are equivalent to the full FITYMI setup, that we us as a baseline everywhere else. Experiment |  Top-1 Acc. (%) |  Top-5 Acc. (%) |  Train Loss
---|---|---|---
Real Data Only (R) |  70.1 ±0.1 |  88.2 ±0.2 |  0.007 ±0.001
\+ AutoAugment |  79.7 ±0.1 |  94.6 ±0.1 |  0.033 ±0.001
\+ PyTorch Example [1] |  86.2 ±0.1 |  96.5 ±0.1 |  0.308 ±0.003
\+ AugMix [22] |  86.3 ±0.1 |  97.2 ±0.1 |  0.318 ±0.002
\+ PixMix [23] |  86.5 ±0.1 |  97.2 ±0.1 |  0.848 ±0.007
\+ DINO [7] |  87.8 ±0.1 |  97.1 ±0.1 |  0.742 ±0.002
Synth. Data Only (S) |  46.9 ±0.2 |  71.7 ±0.2 |  0.006 ±0.001
\+ AutoAugment |  56.2 ±0.2 |  81.4 ±0.1 |  0.026 ±0.100
\+ PyTorch Example [1] |  59.5 ±0.1 |  83.9 ±0.2 |  0.261 ±0.002
\+ AugMix [22] |  60.3 ±0.2 |  84.5 ±0.2 |  0.281 ±0.003
\+ PixMix [23] |  61.8 ±0.2 |  84.7 ±0.2 |  0.680 ±0.006
\+ DINO [7] |  64.8 ±0.1 |  87.6 ±0.1 |  0.696 ±0.003

##### Local Textures

Images generated by Stable Diffusion, often have inconsistencies in finer
structures (like human hands) or local textures [37]. This leads us to the
question of whether local textures generated by Stable Diffusion are less
useful to the student model when generalizing to real data. This would
contribute to the accuracy gap, as it has been shown repeatedly, that
convolutional neural networks, such as our student models are biased towards
local textures [17]. To test this, we generated two new datasets by applying
the techniques from Stylized ImageNet [17] to our real and synthetic
ImageNet-100 datasets respectively. Stylized ImageNet uses style transfer
techniques in combination with paintings to remove local texture cues and in
return only retain global shape information. Despite this intervention, the
accuracy gap remains (see Table 3). Note the lower accuracy values w.r.t. the
baseline. This is expected, as we introduce an information loss to our data by
removing features from local textures.

Table 3: Stylized ImageNet Experiments. The entire training dataset was modified using the techniques from Stylized ImageNet, removing local texture cues but retaining the global shape information. Experiment |  Top-1 Acc. (%) |  Top-5 Acc. (%) |  Train Loss
---|---|---|---
Real Data Only (R) |  87.8 ±0.1 |  97.1 ±0.1 |  0.742 ±0.002
\+ Stylized ImageNet |  75.6 ±0.2 |  92.2 ±0.1 |  2.099 ±0.005
Synth. Data Only (S) |  64.8 ±0.1 |  87.6 ±0.1 |  0.696 ±0.003
\+ Stylized ImageNet |  50.5 ±0.1 |  77.0 ±0.2 |  2.053 ±0.006

##### Prompt Optimization

Based on the findings in FITYMI, we asked whether the accuracy gap may be
caused by unused potential in prompt engineering. To simulate optimal prompts,
we applied unCLIP [34] to generate synthetic images with Stable Diffusion
based on CLIP [33] embeddings of real ImageNet images as oracle prompts
instead of text embeddings as is in our other experiments, i.e., we generate
one synthetic image for each ImageNet-100 image using the real image as an
input for unCLIP. As CLIP creates a shared embedding space for text and
images, this in essence simulates the perfect text description for each of the
real images. Note that, since the Stable Diffusion (SD) model used in unCLIP
is fine-tuned to also accept image embeddings, the image prompts provide more
information than just perfect text prompts, as the resulting embedding vectors
will have the perfect magnitude in addition to the perfect angle. In our
evaluation, we switch to a synthetic dataset generated by Stable Diffusion
v2.1 for a fair comparison as this is the version used by unCLIP. As seen in
Table 4, we do not bridge the accuracy gap with this intervention. However,
compared to the lower-performing SD v2.1 baseline, we achieve a top-1 accuracy
improvement of $26.3$ pp, which is larger than the original accuracy gap of
$23$pp between real data and synthetic data generated by SD v1.4. Potentially,
this could allow prompt engineering to bridge the accuracy gap, but, one
should be careful drawing conclusions from this experiment. CLIP is trained on
a contrastive loss based on cosine similarity between text and image
embeddings while unCLIP is trained on CLIP image embeddings. Hence, prompts
based on image embeddings could induce data leakage, as they provide more
information than even a perfect text prompt would do, e.g., by having both the
perfect angle and magnitude.

Table 4: unCLIP Experiment. We compare student models trained on various synthetic datasets generated by Stable Diffusion. "unCLIP" indicates that the model was trained on a dataset generated using unCLIP (based in Stable Diffusion v2.1) with the real ImageNet-100 images as prompts. SD v1.4 and SD v2.1 indicate the respective Stable Diffusion versions used to generate the dataset. Experiment |  Top-1 Acc. (%) |  Top-5 Acc. (%) |  Train Loss
---|---|---|---
unCLIP, (SD v2.1) |  66.6 ±0.3 |  90.8 ±0.1 |  0.690 ±0.004
Synth. Data Only (SD v2.1) |  40.3 ±0.3 |  66.3 ±0.4 |  0.511 ±0.002
Synth. Data Only (SD v1.4) |  64.8 ±0.1 |  87.6 ±0.1 |  0.696 ±0.003

##  5 Discussion

###  5.1 Implications to Transfer Learning

We showed that choosing specific layers to transfer can have a positive impact
on transferring knowledge from the teacher to the student model while fine-
tuning with real data. This knowledge can be used for mitigating catastrophic
forgetting in situations where knowledge distillation is necessary to cope
with data sparsity and to produce more compact models. Because synthetic data
generation is extremely flexible, this extra control leverage can support a
more fine-grained study of how certain image features in a particular class
(e.g., vehicle type) or domain (e.g., weather conditions) are more prone to be
forgotten given the previously learned features. Such extra flexibility could
enable "negative transfer" strategies for mitigating shortcut learning [16],
hence endowing the fine-tuning pipeline and the resulting models with
robustness to non-stationary data generation processes. Ultimately, combining
synthetic data with layer freezing could provide control baselines to evaluate
the existing mitigation strategies against CF in continuously trained GANs
[46, 27], particularly in DFKD approaches [5, 14].

###  5.2 Threats to Validity

Next, we discuss possible threats to the validity [43] of our evidences and
claims. Construct validity threats consist of concepts misrepresenting the
object of study, for instance, assuming that accuracy represents the
performance of the classifier when the samples are imbalanced (i.e., unequal
number of negative and positive instances) or when false negatives and false
positives have distinct levels of importance (i.e., costs, risks/safety) for
downstream tasks (e.g., planning). We mitigated this validity threat by
working only with balanced datasets. Although sound, this assumption might be
too strong for real-world settings, where number of instances, relative
costs/risks, and error rates are not homogeneous or comparable across classes
[45]. Internal validity threats relate to assumptions that can be invalidated
by alternative explanations for the measured effects or lack thereof, e.g.,
hidden confounders or selection bias induced by data leakage. We mitigated
this by controlling certain parameters (e.g., freezing vs. reinitializing
layers), taking the last 5 training epochs, and evaluating the top-1/top-5
outcomes. However, as we were unable to control all the parameters, there
might be vestiges of confounding, e.g., (1) how the data augmentation might
have produced selection biases (certain types of images benefiting more from
the augmentations) or (2) the competing effects of oracle prompts and
additional information the image embeddings used to simulate them may contain.
A promising avenue is to further probe with the simulated prompts, as the
positive effects of prompt engineering are corroborated by similar settings
like WaffleCLIP [38]. External validity threats correspond to configurations
(e.g., hyperparameters) that hinder the generalization/reproduction of similar
accuracy gains in future experiments (e.g., under different methods/domains).
We mitigated this risk by performing various sensitivity analyses, for
instance, varying the intake order of real vs. synthetic data in pre-training
and the number of frozen layers. Albeit extensive, our evaluations were not
exhaustive. Further promising work is to study if different architecture types
and domains/classes can induce the phenomenon of underspecification [11],
where models with equivalent performance on holdout sets still show distinct
outcomes after being deployed.

###  5.3 Conclusion and Future Work

We investigated the accuracy gap between models trained on synthetic data vs.
real data. Looking into the role of data normalization, batch normalization,
and data augmentations for synthetic vs. real data, we identified improvements
that explain part of the observed accuracy gap. Our results indicate that the
final layers of the student model play a significant role in bridging this
gap. When pre-training all but the last two layers using synthetic data, we
observed only a minor drop in accuracy. Furthermore, we demonstrated that pre-
training the majority of the layers with synthetic data and fine-tuning the
remaining layers with a fraction of real data resulted in improved performance
compared to models trained merely on a subset of real data. These results
suggest leveraging synthetic data to mitigate a lack of labeled real training
data. Our findings point to promising new avenues for future research.
Firstly, a deeper exploration of the specific features learned by the final
layers could shed light on their contribution to the observed accuracy gaps.
Additionally, investigating various ratios of synthetic and real data during
fine-tuning may reveal better trade-offs, potentially incorporating active
learning to select the most informative labeled data. Lastly, an investigation
into how different generative foundation models impact student model
performance would provide a comprehensive understanding of the potential of
synthetic data in transfer learning.

####  5.3.1 Acknowledgements

This work was supported in part by Bosch Center for Artificial Intelligence
and the Hasso-Plattner-Institute. We thank Claudia Blaiotta, Bastian Bischoff,
Niclas Popp, Anna Khoreva, and Jan H. Metzen for their discussions and
feedback on the manuscript. The Bosch Group is carbon neutral. Administration,
manufacturing and research activities do no longer leave a carbon footprint.
This also includes GPU clusters on which most experiments have been performed.

####  5.3.2

The authors have no competing interests to declare that are relevant to the
content of this article.

## References

  * [1] AI, M.: Pytorch examples. http://github.com/pytorch/examples (2013)
  * [2] Azizi, S., Kornblith, S., Saharia, C., Norouzi, M., Fleet, D.J.: Synthetic data from diffusion models improves imagenet classification. arXiv:2304.08466 (2023)
  * [3] Bhardwaj, K., Suda, N., Marculescu, R.: Edgeal: A vision for deep learning in the iot era. IEEE Design & Test 38(4), 37–43 (2019)
  * [4] Binici, K., Aggarwal, S., Pham, N.T., Leman, K., Mitra, T.: Robust and resource-efficient data-free knowledge distillation by generative pseudo replay. In: AAAI Conf. on Artificial Intelligence. vol. 36, pp. 6089–6096 (2022)
  * [5] Binici, K., Pham, N.T., Mitra, T., Leman, K.: Preventing catastrophic forgetting and distribution mismatch in knowledge distillation via synthetic data. In: CVF Conf. on Applications of Computer Vision. pp. 663–671 (2022)
  * [6] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., et al.: On the opportunities and risks of foundation models. arXiv:2108.07258 (2021)
  * [7] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging Properties in Self-Supervised Vision Transformers. In: ICCV Intl. Conf. on Computer Vision (2021)
  * [8] Chang, X., Yang, Y., Xiang, T., Hospedales, T.M.: Disjoint label space transfer learning with common factorised space. In: AAAI Conf. on Artificial Intelligence. vol. 33, pp. 3288–3295 (2019)
  * [9] Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In: CVPR Conf. on Computer Vision and Pattern Recognition. pp. 3558–3568 (2021)
  * [10] Chen, H., Wang, Y., Xu, C., Yang, Z., Liu, C., Shi, B., Xu, C., Xu, C., Tian, Q.: Data-free learning of student networks. In: CVF Intl. Conf. on Computer Vision. pp. 3514–3522 (2019)
  * [11] D’Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M.D., et al.: Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research 23(226), 1–61 (2022)
  * [12] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet a large-scale hierarchical image database. In: CVPR Conf. on computer vision and pattern recognition. pp. 248–255 (2009)
  * [13] Desai, K., Kaul, G., Aysola, Z., Johnson, J.: RedCaps: Web-curated image-text data created by the people, for the people. arXiv:2111.11431 (2021)
  * [14] Do, K., Le, T.H., Nguyen, D., Nguyen, D., Harikumar, H., Tran, T., Rana, S., Venkatesh, S.: Momentum adversarial distillation: Handling large distribution shifts in data-free knowledge distillation. Advances in Neural Information Processing Systems 35, 10055–10067 (2022)
  * [15] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929 (2020)
  * [16] Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A.: Shortcut learning in deep neural networks. Nature Machine Intelligence 2(11), 665–673 (2020)
  * [17] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel, W.: Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv:1811.12231 (2018)
  * [18] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Communications of the ACM 63(11), 139–144 (2020), publisher: ACM New York, NY, USA
  * [19] Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: A survey. Intl. Journal of Computer Vision 129(6), 1789–1819 (2021)
  * [20] Hammoud, H.A.A.K., Itani, H., Pizzati, F., Torr, P., Bibi, A., Ghanem, B.: Synthclip: Are we ready for a fully synthetic clip training? arXiv:2402.01832 (2024)
  * [21] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR Conf. on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [22] Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.: AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. ICLR Intl. Conf. on Learning Representations (2020)
  * [23] Hendrycks, D., Zou, A., Mazeika, M., Tang, L., Li, B., Song, D., Steinhardt, J.: PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures. In: CVPR Conf. on Computer Vision and Pattern Recognition (2022)
  * [24] Hinton, G., Vinyals, O., Dean, J., et al.: Distilling the knowledge in a neural network. arXiv:1503.02531 (2015)
  * [25] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv:2207.12598 (2022)
  * [26] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553) (2015)
  * [27] Liang, K.J., Li, C., Wang, G., Carin, L.: Generative adversarial network training is a continual learning problem. arXiv:1811.11083 (2018)
  * [28] Lopes, R.G., Fenu, S., Starner, T.: Data-free knowledge distillation for deep neural networks. arXiv:1710.07535 (2017)
  * [29] McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of learning and motivation, vol. 24, pp. 109–165. Elsevier (1989)
  * [30] Miller, G.A.: Wordnet: a lexical database for english. Communications of the ACM 38(11), 39–41 (1995)
  * [31] Miotto, R., Wang, F., Wang, S., Jiang, X., Dudley, J.T.: Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics 19(6), 1236–1246 (2018)
  * [32] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML Intl. Conf. on Machine Learning. pp. 8748–8763 (2021)
  * [33] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML Intl. Conf. on Machine Learning. pp. 8748–8763 (2021)
  * [34] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv:2204.06125 1(2), 3 (2022)
  * [35] Ravi, D., Wong, C., Lo, B., Yang, G.Z.: A deep learning approach to on-node sensor data analytics for mobile or wearable devices. IEEE Journal of Biomedical and Health Informatics 21(1), 56–64 (2016)
  * [36] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR Conf. on Computer Vision and Pattern Recognition. pp. 10684–10695 (2022)
  * [37] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR Conf. on Computer Vision and Pattern Recognition. pp. 10684–10695 (2022)
  * [38] Roth, K., Kim, J.M., Koepke, A., Vinyals, O., Schmid, C., Akata, Z.: Waffling around for performance: Visual classification with random words and broad concepts. In: CVF Intl. Conf. on Computer Vision. pp. 15746–15757 (2023)
  * [39] Ruder, S.: An overview of gradient descent optimization algorithms. arXiv:1609.04747 (2016)
  * [40] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., et al.: Photorealistic text-to-image diffusion models with deep language understanding. arXiv:2205.11487 (2022)
  * [41] Sarıyıldız, M.B., Alahari, K., Larlus, D., Kalantidis, Y.: Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In: CVPR Conf. on Computer Vision and Pattern Recognition. pp. 8011–8021 (2023)
  * [42] Sariyildiz, M.B., Kalantidis, Y., Alahari, K., Larlus, D.: No reason for no supervision: Improved generalization in supervised models. arXiv preprint arXiv:2206.15369 (2022)
  * [43] Shadish, W.R., Cook, T.D., Campbell, D.T.: Experimental and quasi-experimental designs for generalized causal inference. Houghton, Mifflin and Company (2002)
  * [44] Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: ACL Association for Computational Linguistics. pp. 2556–2565 (2018)
  * [45] Stöckl, A.: Evaluating a synthetic image dataset generated with stable diffusion. arXiv:2211.01777 (2022)
  * [46] Thanh-Tung, H., Tran, T.: Catastrophic forgetting and mode collapse in gans. In: IJCNN Intl. Joint Conf. on Neural Networks. pp. 1–10. IEEE (2020)
  * [47] Tian, Y., Fan, L., Isola, P., Chang, H., Krishnan, D.: Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems 36 (2024)
  * [48] Tian, Y., Krishnan, D., Isola, P.: Contrastive multiview coding. In: ECCV European Computer Vision Conf. pp. 776–794. Springer (2020)
  * [49] Wang, L., Zhang, X., Su, H., Zhu, J.: A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
  * [50] Wang, X., Lin, B., Liu, D., Xu, C.: Efficient transfer learning in diffusion models via adversarial noise (2023)
  * [51] Wang, Z., Yang, E., Shen, L., Huang, H.: A comprehensive survey of forgetting in deep learning beyond continual learning. arXiv:2307.09218 (2023)
  * [52] Yamaguchi, S., Kanai, S., Kumagai, A., Chijiwa, D., Kashima, H.: Transfer learning with pre-trained conditional generative models. arXiv:2204.12833 (2022)
  * [53] Zhang, T., Gao, C., Ma, L., Lyu, M., Kim, M.: An empirical study of common challenges in developing deep learning applications. In: ISSRE Intl. Symp. on Software Reliability Engineering. pp. 104–115 (2019)
  * [54] Zoph, B., Ghiasi, G., Lin, T.Y., Cui, Y., Liu, H., Cubuk, E.D., Le, Q.: Rethinking pre-training and self-training. Advances in neural information processing systems 33, 3833–3845 (2020)


# MM-BSN: Self-Supervised Image Denoising for Real-World with Multi-Mask based
on Blind-Spot Network

Dan Zhang  Fangfang Zhou  Senslab Technology, Shanghai, China  Yuwen Jiang
Senslab Technology, Shanghai, China  Zhengming Fu  NeuroSens Technology,
Austin, U.S.

###### Abstract

Recent advances in deep learning have been pushing image denoising techniques
to a new level. In self-supervised image denoising, blind-spot network (BSN)
is one of the most common methods. However, most of the existing BSN
algorithms use a dot-based central mask, which is recognized as inefficient
for images with large-scale spatially correlated noise. In this paper, we give
the definition of large-noise and propose a multi-mask strategy using multiple
convolutional kernels masked in different shapes to further break the noise
spatial correlation. Furthermore, we propose a novel self-supervised image
denoising method that combines the multi-mask strategy with BSN (MM-BSN). We
show that different masks can cause significant performance differences, and
the proposed MM-BSN can efficiently fuse the features extracted by multi-
masked layers, while recovering the texture structures destroyed by multi-
masking and information transmission. Our MM-BSN can be used to address the
problem of large-noise denoising, which cannot be efficiently handled by other
BSN methods. Extensive experiments on public real-world datasets demonstrate
that the proposed MM-BSN achieves state-of-the-art performance among self-
supervised and even unpaired image denoising methods for sRGB images
denoising, without any labelling effort or prior knowledge. Code can be found
in https://github.com/dannie125/MM-BSN.

(a) DnCNN [43]
(Supervised)

(b) C2N [16]+DIDN [39]
(Unpaired)

(c) AP-BSN [22]
(Self-supervised)

(d) MM-BSN
(Self-supervised)

Figure 1: Visual comparison of our MM-BSN with other competing methods on the
DND benchmark. (a) DnCNN is trained on real-world noisy-clean pairs from the
SIDD Medium dataset [1]. (b) C2N uses clean SIDD [1] and noisy DND [33]
samples to simulate the real-world noise distribution in an unsupervised
manner. (c) AP-BSN is trained directly on the noisy images in the SIDD Medium
dataset [1]. (d) MM-BSN is trained on images with real noise from SIDD. We
mark the PSNR (dB) and SSIM with respect to the groundtruth for the
quantitative comparison.

##  1 Introduction

Image denoising is a key step in image processing, and the denoising
performance has a significant impact on the subsequent image processing tasks.
Traditional image denoising methods [8, 27, 11] are time consuming and costly,
but usually have poor robustness in real-world applications. With the
advancement of deep learning, learning-based image denoising algorithms have
made great progress and can be divided into two classes, supervised methods
and self-supervised methods.

The supervised denoising methods [43, 2, 5, 12, 40, 41, 7] have relatively
better performance than the self-supervised. However, supervised image
denoising requires a large number of noisy-clean image pairs, which are
difficult to collect in practical applications, and generating such image
pairs requires massive human effort and cost. One of the most common ways is
to add simulated real-world noises, such as Additive White Gaussian Noise
(AWGN), to clean images to artificially synthesize noisy images so as to
obtain synthetic noisy-clean pairs [43, 12, 44, 17, 25, 32]. Nevertheless,
there is always an unavoidable gap between the synthetic noise and the real
noise, which severely affects the performance of these supervised models
trained on synthetic noise in the real-world image denoising applications. In
addition, in some cases, it is also difficult to obtain clean images.

In this situation, many self-supervised image denoising methods [15, 45, 3,
18, 19, 23] that do not require noisy-clean image pairs have been proposed.
Noise2Noise [23] used noisy-noisy image pairs to train the model, which
achieved comparable performance to supervised algorithms. But it requires two
perfectly aligned noisy images, which are difficult to obtain in practice.
Noisier2Noise [29] and NAC [38] added the same type of noise as the existing
noise to the original noisy image to form noisier-noisy pairs as the training
set. This requires the model users to know the specific types of the noise in
the image, which is unrealistic in practice because the causes of noise are
diverse and the type of noise can change constantly in real-world. IDR [45]
adopted an iterative approach, taking the noisy images as inputs to the
existing denoising model trained by noisier-noisy pairs, and treating the
output as the next round optimization target to further refine the denoising
model. In this way, the denoising model is optimized by iterations, which can
easily lead to the final denoised image being over-smoothened. Noise2Void [18]
proposed a blind spot network (BSN) denoising method based on the assumption
that pixel signals in the image are spatially correlated in the image, and
noise signals are spatially independent with zero-mean. In recent years,
several publications [18, 20, 13, 37] have shown that BSN is effective in
synthesizing noise for denoising. However, real-world noise is usually
spatially continuous. In most existing BSN denoising methods [3, 18, 19, 22,
13, 37], the masks used to generate blind spots have a single pixel blinded in
the center, which makes it difficult to denoise when the noise correlated area
is large. Zhang et al. [42] combined Transformer and CNN to achieve a trade-
off between denoising images with global spatially correlated noise and
preserving local detail. However, Transformer is computationally intensive,
making it difficult to deploy in practical applications on mobile devices
[36].

Motivated by the fact that different shapes of convolution kernels can extract
different features, we propose a variety of masks with different shapes to
generate blind spots, such as ’+’-shaped mask, ’$\square$’-shaped mask,
’×’-shaped mask, and so on. The multi-masks with different blind spots are
used to mask the surrounding pixels at different positions, so as to destroy
the spatial correlations of the noise in multi-direcion. And we systematically
demonstrate the effectiveness of using different masks or different mask
combinations for image denoising. In addition, we propose an enhanced BSN that
combines with the multi-mask strategy, namely MM-BSN, to more efficiently
integrate multi-mask paths, recover the destroyed textures, and control the
model size. Extensive experiments demonstrate the effectiveness and
superiority of the proposed method.

The main contributions of our work are as follows:

1.To the best of our knowledge, we are the first to explore the combination of
different convolution kernels with multi-mask to extract features, and to
perform denoising on images with large-scale spatially correlated noise in
self-supervised. Furthermore, our multi-mask strategy can be integrated with
other methods.

2\. We propose a novel self-supervised MM-BSN that can integrate the features
extracted by multi-masked convolution kernels, control the model size growth,
and preserve the image detail when denoising.

3\. Our approach achieves the state-of-the-art performance among published
self-supervised sRGB image denoising methods, which is significant for
practical applications.

##  2 Related Work

Supervised image denoising. Zhang et al. [43] first proposed a deep learning
based image denoising method called DnCNN, which trained the model with
generating noisy-clean pairs by manually adding AWGN to clean images.
Subsequently, many researchers proposed other image denoising methods [2, 5,
17, 9, 21, 25, 32, 10] based on deep learning by adding AWGN to clean sRGB
images. However, the denoising performance of these models in the real world
was unsatisfactory due to the large gap between artificial and real-world
noise. Scholars [4, 28] proposed to convert sRGB images to rawRGB first, and
then added Poisson noise corresponding to shot noise and Gaussian noise
corresponding to read noise to rawRGB. After denoising in rawRGB space, the
final denoised result image was converted back to sRGB space using ISP tools.
For this denoising method, accurate noise estimation and modelling was
essential for success. Although the noise obtained by statistical modelling
reduced the gap between the synthetic noise and the real noise, the injected
noise was not real and external factors could alter the accuracy of the noise
modelling. To this end, it was recognized that the most effective way to
denoise was to use the noisy-clean pairs [41, 14, 34, 7] directly from the
real-world when available. However, such a noisy-clean pair dataset requires a
huge amount of human labour to collect and a huge amount of time to construct
in the real world, and was even more impractical given the diverse application
scenarios.

Self-supervised image denoising. Noise2Noise [23] used two perfectly aligned
noisy images from the same scene as input and target, respectively. L2 loss
was used to minimize the difference between the two noisy images in order to
make the model capable of denoising. Then Noise2Void [18], Noise2Self [3],
Probabilistic Noise2Void [19], Neighbor2Neighbor [15], IDR [45], CVF-SID [30],
Blind2Unblind [35] and AP-BSN [22] were proposed to use only noisy images for
training. As the most widely used self-supervised denoising method, BSN was
firstly proposed in Noise2Void [19], which is a special CNN that masks pixel
in the center of the receptive field, and uses the surrounding information to
reconstruct the information of the masked pixels. Its denoising capability is
restricted to the assumption that the noise is spatially independent.
Noise2Void [19] took the masked image as the input and the fully noisy image
as the target to train the model. The masked pixels are not used during
training, which can easily lead to loss of detail and over-smoothing of the
image. Neighbor2Neighbor [15] synthesized two sub-noisy-images by randomly
selecting two adjacent pixels from the 4×4 neighbourhood of the rawRGB image.
Two sub-noisy-images were used as input and target for training, respectively,
forming noisy-noisy pairs. However, training directly on sub-noisy-images
would inevitably lose some image detail. To improve this, Blind2Unblind [35]
used all the pixels for training by generating sub-masked-images with pixels
masked at different positions, and then used a global mask strategy to collect
all pixels from the masked positions in the sub-masked-images after denoising.
Although Blind2Unblind makes full use of all pixel information, it is
difficult to denoise large-noise using only dot-based masks.

Laine19 [20] occluded half of the receptive fields in four different
directions, achieving the effect that the center of the receptive field is not
seen. D-BSN [37] and David et al. [13] used the center-masked convolution
kernel and the dilated convolution layer (DCL) with a specific step size to
construct the BSN. The publications proved that BSN is effective in
synthesizing noise for denoising. However, the real-world noise is usually
spatially continuous and BSNs would fail to handle it. To break the spatial
correlation of real-world noise, AP-BSN [22] adopted Pixel-shuffle
Downsampling (PD) with 5-pixel stride on images before training, and utilized
center-masked convolution kernel and dilated convolution layer (DCL) to
achieve the effect of blind spots during training. However, AP-BSN relies on
the PD with limited stride to break the spatial correlation of the noise. If
large-noise exists in the image, blindly increasing the PD stride will cause
irreversible damage to image details [22]. Therefore, it is challenging for
AP-BSN to strike a balance between the noise removal and texture information
preservation, especially when denoising large-noise. In this paper, we propose
a joint feature-extraction method using multi-masked convolutional kernels to
destroy large-noise correlations. We also propose a novel architecture that
combines the multi-mask convolutional kernels with BSN (MM-BSN) to make full
use of the extracted features and preserve texture structures of the original
image as much as possible.

##  3 Motivation

We explore the noise spatially correlations that have different shapes as
shown in Figure 2. The sub-images in Figure 2(a) all have a size of
height×width as 10×10, which shows that the correlation area is large. We also
computes the proportion of spatially correlated noise in different areas of
the image in Figure 2(b). We define the spatially correlated noise with a area
bigger than 25 as large-noise. Figure 2(b) shows that the large-noise, which
theoretically cannot be handled by PD stride not bigger than 5, occupies more
than 1/3.

Recently published BSN methods, either the mask in the input [15, 3, 18, 19,
35] or the mask in the network [22, 13, 37], which used a dot-based mask, is
not enough to break the correlation of large-noise. In this way, the blind
pixels recovered from the surrounding information would still contain noise.
Motivated by the prior knowledge that filters with different shapes can be
designed to target different types of noise, such as ’+’, ’$\square$’, etc.,
we propose a novel multi-mask strategy, which ultilizes different convolution
kernels masked in different shapes to further destroy the spatial connection
of noise.

(a) Spatially correlated noise shown by the black area in different shapes.

(b) Proportion of different noise area on a full image

Figure 2: Noise detail on the 0228 N.png from SIDD validation.

##  4 Main Method

(a) o

(b) —

(c) +

(d) /

(e) ×

(f) $\square$

(g) $|$

(h) $\boxplus$

(i) \

(j) $\divideontimes$

Figure 3:  Multi-mask is shown on 5×5 kernel. Gray dots represent 0, and blue
dots represent 1. (a) is the central mask with a single blind spot and (f) is
a ’$\square$’-shaped mask. (b) is a ’—’-shaped mask and (g) is a ’$|$’-shaped
mask. (c) is a ’+’-shaped mask and (h) is a ’$\boxplus$’-shaped mask. (d) is a
’/’-shaped mask and (i) is a ’\\-shaped mask. (e) is a ’×’-shaped mask and (j)
is a ’$\divideontimes$’-shaped mask.

Multi-Mask Strategy. We propose to use the multi-mask strategy to further
destroy the spatial connection of the noise, while preserving useful texture
information of the image. Figure 3 shows the shapes of different masks when
the convolution kernel size is 5×5, such as ’+’, ’$\square$’, ’—’, ’$|$’, ’/’
, ’`\`’, ’×’, etc.

Theoretically, we can arbitrarily combine multiple masks of different shapes
to achieve different denoising results. When using $n$ number of types of
masked convolutional kernels types, the same operations are performed for each
path until the final concatenation, where all features extracted by several
different masked convolutional kernels are fused together. In this way, we can
obtain a number of basis multi-mask BSN models, whose architectures contain
multiple branches corresponding to the number of masks. However, the model
size obtained by this naive method of simple stacking is almost $n$ times the
size of the basic network. Consequently, the workload on the hardware device
is multiplied by $n$. To control the model size, make full use of the
information around the blind spot and avoid information redundancy, we
generally use a combination of only two masks. The feature extracted by
’o’-shaped mask contains complete information. However, it may contain more
unconducive information for denoising because it could not break the spatial
connection of the noise sufficiently. The other types of masks mask more
pixels of the surrounding pixels, which can break the spatial connection of
the noise more, but lose more image information. So we can combine the feature
extracted by ’o’ to provide more detail and the other shape of masks to break
the spatial correlation of the noise and reinforce each other to get a better
denoising performance. Of course, two masks with complementary mask shapes
also can break the spatial connection of the noise while extracting
information from the surrounding pixels. Multi-mask combinations can be
flexibly adjusted according to the real noise distribution.

Our multi-mask strategy can be integrated with other methods by simply
stacking the different mask paths. However, in this way, the increasing number
of different mask types will explode the model size. In addition, the features
extracted by different masks have no interaction between the processing paths
at the intermediate stages before the final concatenation. Without such
interaction, information transfer and co-optimization between these processing
paths is not possible. Therefore, how to use multi-mask to destroy the spatial
connection of noise while retaining more texture information is also a
challenge. Last but not the least, as the mask area increases, the texture
information of the image itself is increasingly destroyed. Finally, we propose
a novel MM-BSN, to address these challenges.

Figure 4: MM-BSN Architecture. The channel of feature maps that are not marked
is 128. N indicates that DCL repeats N times.

MM-BSN Architecture. MM-BSN is initially motivated by AP-BSN [22]. We also use
masked convolutional kernels to extract the shallow features. But instead of
using only the center mask, we add other shapes of masks to extract the masked
features.

The architecture of MM-BSN is shown in Figure 4. The workflow consists of four
steps. First, a linear transformation is performed on the noisy image with a
1×1 convolutional layer, and the output feature containing the complete image
information passes through several different masked convolutional layers in
parallel. Second, each masked feature passes through three layers in parallel,
two 1×1 convolutional layers and a Concatenation-based Dilated Convolutional
Layer (CDCL). CDCL contains a small number of DCLs (set to 2 in this article)
and its output features are combined with one linearly transformed feature
from a 1×1 convolutional layer using a concatenation according to the mask
size. The features extracted by the same size but different masked convolution
kernels are fused together. Third, after passing through several DCLs (set to
7 in this article), all features are concatenated together, and the features
extracted by different masked convolution kernels of different sizes are
fused. Finally, the output is obtained by channel transformation and feature
fusion with several 1×1 convolutional layers.

Due to the interaction between different feature pathways, the resulting MM-
BSN parameter set of 5.3M is larger than AP-BSN of 3.7M, but much smaller than
the model size of a simple stack of AP-BSN with multi-mask (namely SMM-BSN) of
7.3M. The ablation experiments of several models are detailed in Section 9.

Loss Chosen. In this paper, we use L1 loss function to train our MM-BSN:

$E=\|I_{out}-I_{N}\|_{1}$ (1) $I_{out}=PD^{-1}(M(PD(I_{N})))$ (2)

Where $M$ denotes the MM-BSN model, $I_{out}$ is the result of $PD^{-1}$, and
$I_{N}$ is the noisy input. Similar to AP-BSN [22], we use PD to preliminarily
break the spatial connection between the noises of adjacent pixels. After PD
with stride $S_{pd}$, we obtain a group of small sub-images that are inputs to
MM-BSN. The denoised result $I_{out}$, which has the same size as the original
image, is decoded by operating $PD^{-1}$ to the outputs of the model.

##  5 Experiments

###  5.1 Implementation Details

Datasets. We take the public datasets of SIDD [1] and DND [33] for our
experiments. We take the noisy sRGB images in SIDD Medium dataset that
contains 320 pairs of noisy-clean images as the training set and SIDD
validation as the valid set, respectively. SIDD validation and SIDD benchmark
can be used as test sets. DND dataset that contains only 50 noisy image are
generally used as the test dataset. Since only noisy images are needed to
train our self-supervised models, we use DND as both the training set and the
test set.

Training Details. All models are trained with the same hyperparameters. The
batch size is 8 and the number of training epochs is 30. The optimization
function adopted is Adam. The initial learning rate is 0.0001, and the
learning rate of every 8 epochs is multiplied by 0.1. The images are resized
to 128×128, and are randomly rotated within a range of 90° in the horizontal
or vertical direction before training. All experiments are run on a server
with python 3.8.0, pytorch1.12.0, and Nvidia Tesla T4 GPUs. For a relatively
fair comparison, unless otherwise stated, we set the PD stride as 5 for
training, 2 for testing, and the same post-processing as AP-BSN [22].

| Mask | SIDD Validation | SIDD Benchmark
---|---|---|---
$S_{pd}$=2 | ’o’ | 24.27/0.361 | 27.48/0.627
’$\square$’ | 35.29/0.854 | 36.84/0.932
Table 1: Quantitative comparison of the same network using different masks
with $S_{pd}$=2. PSNR/SSIM results are calculated between the denoised-clean
pairs using the Python toolkit for SIDD validation, and the official toolkit
for SIDD benchmark.

(a) Noisy

(b) $S_{pd}$=2, ’o’

(c) $S_{pd}$=5, ’o’

(d) $S_{pd}$=2, ’$\square$’

Figure 5: Visualization performance of several models with the same
architecture but different mask type or different $S_{pd}$. (a) Noisy image.
(b)With a small stride factor $S_{pd}$=2 and center mask, the method cannot
remove noise from noisy image. (c) With $S_{pd}$=5 and ’o’-shaped mask, the
model can denoise better. (d) With $S_{pd}$=2 and ’$\square$’-shaped mask, the
model can get a better performance than that with ’o’-shaped mask.

(a) Clean

(b) Noisy

(c) AP-BSN

(d) ’o’ + ’+’

(e) ’o’ + ’$\boxplus$’ + ’+’

Figure 6: Qualitative comparison of several methods with different mask
combinations on SIDD validation dataset. (a) Clean images. (b) Noisy images.
(c) Denoised images by AP-BSN [22]. (d) Denoised images by SMM-BSN using a
combination of ’o’ mask and ’+’ mask. (e) Denoised images by SMM-BSN using a
combination of ’o’, ’$\boxplus$’ and ’+’-shaped mask.

###  5.2 Analyzing Multi-Mask strategy in BSN

To compare the performance of the proposed method with different mask
combinations, we trained several MM-BSN models with different masks on SIDD
Medium dataset. All trained models are quantitatively evaluated on SIDD
validation and benchmark. Existing Python toolkits are used to compute the
PSNR/SSIM of SIDD validation. At the same time, we upload the denoised results
of SIDD benchmark to the official website and obtain the reported PSNR/SSIM.

Significant effect on breaking the noise structure. To check the effectiveness
of the mask in breaking the structure of large-noise, we take the image after
PD with $S_{pd}$=2 as input to train. Figure 5 shows the denoising performance
of the models with different $S_{pd}$s or masks but the same other settings.
AP-BSN [22] shows that when $S_{pd}$=2, the spatial connection of the noise in
the image cannot be broken well. Using only the center mask, the model is
weakly able to denoise, as shown in Figure 5(b). But if we use the ’$\square$’
mask when $S_{pd}$=2, the denoised result is even better than the ’o’-shaped
masked model with $S_{pd}$=5 [22] as shown in Figure 5(c) and 5(d). Table 1
quantitatively shows that, the PSNR/SSIM on the SIDD validation and benchmark
datasets are greatly improved when the center mask is replaced with the
’$\square$’ mask when $S_{pd}$=2. This proves that the large-noise structure
can be better broken by the ’$\square$’ mask, and it is not only an exception
shown in Figure 5, but also a common case.

Quantitative comparison of MM-BSN models. We summarize the following points
from Table 2: (1) Different mask combinations achieve different denoising
performance. The reason is that different masks target different noise
correlations, and it is common sense that the final denoised result will be
different. (2) The mask combinations combined with ’o’ are overall better than
the combinations without it, due to that the feature extracted by the
’o’-shaped mask preserves the texture information of the image itself more
completely. (3) The combination of ’/’ and ’`\`’ gives the best performance,
followed by the combination of ’o’ and ’/’, which indicates that the dataset
has more ’/’ and ’`\`’ shaped spatially related noise. For different datasets,
the noise structures are different, and users can freely choose the
combination of masks or design the mask shape suitable for the real dataset
according to the needs.

Masks | Test datasets
---|---
o | — | $|$ | $\boxplus$ | + | / | `\` | $\divideontimes$ | × | Validation | Benchmark
✓ | ✓ |  |  |  |  |  |  |  | 37.34/0.881 | 37.31/0.937
✓ |  | ✓ |  |  |  |  |  |  | 37.30/0.881 | 37.30/0.937
✓ |  |  | ✓ |  |  |  |  |  | 37.32/0.882 | 37.31/0.936
✓ |  |  |  | ✓ |  |  |  |  | 37.28/0.879 | 37.28/0.936
✓ |  |  |  |  | ✓ |  |  |  |  37.37/0.882 | 37.35/0.936
✓ |  |  |  |  |  | ✓ |  |  | 37.18/0.878 | 37.18/0.935
✓ |  |  |  |  |  |  | ✓ |  | 37.24/0.881 | 37.23/0.934
✓ |  |  |  |  |  |  |  | ✓ | 37.24/0.881 | 37.24/0.934
| ✓ | ✓ |  |  |  |  |  |  | 37.12/0.879 | 37.12/0.934
|  |  | ✓ | ✓ |  |  |  |  | 37.19/0.880 | 37.18/0.934
|  |  |  |  | ✓ | ✓ |  |  | 37.38/0.882 |  37.37/0.936
|  |  |  |  |  |  | ✓ | ✓ | 37.11/0.879 | 37.11/0.933
Table 2: Quantitative comparison of MM-BSN with different mask combinations on
SIDD validation and benchmark datasets with PSNR/SSIM.

Comparison of BSN models with increasing mask types. Figure 6 shows the
qualitative denoising performance of models with different number of mask
types in the same framework on the SIDD validation dataset. Comparing Figure
6(c), 6(d) and 6(e), it can be observed that by adding other types of masks
based on the ’o’-shaped mask, the denoising performance is significantly
improved. Especially in the second row, the denoised result of AP-BSN has
unacceptable color shifts, while SMM-BSN restores the original color
perfectly, indicating that adding masks of other shapes can effectively
destroy the noise correlation during feature extraction. In addition, it can
be seen from the second row of Figure 6(b) and Figure 6(c) that the PSNR value
after denoising decreases from 22.62dB to 21.69dB when only the center mask is
used, but it increases to 39.84dB/39.66dB when the multi-mask is used. This
indicates that when the spatially correlated noise region is large, the center
mask alone cannot break the noise structure sufficiently. Since the features
extracted by the center mask alone may still be noisy, the final denoising
result will be biased by massive noise. Figure 6(d) and 6(e) show that
increasing the number of mask types does not always improve the denoising
performance. The possible reason for this is that features extracted by
increasing types of masks lead to the information redundancy, which is
unsensive and unuseful for denoising.

Models | SIDD Validation | Parameters(M)
---|---|---
AP-BSN | 35.91/0.870 | 3.7
SMM-BSN | 37.16/0.879 | 7.3
MM-BSN | 37.38/0.882 | 5.3
Table 3: Comparison of Several BSNs. All BSNs are trained on SIDD Medium
dataset. PSNR/SSIM results for SIDD validation and the model size are shown
here.

(a) Noisy

(b) AP-BSN [22]

(c) MM-BSN

Figure 7: Visual comparison of AP-BSN and MM-BSN on the SIDD benchmark. They are trained on SIDD Medium dataset using the same center mask. (a) Noisy image. (b) Denoised result by AP-BSN [22]. (c) Denoised result by MM-BSN. Figure 8: Comparison of denoising results of AP-BSN and MM-BSN on the noisy sets of SIDD Validation. | Method | SIDD | DND
---|---|---|---
PSNR | SSIM | PSNR | SSIM
Non-learning based | BM3D [8] | 25.65 | 0.685 | 34.51 | 0.851
WNNM [11] | 25.78 | 0.809 | 34.67 | 0.865
Supervised Synthetic pairs | DnCNN [43] | 23.66 | 0.583 | 32.43 | 0.790
CBDNet [12] | 33.28 | 0.868 | 38.05 | 0.942
Supervised Real pairs | DnCNN [43] | 36.07$\diamond$ | 0.911$\diamond$ | 37.81$\diamond$ | 0.931$\diamond$
DnCNN [43] | 36.07 | 0.911 | 37.81 | 0.931
AINDNet(R)$\ast$[17] | 38.84 | 0.951 | 39.34 | 0.952
VDN [40] | 39.26 | 0.955 | 39.38 | 0.952
| NAFNet [7] | 40.30 | 0.962 | - | -
Unsupervised Unpaired | GCBD [6] | - | - | 35.58 | 0.922
C2N [16] \+ DIDN$\ast$ [39] | 35.35 | 0.937 | 37.28 | 0.924
D-BSN [37] \+ MWCNN [26] | - | - | 37.93 | 0.937
Self-supervised | Noise2Void [18] | 27.68R | 0.668R | - | -
Noise2Self [3] | 29.56R | 0.808R | - | -
NAC [38] | - | - | 36.20 | 0.925
R2R [31] | 34.78 | 0.898 | - | -
CVF-SID (S2) [30] | 34.71 | 0.917 | 36.50 | 0.924
AP-BSN [22] | 35.97 | 0.925 | 38.09 | 0.937
AP-BSN$\dagger$ [22] | 36.91 | 0.931 | - | -
MM-BSN(Ours) | 37.37 | 0.936 | 38.46 | 0.940
MM-BSN$\dagger$(Ours) | - | - | 38.74 | 0.943
Table 4: Quantitative comparison of different denoising models on SIDD and DND
benchmarks. By default, we get the official evaluation results from SIDD and
DND benchmark websites. $\diamond$ indicates that we have retrained the model,
uploaded the test results and received the results. R indicates that the
result is reported by R2R [31]. $\ast$ denotes the method with self-ensemble
strategy [24]. $\dagger$ denotes the model trained with the same training and
test data sets. The highest value is highlighted in bold for each type of
denoising model.

###  5.3 Analyzing our network architecture

For fairly comparing, all models are trained on SIDD Medium dataset and
evaluated on SIDD validation. AP-BSN [22] uses only the center mask, SMM-BSN
and MM-BSN use the combination of ’/’-shaped mask and ’`\`’-shaped mask for
training, and other settings are the same as before.

(a) DnCNN [43]
Supervised-Real SIDD

(b) C2N[16]+DIDN$\ast$[39]
Unpaired

(c) CBDNet [12]
Supervised-Synthetic noise

(d) AP-BSN [22]
Self-supervised

(e) MM-BSN
Self-supervised

Figure 9: Qualitative comparison between different denoising methods on SIDD
and DND benchmarks. (a) DnCNN is trained on the real paired SIDD Medium
dataset. (b) C2N generates a realistic noisy image from the clean input, where
the following denoising model, i.e., DIDN, is trained on the generated pairs.
(c) CBDNet is trained in a supervised manner using noisy-clean pairs, where
the noisy image is obtained by adding synthetic noise to the clean image.
(d-e) The methods are trained directly on real sRGB images. Note that the DND
benchmark (upper) provides some per-sample PSNR/SSIMs, while SIDD benchmark
(lower) does not.

Table 3 compares AP-BSN and its corresponding extended versions SMM-BSN, which
indicates that the denoising performance is significantly improved by applying
the multi-mask strategy. The PSNR/SSIM of SIDD validation shows that MM-BSN
(37.38/0.882) outperforms AP-BSN (35.91/0.870) by a large margin. Figure 7(b)
shows that using AP-BSN, the alphabets in the yellow box of the image are
blurred and a lot of detail is lost. Figure 7(c) shows that the alphabets in
the yellow box are generally preserved. This observation suggests that by
adding concatenation-based skip-connections from the shallow features in MM-
BSN, the lost detail can be supplemented in time.

We classify images with different large-noise ratios for SIDD validation and
calculate the average PSNR of AP-BSN and MM-BSN in each image set, as shown in
Figure 8. Our MM-BSN with multi-mask strategy outperforms AP-BSN with only ’o’
masks by a large magin, with PSNR improvements of up to 4, particularly in
large-noise.

###  5.4 MM-BSN in real-world sRGB image denosing

The proposed MM-BSN aims to denoise large-noise in sRGB images by combining
multi-mask in the self-supervised manner, while preserving the texture detail
and controlling the model size.

Table 4 quantitatively compares the denoising performance of several
traditional algorithms, supervised denoising algorithms, unsupervised and
self-supervised algorithms on SSID and DND benchmarks. The table shows that
MM-BSN performs best in self-supervised methods and even outperforms some
supervised algorithms. Furthermore, our MM-BSN does not require rawRGB images
and noise estimation like R2R, nor real noisy-clean pairs like supervised
models. Therefore, in practical applications, researchers can train MM-BSN
directly on the noisy images from the target scene for denoising, avoiding
degradation of the model performance when the scenario changes.

Figure 9 qualitatively compares the visual denoising performance of state-of-
the-art models on a random image in SIDD and DND benchmarks. Compared with its
yellow box in the upper images, the lines area denoised by self-supervised
models in Figure 9(d) and Figure 9(e) are more smoothing, while there are
unwanted but obvious burring effects near the lines denoised by other models
shown in Figure 9(a), Figure 9(b) and Figure 9(c). MM-BSN performs better than
most of the models, and can even compete with the supervised method of CBDNet
[12] with slightly lower PSNR/SSIM. Comparing the yellow box in the lower
images, our MM-BSN has a more clear boundary contour of outer boundary of the
alphabets, and the noise on the alphabets themselves is more obviously
reduced.

##  6 Conclusion

In this paper, we propose a multi-mask strategy worked on BSNs for self-
supervised sRGB image denoising. Multi-mask can significantly break the large-
noise structure, which previously cannot be efficiently handled by the only-
center-masked models. In addition, we develop MM-BSN to effectively combine
the features extracted by multi-masked convolutional layers and control the
model size to grow without explosion. In particular, the utilization of
concatenation-based skip-connections can help to compensate for the loss of
information caused by the masks. Extensive experiments prove that our method
can effectively denoise the images with a large scale spatially correlated
noise and can preserve more textures, achieving a better denoising performance
than other unsupervised and self-supervised methods in the literature. Our
proposed MM-BSN is well suited for a real practical application scenario
considering that it only needs noisy sRGB images to train.

## References

  * [1] Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown.  A high-quality denoising dataset for smartphone cameras.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1692–1700, 2018.
  * [2] Saeed Anwar and Nick Barnes.  Real image denoising with feature attention.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 3155–3164, 2019.
  * [3] Joshua Batson and Loic Royer.  Noise2self: Blind denoising by self-supervision.  In International Conference on Machine Learning, pages 524–533. PMLR, 2019.
  * [4] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron.  Unprocessing images for learned raw denoising.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11036–11045, 2019.
  * [5] Meng Chang, Qi Li, Huajun Feng, and Zhihai Xu.  Spatial-adaptive network for single image denoising.  In European Conference on Computer Vision, pages 171–187. Springer, 2020.
  * [6] Jingwen Chen, Jiawei Chen, Hongyang Chao, and Ming Yang.  Image blind denoising with generative adversarial network based noise modeling.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3155–3164, 2018.
  * [7] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun.  Simple baselines for image restoration.  arXiv preprint arXiv:2204.04676, 2022.
  * [8] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Image denoising by sparse 3-d transform-domain collaborative filtering.  IEEE Transactions on image processing, 16(8):2080–2095, 2007.
  * [9] Faming Fang, Juncheng Li, Yiting Yuan, Tieyong Zeng, and Guixu Zhang.  Multilevel edge features guided network for image denoising.  IEEE Transactions on Neural Networks and Learning Systems, 32(9):3956–3970, 2020.
  * [10] Shuhang Gu, Yawei Li, Luc Van Gool, and Radu Timofte.  Self-guided network for fast image denoising.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2511–2520, 2019.
  * [11] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng.  Weighted nuclear norm minimization with application to image denoising.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2862–2869, 2014.
  * [12] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Toward convolutional blind denoising of real photographs.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1712–1722, 2019.
  * [13] David Honzátko, Siavash A Bigdeli, Engin Türetken, and L Andrea Dunbar.  Efficient blind-spot neural network architecture for image denoising.  In 2020 7th Swiss Conference on Data Science (SDS), pages 59–60. IEEE, 2020.
  * [14] Xiaowan Hu, Ruijun Ma, Zhihong Liu, Yuanhao Cai, Xiaole Zhao, Yulun Zhang, and Haoqian Wang.  Pseudo 3d auto-correlation network for real image denoising.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16175–16184, 2021.
  * [15] Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu.  Neighbor2neighbor: Self-supervised denoising from single noisy images.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14781–14790, 2021.
  * [16] Geonwoon Jang, Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee.  C2n: Practical generative noise modeling for real-world denoising.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2350–2359, 2021.
  * [17] Yoonsik Kim, Jae Woong Soh, Gu Yong Park, and Nam Ik Cho.  Transfer learning from synthetic to real-noise denoising with adaptive instance normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3482–3492, 2020.
  * [18] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug.  Noise2void-learning denoising from single noisy images.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2129–2137, 2019.
  * [19] Alexander Krull, Tomáš Vičar, Mangal Prakash, Manan Lalit, and Florian Jug.  Probabilistic noise2void: Unsupervised content-aware denoising.  Frontiers in Computer Science, 2:5, 2020.
  * [20] Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila.  High-quality self-supervised deep image denoising.  Advances in Neural Information Processing Systems, 32, 2019.
  * [21] Rushi Lan, Haizhang Zou, Cheng Pang, Yanru Zhong, Zhenbing Liu, and Xiaonan Luo.  Image denoising via deep residual convolutional neural networks.  Signal, Image and Video Processing, 15(1):1–8, 2021.
  * [22] Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee.  Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17725–17734, 2022.
  * [23] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila.  Noise2noise: Learning image restoration without clean data.  arXiv preprint arXiv:1803.04189, 2018.
  * [24] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136–144, 2017.
  * [25] Pengju Liu, Hongzhi Zhang, Wei Lian, and Wangmeng Zuo.  Multi-level wavelet convolutional neural networks.  IEEE Access, 7:74973–74985, 2019.
  * [26] Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo.  Multi-level wavelet-cnn for image restoration.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 773–782, 2018.
  * [27] Enming Luo, Stanley H Chan, and Truong Q Nguyen.  Adaptive image denoising by targeted databases.  IEEE transactions on image processing, 24(7):2167–2181, 2015.
  * [28] Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll.  Burst denoising with kernel prediction networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2502–2510, 2018.
  * [29] Nick Moran, Dan Schmidt, Yu Zhong, and Patrick Coady.  Noisier2noise: Learning to denoise from unpaired noisy data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12064–12072, 2020.
  * [30] Reyhaneh Neshatavar, Mohsen Yavartanoo, Sanghyun Son, and Kyoung Mu Lee.  Cvf-sid: Cyclic multi-variate function for self-supervised image denoising by disentangling noise from image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17583–17591, 2022.
  * [31] Tongyao Pang, Huan Zheng, Yuhui Quan, and Hui Ji.  Recorrupted-to-recorrupted: unsupervised deep learning for image denoising.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2043–2052, 2021.
  * [32] Bumjun Park, Songhyun Yu, and Jechang Jeong.  Densely connected hierarchical network for image denoising.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0, 2019.
  * [33] Tobias Plotz and Stefan Roth.  Benchmarking denoising algorithms with real photographs.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1586–1595, 2017.
  * [34] SMA Sharif, Rizwan Ali Naqvi, and Mithun Biswas.  Learning medical image denoising with deep dynamic residual attention network.  Mathematics, 8(12):2192, 2020.
  * [35] Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han.  Blind2unblind: Self-supervised image denoising with visible blind spots.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2027–2036, 2022.
  * [36] Fei Wen, Mian Qin, Paul Gratz, and Narasimha Reddy.  Software hint-driven data management for hybrid memory in mobile systems.  ACM Transactions on Embedded Computing Systems (TECS), 21(1):1–18, 2022.
  * [37] Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo.  Unpaired learning of deep image denoising.  In European conference on computer vision, pages 352–368. Springer, 2020.
  * [38] Jun Xu, Yuan Huang, Ming-Ming Cheng, Li Liu, Fan Zhu, Zhou Xu, and Ling Shao.  Noisy-as-clean: Learning self-supervised denoising from corrupted image.  IEEE Transactions on Image Processing, 29:9316–9329, 2020.
  * [39] Songhyun Yu, Bumjun Park, and Jechang Jeong.  Deep iterative down-up cnn for image denoising.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [40] Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang.  Variational denoising network: Toward blind noise modeling and removal.  Advances in neural information processing systems, 32, 2019.
  * [41] Zongsheng Yue, Qian Zhao, Lei Zhang, and Deyu Meng.  Dual adversarial network: Toward real-world noise removal and noise generation.  In European Conference on Computer Vision, pages 41–58. Springer, 2020.
  * [42] Dan Zhang and Fangfang Zhou.  Self-supervised image denoising for real-world images with context-aware transformer.  IEEE Access, 2023.
  * [43] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  IEEE transactions on image processing, 26(7):3142–3155, 2017.
  * [44] Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Ffdnet: Toward a fast and flexible solution for cnn-based image denoising.  IEEE Transactions on Image Processing, 27(9):4608–4622, 2018.
  * [45] Yi Zhang, Dasong Li, Ka Lung Law, Xiaogang Wang, Hongwei Qin, and Hongsheng Li.  Idr: Self-supervised image denoising via iterative data refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2098–2107, 2022.