# An AI-driven Intelligent Traffic Management Model for 6G Cloud Radio Access
Networks

Smruti Rekha Swain, Deepika Saxena, Jatinder Kumar, Ashutosh Kumar Singh, ,
and Chung-Nan Lee, Member, IEEE S. R. Swain, J. Kumar, and A. K. Singh are
with the Department of Computer Applications, National Institute of
Technology, Kurukshetra, India. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
D. Saxena is with Department of Computer Science, Goethe University Frankfurt,
Germany. E-mail<EMAIL_ADDRESS>
C. N. Lee is with the Department of Computer Science and Engineering, National
Sun Yat-Sen University, Kaohsiung, Taiwan. E-mail<EMAIL_ADDRESS>
###### Abstract

This letter proposes a novel Cloud Radio Access Network (C-RAN) traffic
analysis and management model that estimates probable RAN traffic congestion
and mitigate its effect by adopting a suitable handling mechanism. A
computation approach is introduced to classify heterogeneous RAN traffic into
distinct traffic states based on bandwidth consumption and execution time of
various job requests. Further, a cloud-based traffic management is employed to
schedule and allocate resources among user job requests according to the
associated traffic states to minimize latency and maximize bandwidth
utilization. The experimental evaluation and comparison of the proposed model
with state-of-the-art methods reveal that it is effective in minimizing the
worse effect of traffic congestion and improves bandwidth utilization and
reduces job execution latency up to 17.07% and 18%, respectively.

###### Index Terms:

C-RAN, Congestion, Stragglers, Network Hogs.

##  I Introduction

Booming C-RAN traffic (i.e., RAN traffic based on a centralized cloud
computing used in the deployment of 5G networks to provide improved cellular
coverage, capacity, and reliability) has become inevitable with the emerging
technologies, including 6G wireless communication networks, Internet of Things
(IoT), and unmanned aerial vehicle (UAV) assisted networks, etc [1]. According
to a recent report, the C-RAN market size is projected to grow from USD 5.5
billion in 2022 to USD 43.2 billion by 2032, at a Compound Annual Growth Rate
(CAGR) of 22.9% during the forecast period
111https://www.factmr.com/report/cloud-radio-access-network-market. C-RAN
allows virtualization of networks and customize them to cater the specific
needs of applications, services, devices, and customers [2]. However, the
diversity and complexity of 6G C-RAN environment impose a critical and
challenging issue of wireless traffic management because of conflicting
perspectives of customers and service provider where, the former aspires for
highest service quality with least latency and the latter intends minimize
operational cost by maximizing bandwidth utilization. In this context, a novel
Traffic Management model for 6G Cloud Radio Access Network (TM-CRN) is
proposed that examines the heterogeneous RAN traffic for probable network
congestion and filters the contention-suspected traffic from the normal
traffic to facilitate appropriate management. Firstly, the RAN traffic is
estimated and analysed proactively for classification of live job requests
into distinct traffic states. Thereafter, cloud-based traffic management is
employed which triggers the allocation of job requests confined to various
traffic states according to their criticality and resource requirement such
that job execution latency is minimized and resource (viz., CPU, memory, and
bandwidth) is maximized. The key contributions are: (i) A RAN traffic analyser
is introduced that estimates the respective traffic proactively to detect and
mitigate the probable congestion while reserving sufficient capacity of
physical resources. (ii) A novel approach of RAN traffic states
classification, is determined for resource allocation, scheduling, and mapping
accordingly. (iii) TM-CRN model is implemented and the results are verified
against QoS metrics, and compared with state-of-the-art approaches.

##  II TM-CRN Model

Consider a wireless telecommunications environment such as Open-RAN, GSM-RAN
or C-RAN [3] composed of antennas, radios, and baseband units (BBU)s:
{$RAN_{1}$, $RAN_{2}$, …, $RAN_{n}$}$\in\mathds{RAN}$ producing heterogeneous
network traffic {$Nt_{1}$, $Nt_{2}$, …, $Nt_{M}$}$\in\mathds{NT}$ as
illustrated in Fig. 1. In C-RAN, BBU devices convert digital signals into
radio transmissions and vice-versa, are acting as a centralized control and
processing station which are connected to remotely located Radio Frequency
Units (RFU, the broadcasting antenna of a base station) via high speed optical
fiber.

Figure 1: TM-CRN Architectural Design

The traffic includes job requests varying over execution time and resource
demands such as bandwidth ($BW$), CPU ($C$), and memory ($Mem$), are
transferred to cloud platform or BBU for different purposes including
computation, storage, data sharing, etc., via RAN-Cloud Interface Layer. The
cloud platform consists of three co-operative operational units, including RAN
Traffic Analyser Unit (RTAU), Traffic Management Unit (TMU), and Cloud
Services Unit (CSU). The radio network traffic ($\mathds{NT}$) is passed to
RTAU for probable traffic congestion identification and estimation using the
knowledge generated by AI-driven RAN Traffic Predictor i.e., {$Pt_{1}$,
$Pt_{2}$, …, $Pt_{M}$} and pre-estimated threshold of traffic deviation.
Accordingly, the estimated traffic outcome is transferred to RAN Traffic
Congestion Handling Mechanism which triggers the essential operations to
manage it as depicted in TMU block. Within TMU, $\mathds{NT}$ is segregated
into distinct traffic states such as Fast jobs, Stragglers, Network Hogs, and
Heavy jobs, etc., depending up on the amount of execution time and bandwidth
consumption. The most appropriate single or multiple scheduling and mapping
strategies for the live job requests are decided among the $Z$ resource
allocation strategies, and applied as per the fulfilment of essential resource
availability and latency criteria. Further, TMU explores the selected resource
allocation strategies for parallel execution to minimize the resource wastage,
execution time and reduce CPU idle time. Correspondingly, the job requests are
executed by CSU comprising of Storage, Computing, and Networking services. The
cloud service execution information like, resource ($C$, $BW$, $Mem$) usage,
number and type of job requests per time-interval is utilized to generate
training data samples for the learning process of RAN traffic predictor. The
operational description of RTAU and TMU with CSU is given in the Section III
and Section IV, respectively.

##  III RAN Traffic Analysis

As illustrated in Fig. 1, RTAU analyses approaching traffic using RAN traffic
predictor (RTP) which is based on the AI-driven Extreme-Gradient Boosting
(XGB) algorithm. This predictor is capable of learning and developing the
intuitive and precise correlations among extracted samples or patterns. Let
the RTP is composed of $\mathchorus{l}$ base learners i.e., decision trees
$\mathchorus{BL}^{\ast}=\\{\mathchorus{BL}^{\ast}_{1},\mathchorus{BL}^{\ast}_{2},...,\mathchorus{BL}^{\ast}_{l}\\}$
which estimate $\mathchorus{l}$ respective outcomes
$\mathchorus{O}^{\ast}=\\{\mathchorus{O}^{\ast}_{1},\mathchorus{O}^{\ast}_{2},...,\mathchorus{O}^{\ast}_{l}\\}$
using Eq. (1);

$\mathchorus{O}^{\ast}=\sum_{z=1}^{l}\mathchorus{BL}_{z}^{\ast}(\mathchorus{F}_{i})\quad\forall
i\in\\{1,2,...,s\\}$ (1)

where $\mathchorus{F}$ represents the input vector of size $s$, representing
$s$ training attributes (such as bandwidth, memory, processing usage samples)
for traffic prediction. During each iteration, decision trees are trained
incrementally to reduce prediction errors and the amount of error reduction is
computed as gain or loss term
($L(\mathchorus{O}^{\ast},{\mathchorus{O}^{\ast\ast}_{t-1}}+{\mathchorus{BL}^{\ast}_{t}}(\mathchorus{F}_{i}))$).
Concurrently, a regularisation term ($\Psi(\mathchorus{BL}^{\ast}_{t})$) is
computed at each iteration to measure the complexity of decision trees and
control their pruning in optimized way while preventing overfitting.
Accordingly, the objective function ${L}$ composed of loss term and
regularisation term is minimized using Eq. (2).

${L}_{t}=\sum_{i=1}^{s}{L(\mathchorus{O}^{\ast},{\mathchorus{O}^{\ast\ast}_{t-1}}+{\mathchorus{BL}^{\ast}_{t}}(\mathchorus{F}_{i}))+\sum_{z=1}^{l}\Psi(\mathchorus{BL_{z}}^{\ast})}$
(2)

The term $\Psi(\mathchorus{BL}^{\ast}_{t})$ is calculated using Eq. (3), where
$\gamma$ and $\lambda$ are $L_{1}$ and $L_{2}$ regularisation coefficients,
respectively, $w$ is internal split tree weight and $K$ is number of leaves in
tree.

$\Psi(\mathchorus{BL}^{\ast}_{t})=\gamma K+\dfrac{1}{2}\lambda||w||^{2}$ (3)

Thereafter, adaptive boosting or optimization is applied using Taylor
expansion which calculates exact loss for each decision tree by transforming
Eq. (2) into Eq. (4);

$\displaystyle{L}_{t}=\sum_{i=1}^{s^{\ast}}{[g_{i}\mathchorus{BL}^{\ast}_{t}(\mathchorus{F}_{i})+\frac{1}{2}h_{i}{{\mathchorus{BL}^{\ast}_{t}}}^{2}(\mathchorus{F}_{i})]+\Psi(\mathchorus{BL}^{\ast}_{t})}$
(4)

where
$g_{i}=\partial_{\mathchorus{O}^{\ast\ast}_{t-1}}{L(\mathchorus{O}^{\ast},{\mathchorus{O}^{\ast\ast}_{t-1}})}$,
and
$h_{i}=\partial^{2}_{\mathchorus{O}^{\ast\ast}_{t-1}}{L(\mathchorus{O}^{\ast},{\mathchorus{O}^{\ast\ast}_{t-1}})}$
are the first and second order derivatives of loss function in the gradient,
respectively (more details about XGBoost can be found in [4]). During each
time-interval $\\{t_{a},t_{b}\\}\in t$, $a<b$, live traffic values are
provided as input to RTP for traffic prediction in the next time interval
$\\{t_{a+1},t_{b+1}\\}\in t$, $a<b$.

Let $\mathchorus{Tr}_{\sigma}$ be the deviation of network traffic from the
predicted traffic ($Pt$) over duration $\Delta{t}\in$ {$t_{a}$, $t_{b}$},
$\mathchorus{Tr}_{\sigma}^{thr}$ and $\Delta{t}^{thr}$ are traffic deviation
and time-period thresholds respectively. Eq. (5) investigates traffic status
($\Xi^{status}$) for duration ($\Delta{t}$) based on the comparison between
expected traffic deviation ($\mathchorus{Tr}_{\sigma}$) and threshold
($\mathchorus{Tr}_{\sigma}^{thr}$).

$\Xi^{status}\times\Delta{t}=\begin{cases}1,&{\text{If}(\mathchorus{Tr}_{\sigma}\times\Delta{t}>\mathchorus{Tr}_{\sigma}^{thr}\times\Delta{t}^{thr})}\\\
-1,&{\text{If}(\mathchorus{Tr}_{\sigma}\times\Delta{t}<0)}\\\
0,&{\text{Otherwise}}\end{cases}$ (5)

Accordingly, the traffic status $\Xi^{status}$ value ‘1’ indicates ‘network
congestion’, ‘-1’ specifies ‘sub-normal traffic’, and ‘normal traffic’,
otherwise. Further, the critical and sufficient condition for the congestion
is formulated and defined in Eq. (6) subject to four constraints
($C_{1}$-$C_{4}$) over time ($\Delta{t}$);

$\displaystyle\int_{t_{a}}^{t_{b}}{\bigg{(}\sum_{i=1}^{M}{Pt_{i}}+\sum_{i=1}^{M}{{\mathchorus{Tr}_{\sigma}}_{i}}}\bigg{)}dt\leq\int_{t_{a}}^{t_{b}}{\sum_{i=1}^{M}\big{(}{Nt_{i}}}\big{)}dt$
(6) $\displaystyle
s.t.\quad\forall{C_{1}}\bigvee\exists\\{C_{2},C_{3},C_{4}\\}$
$\displaystyle\left.\begin{array}[]{ll}{C_{1}:}\quad
t_{b}-t_{a}=\Delta{t}>\mathchorus{Tr}_{\sigma}\times\Delta{t}^{thr})\\\
C_{2}:\quad{\sum_{i=1}^{M}{Nt_{i}^{BW}}}\geq\sum_{k=1}^{P}{PM_{k}^{BW}}\\\
C_{3}:\quad{\sum_{i=1}^{M}{Nt_{i}^{C}}}\geq\sum_{k=1}^{P}{PM_{k}^{C}}\\\
C_{4}:\quad{\sum_{i=1}^{M}{Nt_{i}^{Mem}}}\geq\sum_{k=1}^{P}{PM_{k}^{Mem}}\end{array}\right\\}$

where $Nt_{i}^{BW}$, $Nt_{i}^{C}$, and $Nt_{i}^{Mem}$ specify capacity demand
of bandwidth, CPU, and memory, respectively, of the live traffic ($Nt$). If
the aggregated demand of the entire traffic ($\sum_{i=1}^{M}{Nt_{i}}$) is
greater than estimated resource requirement
($\sum_{i=1}^{M}{Pt_{i}}+\sum_{i=1}^{M}{{\mathchorus{Tr}_{\sigma}}_{i}}$),
then congestion is detected. $C_{1}$ states traffic overflow
($\mathchorus{Tr}_{\sigma}$) for longer than threshold time-period
($\Delta{t}^{thr}$); $C_{2}$\- $C_{4}$ state excess demand of resources viz.,
$BW$, $C$, $Mem$ of the live traffic than the available aggregated capacity of
resources on $P$ physical machines ($PM$). If congestion is anticipated, the
network traffic is diverted across multiple pathways and assigned to physical
machines reserved for handling the network hogs and heavy workloads. Likewise,
if the approaching traffic is sub-normal ($\Xi^{status}$ = $-1$), the
respective job requests are executed on least number of physical machines
subject to resource availability constraints; otherwise, the traffic is normal
which is handled by TMU. The traffic handling and management by TMU is
discussed in the following Section IV.

##  IV Cloud-based Traffic Management

The RAN traffic analysis and computation information is transferred to TMU to
allow estimation of the distinguished traffic states, including Light jobs
($\mathchorus{Tr}^{Lit}$), Stragglers ($\mathchorus{Tr}^{Stg}$), Network Hogs
($\mathchorus{Tr}^{Hog}$), Heavy jobs ($\mathchorus{Tr}^{Hvy}$), and Average
jobs ($\mathchorus{Tr}^{Avg}$) etc. Eq. (7) determines various states of live
RAN traffic {$Nt_{1}$, $Nt_{2}$, …, $Nt_{M}$} at $t^{th}$ instance by
considering different circumstances of the bandwidth demand
($(Nt_{i}^{BW}:i\in\\{1,M\\}$) and execution time ($Nt_{i}^{Et}$).

$\leavevmode\resizebox{195.12767pt}{}{${Nt_{i}^{State}}=\begin{cases}\mathchorus{Tr}^{Lit}(1),&{\text{If}(Nt_{i}^{BW}<BW_{i}^{thr}\quad\&\&\quad
Nt_{i}^{Et}<Et_{i}^{thr})}\\\
{\mathchorus{Tr}^{Stg}}(2),&{\text{If}(Nt_{i}^{BW}\leq
BW_{i}^{thr}\quad\&\&\quad Nt_{i}^{Et}\geq Et_{i}^{thr})}\\\
{\mathchorus{Tr}^{Hog}}(3),&{\text{If}(Nt_{i}^{BW}\geq
BW_{i}^{thr}\quad\&\&\quad Nt_{i}^{Et}\leq Et_{i}^{thr})}\\\
\mathchorus{Tr}^{Hvy}(4),&{\text{If}(Nt_{i}^{BW}\geq
BW_{i}^{thr}\quad\&\&\quad Nt_{i}^{Et}\geq Et_{i}^{thr})}\\\
\mathchorus{Tr}^{Avg}(5)&{\text{Otherwise}}\end{cases}$}\vspace{-.50cm}$ (7)

TMU schedules job requests ($Jr$) belonging to distinct traffic states
exclusively in the most admissible way to minimize latency of execution and
maximize the resource ($BW$, $C$, $Mem$) utilization as stated in Eqs. (8-12)
subject to constraints {$C_{1}$-$C_{6}$} specified in Eq. (13). The expression
$\omega_{kji}$ represents mapping among $k^{th}$ job ($Jr_{k}$: $k\in M$),
$j^{th}$ virtual node ($VN_{j}$: $j\in Q$), and $i^{th}$ physical machine
($PM_{i}$: $i\in P$); $R$ and $R^{\ast}$ are resource capacity of virtual node
and physical machine, respectively. The constraint $C_{1}$ specifies $k^{th}$
job can be assigned to only one $VN_{j}$ hosted on one $PM_{i}$ at an
instance; {$C_{2}$-$C_{4}$} state resource capacity of $VN_{j}$ must be lesser
or equal to available resource capacity of $PM_{i}$; and $C_{5}$ & $C_{6}$
specify resource requirement of $k^{th}$ job ($Jr_{k}$) for processing must be
satisfied by the available resources on the respective $VN_{j}$ hosted on
$PM_{i}$. The job requests confined to light traffic state
($\mathchorus{Tr}^{Lit}$) are allocated to virtual nodes ($VN$) hosted on
physical machines with required resource capacity using first-come first-serve
(FCFS) scheduling (Eq. 8).

$\displaystyle{\omega_{kji}^{\mathchorus{Tr}^{Lit}}}=Jr_{k}^{\mathchorus{Tr}^{Lit}}\times
VN_{j}^{R}\times List_{(FCFS)}{PM_{i}^{R^{\ast}}}$ (8)
$\displaystyle{\omega_{kji}^{\mathchorus{Tr}^{Stg}}}=Jr_{k}^{\mathchorus{Tr}^{Stg}}\times
VN_{j}^{R}\times MAX(List{PM_{i}^{R^{\ast}}})$ (9)
$\displaystyle{\omega_{kji}^{\mathchorus{Tr}^{Hog}}}=Jr_{k}^{\mathchorus{Tr}^{Hog}}\times
VN_{j}^{R}\times MAX(List_{BW}{PM_{i}^{R^{\ast}}})$ (10)
$\displaystyle{\omega_{kji}^{\mathchorus{Tr}^{Hvy}}}=Jr_{k}^{\mathchorus{Tr}^{Hvy}}\times\sum_{j=1}^{Z}VN_{j}^{R}\times\sum_{i=1}^{Z^{\ast}}{PM_{i}^{R^{\ast}}}$
(11)
$\displaystyle{\omega_{kji}^{\mathchorus{Tr}^{Avg}}}=Jr_{k}^{\mathchorus{Tr}^{Avg}}\times
VN_{j}^{R}\times MAX(List_{C}{PM_{i}^{R^{\ast}}})$ (12)

$\centering\begin{aligned} \left.\begin{array}[]{ll}\text{subject
to}\\{C_{1}-C_{6}\\}\\\ {C_{1}:}\quad\forall_{k\in M}\forall_{j\in
Q}\forall_{i\in P}{\omega_{kji}}=1\\\ C_{2}:\quad\forall_{k\in M}\forall_{j\in
Q}\forall_{i\in P}{VN_{j}^{C}}\times\omega_{kji}\leq PM_{i}^{C^{\ast}}\\\
C_{3}:\quad\forall_{k\in M}\forall_{j\in Q}\forall_{i\in
P}{VN_{j}^{M}}\times\omega_{kji}\leq PM_{i}^{M^{\ast}}\\\
C_{4}:\quad\forall_{k\in BW}\forall_{j\in Q}\forall_{i\in
P}{VN_{j}^{BW}}\times\omega_{kji}\leq PM_{i}^{BW^{\ast}}\\\
C_{5}:\quad\sum_{k\in M}{R}_{k}\leq\sum_{i\in
P}PM_{i}^{\mathds{R^{\ast}}}\quad
R^{\ast}\in\\{C^{\ast},M^{\ast},BW^{\ast}\\}\\\ C_{6}:\quad
r_{k}\times{R}_{k}\leq
VN_{j}^{R^{\ast}}\quad\forall_{k}\in[1,M],j\in[1,Q]\end{array}\right\\}\end{aligned}\@add@centering$

(13)

Likewise, the stragglers ($\mathchorus{Tr}^{Stg}$) requiring higher
computational capacity are assigned to virtual nodes hosted on a physical
machine with larger CPU, bandwidth, and memory capacity to allow needed I/O
operations (Eq. 9). Further, a hybrid scheduling is introduced to allow
parallel execution of stragglers with network hogs ($\mathchorus{Tr}^{Hog}$)
such that the bandwidth hogs can be processed by the computing and network
devices having sufficient resource capacity that can serve the requirement
during idle time when the stragglers are performing I/O operations (Eq. 10).
The ‘Heavy jobs’ ($\mathchorus{Tr}^{Hvy}$) demanding resource capacity larger
than the threshold are bound to execute on multiple physical machines with
highest resource capacity which can altogether serve resource demand of the
respective traffic (Eq. 11). All the remaining job requests are considered as
‘average’ and ‘delay-sensitive’ to be scheduled on the remaining physical
nodes to allow faster execution. Specifically, the average jobs
($\mathchorus{Tr}^{Avg}$) based requests are sorted as per the basis of their
deadlines and priorities, and the physical nodes are assorted in decreasing
order of their processing speed in the list (Eq. 12). Such an allocation of
virtual nodes executes lowest deadline requests on highest processing speed
servers to minimize the time of execution.

##  V Operational Design and Complexity

Algorithm 1 imparts the operational summary of TM-CRN for effective traffic
management for cloud-based RAN.

1 Input Number of: PMs ($P$), VNs ($Q$), RANs ($M$) ;

2 Initialize: $List_{{Jr}}$, $List_{{VN}}$, $List_{{PM}}$;

3 Distribute initial RAN traffic: {$Nt_{1}$, $Nt_{2}$, …, $Nt_{M}$} on VNs
hosted at $P$ PMs;

4 for  _each time-interval $\\{t_{a},t_{b}\\}$_ do

5 Estimate traffic load: $Pt_{t+1}$= RAN Traffic Predictor(${NT}_{t_{a}}$);

6 Receive heterogeneous RAN traffic (${NT}_{t_{a}+1}$) and compute traffic
deviation ($\mathchorus{Tr}_{\sigma}$) ;

7 Analyse the status of the traffic ($\Xi$) using Eq. (5) ;

8 Investigate the exact occurrence of a RAN traffic congestion by applying Eq.
(6) ;

9 Determine and distinguish among different traffic states using Eq. (7) ;

10 Apply Eqs. (8-12) to decide allocation of job requests in the most
appropriate manner;

11

12 end for

Algorithm 1 TM-CRN: Operational Summary

Step 1 allows to read the number of PMs, VNs, and RANs’ job requests from
users, consumes $\mathcal{O}(1)$ complexity while step 2 initializes lists of
jobs, VNs, and PMs, has $\mathcal{O}(1)$ complexity. Step 3 schedules $M$
network requests onto $P$ PMs, with $\mathcal{O}(P\times M)$ complexity.
Assuming steps 4-11 repeat for $t$ time intervals, wherein step 5 calls RAN
traffic predictor T $\mapsto$ $\mathcal{O}(thzlogn)$ where $t$ is the number
of trees, $h$ is the height of the trees, $z$ is the number of non-missing
entries in the training, and $n$ is the number of examples. The prediction for
a new sample consumes time $\mathcal{O}(th)$. Steps 6-10 compute Eqs. (7-12)
consume $\mathcal{O}(1)$ complexity. Hence, the total complexity comes out to
be $\mathcal{O}(PMT)$.

##  VI Performance Evaluation and Discussion

###  VI-A Experimental Set-up and Dataset

The simulation experiments are executed on a server machine assembled with two
Intel® Xeon® Silver 4114 CPU with 40 core processor and 2.20 GHz clock speed,
deployed with 64-bit Ubuntu 16.04 LTS having main memory of 128 GB in Python
3.1. The CDC environment is set up with CoS [5] having IBM servers’ CPU
(MIPS), RAM (GB) and bandwidth (bps) configurations: {1060, 2, 2000}; {2660,
4, 2000}; {3067, 8, 4000}; {4076, 64, 8000}. Four types of VMs inspired from
Amazon VM instances with CPU, RAM, and bandwidth: {500, 1,500}; {1000, 2,
1000}; {2000, 3, 1000}; {2500, 4, 2000} are used. We experimented with a
wireless traffic data sets of call detail records [6] and a C-RAN traffic
gathered from Youtube dataset on Mobile streaming [7] containing records of 80
RAN scenarios having 171 bandwidth settings, measured in 1,939 runs with
emulated 3G/4G traces.

TM-CRN is compared with Federated Meta-Learning Approach (FMLA) based wireless
traffic prediction [8], Joint UE and Fog Optimization (JUFO) scheme [2], and
Online Secure Communication Model Cloud (OSC-MC) [9] for different performance
metrics. A wireless traffic prediction approach, FMLA is proposed in [8] to
manage the highly dynamic and low latency wireless communication networks by
learning a sensitive global model with knowledge gathered from diverse
regions. A minimized energy computation derived tasks offloading and C-RAN
traffic management scheme is presented in [2], wherein cloud tasks priority is
investigated to allow the task execution within the determined delay bound.
OSC-MC model [9] prevents network traffic congestion and minimize occurrence
of network hogs by computing the traffic deviation proactively while
maintaining secure workload execution.

###  VI-B Numerical Results

Table I reports resultant values obtained for the key performance metrics for
TM-CRN with varying bandwidth ($BW^{thr}$) and execution-time ($Et^{thr}$)
threshold values (including 25%, 50%, 75%, and 90% of the respective
quantitative values) for 500 C-RAN jobs over progressive time-period. By
applying Eq. (7), traffic is filtered into different traffic states such that
for $BW^{thr}$=25% and $\mathchorus{Tr}^{Lit}$ (%) and
$\mathchorus{Tr}^{Hog}$(%) increase, while $\mathchorus{Tr}^{Stg}$(%) and
$\mathchorus{Tr}^{Hvy}$(%) decrease with increasing execution time consumption
($Et^{thr}$) and the similar trend is followed for rest of the $BW^{thr}$
values. Furthermore, with growing $BW^{thr}$, the traffic states
$\mathchorus{Tr}^{Lit}$ (%) and $\mathchorus{Tr}^{Hvy}$(%) show reduction, and
$\mathchorus{Tr}^{Hog}$(%) and $\mathchorus{Tr}^{Stg}$(%) upgrade. The
bandwidth utilization ($BW^{Util}$ %) is achieved in the range [16% - 30%] and
the jobs’ execution time ($Et$) varies with their size and processing speed of
cloud servers. However, the significant reduction in overall job execution
latency ($\mathchorus{Lt}^{Redc}$) is observed for TM-CRN as compared to the
default traffic execution (using first-come first-serve (FCFS) scheduling).
The reason is that TM-CRN efficiently exploits the possible parallelism
between jobs confined to $\mathchorus{Tr}^{Hog}$ and $\mathchorus{Tr}^{Stg}$.

TABLE I: Key Performance Indicators for TM-CRN

$BW^{thr}$ | $Et^{thr}$ | $\mathchorus{Tr}^{Lit}$ | $\mathchorus{Tr}^{Stg}$ | $\mathchorus{Tr}^{Hog}$ | $\mathchorus{Tr}^{Hvy}$ | $BW^{Util}$ | $Et^{Cons}\times 10^{4}$ | $\mathchorus{Lt}^{Redc}$
---|---|---|---|---|---|---|---|---
(%) | (%) | (%) | (%) | (%) | (%) | (%) | (sec) | (%)
25 | 25 | 11.4 | 24.12 | 11.41 | 53.07 | 19.51 | 18.72 | 23.80
25 | 50 | 41.67 | 05.70 | 29.82 | 22.81 | 17.85 | 22.02 | 10.43
25 | 75 | 56.58 | 01.32 | 34.21 | 07.90 | 20.75 | 23.06 | 6.17
25 | 90 | 64.48 | 0.00 | 35.52 | 0.00 | 22.46 | 24.58 | 0.00
50 | 25 | 7.89 | 44.29 | 14.91 | 32.89 | 25.03 | 14.22 | 42.10
50 | 50 | 24.12 | 11.84 | 47.37 | 16.67 | 26.16 | 20.06 | 18.37
50 | 75 | 33.77 | 02.19 | 57.02 | 07.02 | 26.11 | 22.41 | 8.80
50 | 90 | 33.77 | 01.75 | 57.46 | 07.01 | 26.11 | 22.59 | 8.08
75 | 25 | 5.70 | 57.46 | 17.10 | 19.74 | 28.54 | 09.30 | 62.12
75 | 50 | 14.03 | 17.54 | 57.46 | 10.97 | 27.82 | 16.47 | 32.97
75 | 75 | 20.61 | 04.83 | 70.18 | 04.38 | 29.07 | 19.48 | 20.73
75 | 90 | 25.00 | 0.00 | 75.00 | 0.00 | 28.82 | 24.58 | -0.02
90 | 25 | 01.75 | 67.11 | 21.05 | 10.08 | 18.26 | 06.02 | 75.52
90 | 50 | 06.14 | 22.80 | 65.35 | 05.70 | 25.89 | 13.90 | 43.42
90 | 75 | 09.21 | 06.58 | 81.57 | 02.63 | 24.64 | 17.73 | 27.84
90 | 90 | 09.21 | 06.14 | 82.01 | 02.63 | 24.57 | 17.91 | 27.12

Table II reports the performance of the XGBoost-based prediction model, where
the achieved mean squared error (MSE) and mean absolute error (MAE) vary in
the ranges: [0.0033-0.0080] and [0.05-0.11], respectively (Appendix provides
more prediction results using Google Cluster Dataset to further validate the
XGBoost’s performance). The prediction error increases and training time
elapsed (TTE) decreases with the size of the prediction interval because of
decreased number of data samples. Further, the achieved traffic estimation
errors: MSE and MAE are compared against different versions of FMLA [8]
including standard network (FMLA-s), wide network (FMLA-w), deep network
(FMLA-d), and Optimal case (prediction error is zero) as depicted in Fig. 2.
MAE and MSE obtained for Milan (viz., $Mi^{MAE}$ and $Mi^{MSE}$) are lesser
for both the proposed and FMLA, however, significant reduction in error of
90.8% is observed for Trentino in case of TM-CRN over FMLA-d because of
intuitive pattern learning and optimization of extreme gradient boosting
approach.

TABLE II: XGBoost-based Prediction performance metrics

PWS (min) | MSE | MAE | TTE
---|---|---|---
5 | 0.0038 | 0.0547 | 28.89
10 | 0.0047 | 0.0857 | 24.09
30 | 0.0056 | 0.0860.2 | 21.64
60 | 30.0078 | 0.1055 | 19.95

PWS: Prediction window size, TTE: training time elapsed (msec)

$Mi^{MAE}$$Mi^{MSE}$$Tr^{MAE}$$Tr^{MSE}$$0$$0.2$$0.4$$0.6$DatasetsErrorOptimal
TM-CRNFMLA-s [8]FMLA-w FMLA-d  Figure 2: Traffic estimation error

Fig. 3 compares TM-CRN with OSC-MC [9] and Optimal case for occurrence of
C-RAN traffic congestion using Youtube mobile streaming dataset. TM-CRN
reduces the congestion up to 10.15% over OSC-MC due to engagement of improved
prediction capability and effective distribution and management of the live
traffic into distinct traffic states. TM-CRN is closer to Optimal for
$Tr^{{25\%}}_{{25\%}}$, $Tr^{{50\%}}_{{50\%}}$ and show 2%-9% more congestion
as compared with remaining cases of traffic thresholds.

$Tr^{{25\%}}_{{25\%}}$$Tr^{{50\%}}_{{50\%}}$$Tr^{{50\%}}_{{75\%}}$$Tr^{{75\%}}_{{50\%}}$$Tr^{{75\%}}_{{75\%}}$$Tr^{{90\%}}_{{90\%}}$$0$$5$$10$$15$$20$CRAN-
Traffic threshold ($Tr^{BW}_{Et}$)Traffic Congestion (%)OptimalTM-CRNOSC-MC
[9] Figure 3: Traffic congestion

Fig. 4 compares the network bandwidth utilization percent of TM-CRN with
Optimal case, JUFO [2] and baseline methods: random-fit and first-fit over
varying thresholds for both $BW^{thr}$ and $Et^{thr}$ viz., 5%, 25%, 75%, and
90% during experiments. The bandwidth utilization has improved by 17.07%,
17.31%, and 18% over JUFO, random-fit, and first-fit, respectively and reduced
by 2.7% over Optimal case for $BW^{thr}$ = $Et^{thr}$ = 75% because of the
division of the jobs according to their bandwidth and execution time
requirement into distinct traffic states followed by the privileged management
accordingly that accelerates the optimization of C-RAN jobs allocation and
execution.

5%25%50%75%90%$10$$20$$30$CRAN-Traffic ${BW^{thr}}$$BW$ Utilization
(%)OptimalTM-CRNJUFO [2]Rand-FitFirst-Fit Figure 4: Network bandwidth
utilization

The comparison of execution time consumed is shown in Fig. 5, the proposed
approach of C-RAN traffic management consumes 19.33%, 26.90%, and 27.12%
lesser time over JUFO, random-fit, and first-fit, respectively. Accordingly,
the job execution latency for TM-CRN management is reduced by 23.80 %, 18.37%,
20.73%, and 27.12% for the $BW^{thr}$ and $Et^{thr}$ values: 25%, 50%, 75%,
and 90%, respectively over default FCFS scheduling-based traffic management.
On the other hand, the job latency has been reduced up to 1.22 %, 10.68%, and
2.74% for JUFO. However, the time elapsed for TM-CRN over Optimal case is
higher up to 33.5%-51.27% because of the intended traffic prediction and
analysis in case of TM-CRN which is avoided for Optimal case. The significant
time reduction is achieved for the proposed approach due to the traffic
classification into distinct states, and incorporation and exploitation of
parallelism by executing hogs having higher bandwidth and lesser execution
time requirement in-between stragglers (slow executing jobs).

5%25%50%75%90%$1$$1.5$$2$$2.5$$\cdot 10^{5}$CRAN-Traffic ${Et^{thr}}$$Et$
consumption (sec)OptimalTM-CRNJUFO [2]Rand-FitFirst-Fit Figure 5: Service
execution time

##  VII Conclusion

A novel TM-CRN model is proposed to facilitate effective RAN traffic
management with minimum latency period and maximum bandwidth utilization. The
probable resource demand and traffic congestion is detected using XGBoost
predictor. Traffic management unit classifies that traffic into distinct
states to utilize available bandwidth productively for parallel execution of
RAN traffic confined to stragglers and network hogs. The performance
evaluation and comparison confirmed that TM-CRN model is more admissible as
compared with state-of-the-art approaches.

## References

  * [1] Z. Zhao, C. Feng, H. H. Yang, and X. Luo, “Federated-learning-enabled intelligent fog radio access networks: Fundamental theory, key techniques, and future trends,” _IEEE wireless comm._ , vol. 27, no. 2, pp. 22–28, 2020\.
  * [2] J. Kim, T. Ha, W. Yoo, and J.-M. Chung, “Task popularity-based energy minimized computation offloading for fog computing wireless networks,” _IEEE Wireless Comm. Letters_ , vol. 8, no. 4, pp. 1200–1203, 2019.
  * [3] M. A. Habibi, M. Nasimi, B. Han, and H. D. Schotten, “A comprehensive survey of ran architectures toward 5g mobile communication system,” _IEEE Access_ , vol. 7, pp. 70 371–70 421, 2019.
  * [4] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , 2016, pp. 785–794.
  * [5] D. Saxena, I. Gupta, A. K. Singh, and C.-N. Lee, “A fault tolerant elastic resource management framework towards high availability of cloud services,” _IEEE Transactions on Network and Service Management_ , 2022.
  * [6] G. Barlacchi, M. De Nadai, R. Larcher, A. Casella, C. Chitic, G. Torrisi, F. Antonelli, A. Vespignani, A. Pentland, and B. Lepri, “A multi-source dataset of urban life in the city of milan and the province of trentino,” _Scientific data_ , vol. 2, no. 1, pp. 1–15, 2015.
  * [7] F. Loh, F. Wamser, F. Poignée, S. Geißler, and T. Hoßfeld, “Youtube dataset on mobile streaming for internet traffic modeling, network management, and streaming analysis. figshare. dataset,” _https://doi.org/10.6084/m9.figshare.19096823.v2_ , 2022.
  * [8] L. Zhang, C. Zhang, and B. Shihada, “Efficient wireless traffic prediction at the edge: A federated meta-learning approach,” _IEEE Comm. Letters_ , 2022\.
  * [9] D. Saxena and A. K. Singh, “OSC-MC: Online secure communication model for cloud environment,” _IEEE Comm. Letters_ , vol. 25, no. 9, pp. 2844–2848, 2021.


# Masked and Shuffled Blind Spot Denoising for Real-World Images

Hamadi Chihaoui
Computer Vision Group
University of Bern, Switzerland
<EMAIL_ADDRESS>Paolo Favaro
Computer Vision Group
University of Bern, Switzerland
<EMAIL_ADDRESS>
###### Abstract

We introduce a novel approach to single image denoising based on the Blind
Spot Denoising principle, which we call MAsked and SHuffled Blind Spot
Denoising (MASH). We focus on the case of correlated noise, which often
plagues real images. MASH is the result of a careful analysis to determine the
relationships between the level of blindness (masking) of the input and the
(unknown) noise correlation. Moreover, we introduce a shuffling technique to
weaken the local correlation of noise, which in turn yields an additional
denoising performance improvement. We evaluate MASH via extensive experiments
on real-world noisy image datasets. We demonstrate state-of-the-art results
compared to existing self-supervised denoising methods. Website:
https://hamadichihaoui.github.io/mash.

##  1 Introduction

The removal of noise from real images, _i.e_., image denoising, is a
fundamental and still open problem in image processing despite having a long
history of dedicated research (see [9] for an overview of the classic and
recent methods). In classic methods, the primary strategies involve manually
designing image priors and optimization techniques to enhance both
reconstruction accuracy and speed. In contrast, in the context of deep
learning methods, neural networks naturally introduce a very powerful prior
for images [24] and provide models that could perform denoising efficiently at
inference time. These innate capabilities of neural networks opened the doors
to a wide range of methods that could not only learn to denoise image from
examples of noisy and clean image pairs, but, even more remarkably, directly
from single noisy images [13, 26, 11, 25].

In this work, we push the limits of these advanced methods one step further.
We focus on the family of methods called Blind Spot Denoising (BSD)[13], since
it provides a powerful and general framework. Moreover, we consider the case
where only a single image is used for denoising (_i.e_., we do not rely on a
supporting dataset). As also observed by Wang et al [25], training on a
dataset may not generalize well on new data, where the noise distribution is
unknown. This is particularly true for real images, where noise is often
correlated. In these settings, most modern methods find it challenging to
handle non-iid data.

In our work, similar to the approach in [19], we explore the more general
setting of random masking beyond the single blind spot method introduced in
[13]. In our analysis, we uncover valuable connections between the performance
of Blind Spot Denoising (BSD) methods trained with various input masking
techniques and the degree of noise correlation. Surprisingly, we observe that
models trained with a higher masking ratio tend to perform better when dealing
with highly correlated noise, whereas models trained with a lower masking
ratio excel in denoising tasks with iid noise. This discovery offers two key
contributions: 1) it provides a method to estimate the unknown level of noise
correlation, and 2) it offers a strategy for achieving enhanced denoising
performance. Furthermore, our analysis reveals that noise correlation
significantly hampers the denoising capabilities of BSD models. This suggests
that a more radical approach would be to directly eliminate the correlation in
the input data. An intuitive method to achieve this would involve randomly
permuting all pixels that correspond to the same clean-image color intensity.
However, this presents a classic chicken and egg dilemma, as we would
typically need the clean image to perform the permutation, yet the clean image
is precisely what we are trying to restore. To tackle this challenge, we
utilize an intermediate denoised image as a pseudo-clean image to define the
permutation set. Furthermore, given that adjacent pixels are likely to have
similar color intensities, we focus on shuffling only pixels within small
neighborhoods. We incorporate these insights into a novel method called MAsked
and SHuffled Blind Spot Denoising (MASH), which we elaborate on further in
Sec. 3. Our contributions are summarized as follows

  * •

We provide an analysis of BSD, showcasing the impact of various masking ratios
on correlated noise and presenting a method for estimating the noise
correlation level;

  * •

We introduce MASH, an enhanced version of BSD that dynamically selects the
optimal masking ratio. We also introduce the local pixel shuffling technique
to address noise correlation at its source;

  * •

MASH demonstrates marked enhancements over the baseline BSD and attains on par
or better results in real-world denoising across multiple datasets.

##  2 Related Work

In this section, we only focus on work that is closely related to unsupervised
image denoising.

Non-learning-based image denoisers. Classic image denoisers [4, 27, 22]
manually define image priors of what a clean image is. Some approaches explore
sparse representations [2, 16], while others benefit from the patch recurrence
prior [32]. BM3D [6], which applies collaborative filtering to similar
patches, is one of the best-known non-local methods, due to its high
performance across several benchmarks. NLM [3] and WNNM [20] also relate
similar patches and use them to reduce noise through a form of implicit
averaging.

Unsupervised learning-based image denoisers. The unsupervised learning-based
approaches can be classified into two categories based on their training data.
The first category is the dataset-based one, which uses a dataset of noisy
images to train a denoising model. The second category is the single-image
one, which learns a denoiser from a single image at a time.
1) Dataset-based learning approaches.
Noise2Noise [12] shows that it is possible to train a denoising neural network
without the need for clean images. Indeed, using only pairs of noisy images of
the same scene as input/output training data leads to a comparable performance
of standard noisy/clean image pair supervised training. Some approaches [13,
26, 11, 25] go one step further and remove the need for noisy image pairs
altogether. The main contributions in these approaches are ways to avoid
learning the trivial solution of the identity mapping. Recently, [31] proposed
the use of pixel-shuffle downsampling (PD) to weaken the spatial correlation
of structured noise. In a similar vein, AP-BSN [14] utilized two PDs with
different strides for training and testing. CVF-SID [18] attempted to separate
the latent image and noise from the noisy input through a cyclic module and
self-supervised losses. LUD-VAE [30] established two hidden variables for the
noise domain and the latent image domain and optimize for them using unpaired
data. [8, 21] explicitly learned a model for real-world noise. These methods
rely on noisy or unpaired data, and generalization issues persist.
2) Test-time training approaches.
A learning-based image denoising approach without any a-priori dependence on
training samples is the least demanding option to denoise a given noisy image.
There are a few methods that use a single noisy image to train a deep neural
network for denoising. The DIP [24] shows that the inductive bias of
convolutional neural networks (CNN) favors the learning of noise-free texture
patterns rather than noisy ones, when trying to reconstruct a degraded image.
The method trains a CNN to generate a given degraded image from a random
input, while early stopping is used as a regularization. DIP is highly
sensitive to the choice of the stopping time and thus is not practical as a
denoising method. ScoreDVI [5] recently proposes a method for denoising a
single image by exploiting score priors embedded in MMSE. [29] addresses the
complexity of real noise by mapping the noisy image into a latent space in
which the additive white Gaussian noise (AWGN) assumption holds. An encoder-
decoder is used for the mapping and an off-the-shelf Gaussian denoiser is used
to denoise the encoded image. Finally the image is decoded back to the
original space. [19] is a self-supervised approach that use pairs of
Bernoulli-sampled instances of the input noisy image to train a denoiser with
dropout. The denoiser predicts the masked pixels based on the visible ones.
The final denoised image is the average of the predictions generated from
multiple instances of the trained model with dropout.

MASH belongs to the category of BSD methods applied to a single noisy image.
However, in contrast to the above methods, it uses an optimal masking ratio
that is automatically estimated and introduced the technique of local pixel
shuffling. To the best of our knowledge, these contributions have not been
presented before.

##  3 Unsupervised Single Image Denoising

In this section, we begin by revisiting the concept of BSD as introduced in a
prior work by Krull et al. [13]. Subsequently, we conduct experimental
analyses to delve deeper into understanding the effects of various design
factors in the BSD method on its performance. For this purpose, we generate a
synthetic noisy dataset with noise ranging from independent and identically
distributed (iid) to highly correlated, using which we train a base blind spot
network with differing levels of ”blindness” (i.e., image masking ratios).
These empirical observations lead us to create a novel self-supervised blind-
spot framework for unsupervised single image denoising, referred to as MAsked
and SHuffled Blind Spot Denoising (MASH), which achieves state-of-the-art
performance.

(a)

(b)

(c)

Figure 1:  Samples of generated noisy images depending on the spatial
correlation level. From left to right: noisy image with iid noise, noisy image
with moderately correlated noise and noisy image with heavily correlated noise

###  3.1 Revisiting Blind Spot Denoising (BSD)

BSD operates as a self-supervised technique, meaning that its training process
does not necessitate pairs of noisy images or pairs of noisy and clean images.
Instead, only noisy images are employed. The method applies a masking scheme
that hides part of the image at the input and then aims to predict the same
hidden part at the output. In this paper, we apply this idea to a single image
at a time. Formally, considering the noisy observation
$\mathbf{y}\in\mathbb{R}^{H\times W\times C}$ corresponding to the clean image
$\mathbf{x}\in\mathbb{R}^{H\times W\times C}$, where $H\times W$ represents
the image dimensions and $C$ denotes the color channels (usually $1$ for
grayscale or $3$ for RGB), the objective of BSD is to minimize the following
empirical risk:

$\mathop{\arg\min\limits_{\theta}}\sum_{\mathbf{i}\in\Omega}\|f_{\theta}(\mathbf{y}_{\text{masked}(\mathbf{i})})[\mathbf{i}]-\mathbf{y}[\mathbf{i}]\|_{2}^{2}$
(1)

In Eq. 1, $f_{\theta}:\mathbb{R}^{H\times W\times
C}\rightarrow\mathbb{R}^{H\times W\times C}$ represents the denoising model
realized through neural networks with parameters $\theta$, which transforms
noisy images into denoised versions. Here,
$\mathbf{y}_{\text{masked}(\mathbf{i})}$ denotes the image $\mathbf{y}$ where
the pixel $\mathbf{i}\in\Omega$ ($\Omega=[1,\dots,H]\times[1,\dots,W]$), has
been masked. Additionally, $\mathbf{y}[\mathbf{i}]$ refers to the RGB color at
the pixel $\mathbf{i}$. Notice that the above formulation is used for a single
image $\mathbf{y}$. It is also possible to extend it to a dataset of images
simply by taking the expectation of the above loss with respect to the
distribution of $\mathbf{y}$. However, as already mentioned, in this paper we
only consider denoising a single image without additional data. In our
terminology, we also refer to _signal_ when talking about the clean image. BSD
assumes that noise at each pixel is statistically independent of its neighbors
(thus a noise instance cannot be predicted by nearby noise), while a pixel of
the signal (_i.e_., the clean image) is spatially correlated to its
neighboring pixels. These assumptions are key in enabling the separation of
noise from signal. More in general, the BSD method can also be seen as an
extreme case of masking or sparse image reconstruction/inpainting such as MAE
[10], where the masking consists of a single pixel. In practice, the BSD
method avoids trivial solutions, such as the identity mapping (simply
reconstructing the noisy image), only as long as $f_{\theta}$ does not overfit
the data. This is particularly important in the context of single image
denoising, where the training dataset is limited in size. Therefore,
considering that the degree of masking may have a regularizing effect on
$f_{\theta}$, we investigate the influence of this design choice on single
image denoising.

###  3.2 Diving Deep into Blind Spot Denoising

In this section, our goal is to empirically evaluate the performance of the
BSD method by examining the effects of two key factors: 1) the masking ratio
used in the BSD method and 2) the characteristics of the noise (specifically,
its level of spatial correlation). To conduct this analysis, we generate a
denoising dataset synthetically using images from the Kodak dataset [7]. We
model the noise using a multivariate normal distribution with a variance-
covariance matrix $\Sigma$ to simulate real-world correlated noise. The
correlation $\Sigma[\mathbf{i},\mathbf{j}]$ between pixels $\mathbf{i}$ and
$\mathbf{j}$ in the set $\Omega$ is defined as follows:

$\Sigma[\mathbf{i},\mathbf{j}]=\begin{cases}\sigma^{2}&\text{
$\mathbf{i}=\mathbf{j}$}\\\
\beta\frac{k-\|\mathbf{i}-\mathbf{j}\|}{k}\sigma^{2}&\text{$0<\|\mathbf{i}-\mathbf{j}\|\leq
k$}\\\ 0&\text{ otherwise}\end{cases}$ (2)

where $\|\mathbf{i}-\mathbf{j}\|$ is the distance between the pixels
$\mathbf{i}$ and $\mathbf{j}$ and $k>0$ denotes the correlation kernel width.
The parameter $\beta>0$ is used to control the level of spatial correlation in
the noise. A higher value of $\beta$ indicates stronger correlation in the
noise. We define three distinct regimes based on the noise correlation: 1) The
_iid regime_ , where $\beta=0$, and the noise is independent and identically
distributed (iid); 2) The _moderately correlated regime_ , where $\beta=0.5$;
3) The _heavily correlated regime_ , where $\beta=1.0$; We set $k=3$ and
$\sigma=25$. An extension to this analysis, considering lower noise levels
($\sigma=15$) and higher noise levels ($\sigma=40$), is provided in the
supplementary material due to space constraints. In this extension, we draw
similar conclusions to those obtained for $\sigma=25$. Fig. 1 illustrates an
example of an image with synthetically generated noise for each of the defined
regimes.

We define the _blindness mask_ $\mathbf{m}:\Omega\mapsto\\{0,1\\}^{{H\times
W\times C}}$, where $\Omega$ represents the set of pixels in our noisy image
$\mathbf{y}$.

$\mathbf{m[\mathbf{i}]}=\begin{cases}0&\text{with probability $\tau$ };\\\
1&\text{with probability $1-\tau$ }.\end{cases}$ (3)

$\mathbf{m}$ is of the same size as the input images and is determined by a
masking ratio parameter $\tau$. The masking ratio $\tau$ represents the
proportion of zeros in the mask $\mathbf{m}$, calculated as the number of
zeros divided by the total number of pixels in the image ($HWC$). We adopt a
similar BSD formulation as [19] with the blindness mask via the following loss

$L(\theta)=\mathbb{E}_{\mathbf{m}}\left[\|(\mathbf{1}-\mathbf{m})\odot(f_{\theta}(\mathbf{y}\odot\mathbf{m})-\mathbf{y})\|_{2}^{2}\right]$
(4)

where $\mathbb{E}_{\mathbf{m}}$ denotes the expectation with respect to the
mask $\mathbf{m}$ and $\odot$ denotes the element-wise product.

####  3.2.1 Impact of the masking ratio on BSD performance

We investigate the impact of the masking ratio $\tau$ on the denoising
performance depending on the correlation magnitude controlled by $\beta$. We
train a denoising network by minimizing the loss (4) with different masking
levels in the three noise regimes (_i.e_., with different correlation
magnitudes $\beta$). The results are shown in Fig. 2. We plot the Peak Signal-
to-Noise Ratio (PSNR) values in decibels across the three regimes for various
masking configurations (displayed on the x-axis). We notice that the
effectiveness of our generalized BSD is greatly influenced by: 1) the
correlation of the noise, and 2) the masking ratio. Specifically, in the case
of independent and identically distributed (iid) noise, we discovered that a
lower masking ratio results in the best performance. Conversely, in scenarios
where the noise exhibits high correlation, a higher masking ratio produces
superior performance. In situations of moderate correlation, an intermediate
masking ratio (ranging from 0.3 to 0.5) is deemed optimal. As observed in the
analysis of DIP [24] a neural network model, such as $f_{\theta}$, tends to
first fit clean image patterns and then noise patterns. In scenarios of high
spatial noise correlation, the noise tends to exhibit patterns resembling
textures present in clean images (refer to the rightmost image in Fig. 1).
Consequently, in cases of increased noise correlation, the efficacy of BSD
could potentially improve with greater regularization, _i.e_., a higher
masking ratio $\tau$.

Figure 2:  Impact of the masking ratio $\tau$ on the generalized BSD denoising
performance (PSNR). On the horizontal axis we consider several masking ratios
$\tau$ and for each we train a BSD model on data with different levels of
correlation $\beta$. The optimal performance of the trained model shows a
strong correlation between the masking ratio an the noise correlation. Low
masking benefits the training on data with iid noise and high masking benefits
the training on data with highly correlated noise. Figure 3:  Impact of the
local pixel shuffling on the denoising performance (PSNR) when noise is highly
correlated. The shuffling of image regions that are approximately constant
destroys the noise correlation. This brings a consistent benefit across all
masking ratios.

####  3.2.2 Handling correlation with local pixel shuffling

By comparing the top performances of BSD under each noise regime in Fig. 2, we
can readily observe a significant decrease in performance in the highly
correlated noise scenario compared to the iid case, despite both scenarios
having the same noise level. This observation prompts the consideration of
another strategy to enhance performance in the presence of correlated noise.
The idea is to reduce noise correlation without altering the underlying image
signal. One intuitive approach is to identify sets of noisy pixels
corresponding to identical colors in the clean image and then randomly permute
them. By doing so, we can disrupt the spatial correlation of these pixels
without affecting the clean image structure. A practical method to implement
this concept is to utilize the predicted denoised image as a pseudo-clean
image. This pseudo-clean image can be leveraged to identify sets of iso-
intensity pixels. Since neighboring pixels are more likely to share similar
intensities, the swapping process can be performed locally to maintain
coherence in the image. To implement this idea, we first divide the image into
two categories of regions: regions with nearly constant intensity and regions
with texture. We define the image partition denoted as $\Omega_{\text{const}}$
as the set of pixels where the local neighborhoods, such as those defined by
$4\times 4$ pixel blocks, exhibit similar or identical color intensity values.
The remaining partition is categorized with texture ( significant variation in
intensity patterns).

We define $\mathbf{c}(\mathbf{y})$ as the mapping that assigns a value of $1$
to the pixels within the constant intensity regions ($\Omega_{\text{const}}$)
of $\mathbf{y}$ and a value of $0$ to all other pixels within the image domain
$\Omega$. $\mathbf{c(y)}$ helps to differentiate between the constant
intensity regions and the textured regions within $\mathbf{y}$. The
illustration in Fig. 4 provides a visual representation of this partitioning
concept. Let $\Gamma(\mathbf{y})$ define the local random permutation of
pixels within $s\times s$ (_e.g_., $s=4$) tiles of $\mathbf{y}$. We note that
the pixel shuffling is performed only within each tile. We now define the
shuffled noisy image $\mathbf{y}^{\text{shuffled}}$ as:

$\mathbf{y}^{\text{shuffled}}=\mathbf{c(y)}\odot\Gamma(\mathbf{y})+(\mathbf{1}-\mathbf{c(y)})\odot\mathbf{y}$
(5)

Then, the shuffled image can serve as the target for the BSD loss in Eq. 4. We
call this decorrelation technique Local Pixel Shuffling (LPS). Now we are
ready to define the loss of MASH to further reduce the model overfitting
caused by highly correlated noise in the BSD approach:

$L(\theta)=\mathbb{E}_{\mathbf{m}}\left[\|(\mathbf{1}-\mathbf{m})\odot(f_{\theta}(\mathbf{y}\odot\mathbf{m})-\mathbf{y^{\text{shuffled}}})\|_{2}^{2}\right]$
(6)

To determine the region $\Omega_{\text{const}}$, we adopt a similar approach
to [15]. We utilize the intermediate pseudo-clean image output
$\hat{\mathbf{y}}$, which we obtain from $f_{\theta}$ (details in the
following sections) after a certain number of training iterations. An
intuitive indicator of the similarity of pixels within a tile is their
standard deviation $\sigma:\Omega\mapsto[0,\infty)$. Specifically, we
calculate the standard deviations within patches of the size of the local
tiles independently for each color channel and then average them. Finally, we
can derive the partition as defined in Eq. 7, with the parameter $\lambda>0$
serving as a threshold for the color similarity.

$\Omega_{\text{const}}=\\{\mathbf{i}:\sigma[\mathbf{i}]<\lambda\\},$ (7)

Fig. 3 shows that applying the local pixel shuffling improves the denoising
performance when the noise is spatially highly correlated.

(a)

(b)

(c)

Figure 4:  (a) Original noisy image (b) Mask capturing region flatness derived
from pseudo-clean prediction (c) Noisy image with local pixel permutation on
flat regions.

####  3.2.3 Automated selection of the BSD masking ratio

To make MASH of practical use, it is essential to have an automated mechanism
for determining the noise correlation and, consequently, selecting the optimal
masking ratio. Towards this goal, we analyze the estimated noise level
predicted by our BSD method using different masking ratios. We denote by
$\hat{\sigma}_{\tau}$ the estimated noise level of the noisy image
$\mathbf{y}$ when using a masking scheme with a masking ratio $\tau$:

$\displaystyle\hat{\sigma}_{\tau}=\sqrt{\frac{1}{HWC}\|f_{\theta}(\mathbf{m}\odot\mathbf{y})-\mathbf{y}\|_{2}^{2}}.$
(8)

Fig. 5 illustrates the estimated noise level $\hat{\sigma}_{\tau}$ during the
denoising iterations, which is dependent on the estimated parameters $\theta$,
for varying correlation regimes and masking ratios. To ensure a dependable
indicator, we focus solely on the noise level estimated at convergence and we
define the _noise level estimation gap_ $\varepsilon$ as the difference in
noise levels when utilizing a low masking ratio $\tau^{\text{low}}$ and a high
masking ratio $\tau^{\text{high}}$:

$\displaystyle\varepsilon=|\hat{\sigma}_{\tau^{\text{high}}}-\hat{\sigma}_{\tau^{\text{low}}}|$
(9)

We set $\tau^{\text{low}}=0.2$ and $\tau^{\text{high}}=0.8$. We experimentally
verify that $\varepsilon$ is proportional to the level of correlation in the
noise. Therefore, we can use $\varepsilon$ as a proxy for assessing the degree
of spatial noise correlation present in the input image $\mathbf{y}$.

Figure 5:  Estimated noise level based on different correlated noise magnitude
and masking ratios.

###  3.3 MASH

As a conclusion of our prior experimental analysis, we propose to integrate
the adaptive masking and local pixel shuffling in the BSD approach. The
pseudo-code of our method (MASH) is described in Algorithm 1. We begin by
training a model $f_{\theta}$ using two different masking ratios: a high one
($\tau^{\text{high}}$) and a low one ($\tau^{\text{low}}$). Subsequently, we
calculate the noise level estimation gap $\varepsilon$ as outlined in Eq. 9.
Based on the value of $\varepsilon$, we dynamically determine the masking
ratio $\tau$ to utilize, as well as whether to activate the local pixel
shuffling. Our automated selection of the optimal masking ratio is depicted in
Eq. 10.

$\tau^{\text{optimal}}=\begin{cases}\tau^{\text{low}}&\text{if
}\varepsilon\leq\varepsilon^{\text{low}};\\\ \tau^{\text{medium}}&\text{if
}\varepsilon^{\text{low}}<\varepsilon<\varepsilon^{\text{high}};\\\
\tau^{\text{high}}&\text{if
}\varepsilon^{\text{high}}<\varepsilon.\end{cases}$ (10)

If $\varepsilon\leq\varepsilon^{\text{high}}$, we do not apply the local pixel
shuffling (indicating low noise correlation) and directly optimize the loss
(4). If $\varepsilon^{\text{high}}<\varepsilon$ (implying highly correlated
noise), we optimize the loss (4) for $N_{1}$ iterations. Subsequently, we
determine the partition $\Omega_{\text{const}}$. We then apply local random
permutation $\Gamma$ within $\Omega_{\text{const}}$ and compute
$\mathbf{y}^{\text{shuffled}}$. Finally, we resume training using the loss
(6). The recovered image $\hat{\mathbf{y}}$ is an ensemble of $K$ predictions.
To get each prediction, we sample a random binary mask $\mathbf{m_{p}}$ with a
masking ratio $\tau^{\text{optimal}}$ and apply it to the input image. We set
$K=10$.

$\hat{\mathbf{y}}=\frac{1}{K}\sum_{p=1}^{K}f_{\theta}(\mathbf{m_{p}}\odot\mathbf{y}).$
(11)

Algorithm 1 MASH

0: Noisy image $\mathbf{y}$, $f_{\theta}$,$\tau^{\text{high}}$,
$\tau^{\text{medium}}$, $\tau^{\text{low}}$, $\varepsilon^{\text{low}}$,
$\varepsilon^{\text{high}}$, $N$ and $N_{1}<N$

0: Restored image $\hat{\mathbf{y}}$

1: Apply the BSD baseline with $\tau^{\text{high}}$ and $\tau^{\text{low}}$
respectively.

2: Compute $\varepsilon$ from eq. (9)

3: $\tau^{\text{optimal}}=$ $\begin{cases}\tau^{\text{low}}&\text{if
}\varepsilon\leq\varepsilon^{\text{low}};\\\ \tau^{\text{medium}}&\text{if
}\varepsilon^{\text{low}}<\varepsilon<\varepsilon^{\text{high}};\\\
\tau^{\text{high}}&\text{if
}\varepsilon^{\text{high}}<\varepsilon.\end{cases}$

4: if $\varepsilon<\varepsilon^{\text{high}}$ then

5: for $t:1\rightarrow N$ do

6: Update $f_{\theta}$ by optimizing eq. (4) using $\tau^{\text{optimal}}$

7: end for

8: else

9: for $t:1\rightarrow N_{1}$ do

10: Update $f_{\theta}$ by optimizing eq. (4) using $\tau^{\text{optimal}}$

11: end for

12: compute the partition $\Omega_{\text{const}}$ using eq. (7)

13: compute $\mathbf{y}^{\text{shuffled}}$ using eq. (5)

14: for $t:N_{1}\rightarrow N$ do

15: Update $f_{\theta}$ by optimizing eq. (6) using $\tau^{\text{optimal}}$

16: end for

17: end if

18: Return the restored image
$\hat{\mathbf{y}}=\frac{1}{K}\sum_{p=1}^{K}f_{\theta}(\mathbf{m_{p}}\odot\mathbf{y})$

##  4 Experiments

In this section, we will first introduce the experimental settings. We will
then present the quantitative and qualitative results of MASH, along with
comparisons with other methods.

###  4.1 Experimental settings

Datasets We evaluated our method on four widely-used real-world noise
datasets: SIDD (validation and benchmark datasets) [1], FMDD [28], and PolyU
[17]. The validation and benchmark datasets of SIDD contain natural sRGB
images captured by smartphones, with each dataset consisting of 1280 patches
sized $3\times 256\times 256$. FMDD contains fluorescence microscopy images
with a size of $512\times 512$. The PolyU dataset consists of 100 natural
images taken from diverse commercial camera brands, with each image having a
size of $3\times 512\times 512$.

Implementation details The network architecture for MASH is the same as in
Noise2Noise [12]. The denoising network is trained from scratch using the Adam
optimizer with cosine annealing. By default, we use the following
hyperparameters in our implementation unless otherwise specified:
$\tau^{\text{high}}=0.8$, $\tau^{\text{low}}=0.2$, $\tau^{\text{medium}}=0.5$,
$\varepsilon^{\text{low}}=1.5$, $\varepsilon^{\text{high}}=2.5$, $s=4$ and
$N=800$. In our supplementary material, we provide a more in-depth discussion
on the selection of hyperparameters.

|  |
---|---|---
|  |

Figure 6:  Top: from left to right: original noisy image, Mask capturing region flatness derived from pseudo-clean, Shuffled noisy image (using LPS). Bottom: from left to right: Result using the baseline ($\tau=0.5$), Result without local pixel shuffling , Ours. Category | Method | SIDD Validation | SIDD Benchmarck | FMDD | PolyU
---|---|---|---|---|---
| BM3D [6] | 25.65/0.475 | 25.65/0.685 | 30.06/0.771 | 37.40/0.953
| DIP [24] | 32.11/0.740 | - | 32.90/0.854 | 37.17/0.912
Single Image | Self2Self [19] | 29.46/0.595 | 29.51/0.651 | 30.76/0.695 | 37.52/0.926
(test-time training) | PD-denoising [31] | 33.97/0.820 | 33.61/0.894 | 33.01/0.856 | 37.04/0.940
| NN+denoiser [29] | - | 33.18/0.895 | 32.21/0.831 |  37.66/0.956
| APBSN-single [14] | 30.90/0.818 | 30.71/0.869 | 28.43/0.804 | 29.61/0.897
| ScoreDVI [5] | 34.75/0.856 | 34.60/0.920 | 33.10/0.865 |  37.77/0.959
| Baseline | 33.12/0.805 | 32.67/0.850 | 32.25/0.824 | 37.12/0.911
| Ours | 35.06/0.851 | 34.78/0.900 |  33.71/0.882 | 37.62/0.932
Noisy/Impaired | APBSN [14] | - | 36.91/0.931 | 31.99/0.836 |  37.03/0.951
Dataset | CVF-SID [18] | 34.81/0.944 | 34.71/0.917 | 32.73/0.843 | 35.86/0.937
| LUD-VAE [30] | 34.91/0.944 | 34.82/0.926 | - |  36.99/0.955
Supervised | DnCNN [23] | 37.73 / 0.943 | 37.61 / 0.941 | - | -
Table 1: Quantitative comparisons (PSNR(dB)/SSIM) of our method and other real-world denoising methods including single image-based methods and dataset-based methods on SIDD, FMDD, PolyU datasets. The best results of the unsupervised approaches are marked in bold, while the second best ones are underlined. Noisy | DIP [24] | Self2Self [19] | NN+denoiser [29] | Baseline | Ours | Ground-truth
---|---|---|---|---|---|---
|  |  |  |  |  |
| 26.64/0.546 | 25.78/0.579 | 28.35/0.646 | 28.13/0.663 | 31.79/0.812 |
|  |  |  |  |  |
| 29.67/0.908 | 28.79/0.854 | 30.68/0.921 | 30.13/0.918 | 32.05/0.937 |
|  |  |  |  |  |
| 28.04/0.622 | 24.36/0.410 | 30.14/0.751 | 25.34/0.467 | 31.42/0.793 |

Figure 7:  Visual comparison of our method against other single image-based
denoising methods in SIDD validation and FMDD datasets. The PSNR/SSIM results
are reported under each image.

###  4.2 Evaluation on real-world noise

In our evaluation, we compare our method against several single image-based
denoising methods, including Self2Self [19], NN+denoiser [29], DIP [24], BM3D
[6], PD-denoising [31], NN+denoiser [29], scoreDVI [5], and APBSN-single [14].
For APBSN-single, we adapt the APBSN method from [14] to directly denoise a
single image. The strides of PD in training and testing are 5 and 2,
respectively. For NN+denoiser [29], we use its best version, which is NN+BM3D
for single image denoising. For the other methods, we either use the authors’
code or directly adopt their published results if available. We also include a
comparison with a baseline method (denoted as Baseline in Table) which is a
blind spot method with the same network architecture as our method but with a
fixed masking ratio of $\tau=0.5$. Additionally, we compare against dataset-
based methods including CVF-SID [18], LUD-VAE [30], and APBSN [14]. We also
provide a reference comparison with the supervised DNCNN [23]. The
quantitative comparisons are summarized in Tab. 1. Qualitative comparisons
among different single image-based methods on SIDD and FMDD are displayed in
Fig. 7. Additional visual comparisons are included in the supplementary
material. MASH shows a significant boost over the baseline, with an
improvement of about 2 dB for both SIDD datasets and 1.5 dB for the FMDD
dataset, highlighting the importance of our adaptive masking scheme and local
pixel shuffling. MASH yields competitive results compared to existing single
image-based methods. Our method excels in the FMDD dataset by outperforming
both the single-image and dataset-based methods, which encompass images with
varying noise levels and correlations. Fig. 6 shows the output of MASH on an
image from the SIDD validation dataset. Utilizing our adaptive masking scheme
and pixel local shuffling results in a significant improvement over the
baseline.

| SIDD | FMDD
---|---|---
Adaptive masking accuracy | 88.7 % | 92.4 %
Table 2: Adaptive masking accuracy on SIDD and FMDD datasets.  Adaptive masking ratio | Local pixel shuffling | SIDD | FMDD |
---|---|---|---|---
No | No | 33.12 | 32.25 |
No | Yes | 33.86 | 32.92 |
Yes | No | 34.45 | 33.56 |
Yes | Yes | 35.06 | 33.71 |
Table 3: Ablation of MASH components .

##  5 Ablations

We perform ablation studies to analyze the impact of each component of our
method.

###  5.1 Influence of Adaptive Masking

We evaluate the effectiveness of our adaptive masking scheme. When the
adaptive masking is not applied, we utilize a default masking ratio of
$\tau=0.5$. Tab. 3 demonstrates that adaptive masking yields a notable
performance enhancement for both the SIDD and FMDD datasets. We further
analyze the effects on each image individually by calculating the success rate
of our adaptive masking scheme. To do this, we denoise each image using the
baseline with three masking ratios: $\tau^{\text{high}}=0.8$,
$\tau^{\text{medium}}=0.5$, and $\tau^{\text{low}}=0.2$, and we store the best
result. Then, we denoise each image using MASH. In Tab. 2, we show how many
times our algorithm succeeds in selecting the optimal masking ratio that leads
to the best performance. Our adaptive masking is considered successful if it
corresponds to the masking ratio that results in optimal performance. Tab. 2
indicates that our adaptive masking succeeds in selecting the optimal masking
ratio in approximately 90% of cases. Further analysis on the robustness of
MASH hyperparameters and showcasing of failure cases can be found in our
Supplementary materials.

###  5.2 Influence of Local Pixel Shuffling

We validate the significance of local pixel shuffling in our method. As shown
in Tab. 3, there is an improvement of approximately 0.7 dB for both the SIDD
and FMDD datasets when applying local pixel shuffling.

###  5.3 Influence of the neighberhood size $s$

Tab. 4 shows an ablation of the neighborhood size $s$ conducted on the SIDD
valadition dataset. As $s$ increases, the performance improves then it starts
to saturate. We note that $s$ is an hyperparameter related to the local pixels
shuffling which is applied only if a high spatial correlated noise is
detected.

| $s=2$ | $s=3$ | $s=4$ | $s=5$ | $s=6$
---|---|---|---|---|---
PSNR | 34.65 | 34.85 | 35.06 | 35.15 | 35.09
Table 4: Ablation of the neighberhood size $s$ on the SIDD validation dataset.

###  5.4 Computational Efficiency

We provide an analysis of the inference time, model parameters and floating-
point operations per second (FLOPs) of the compared deep learning-based
methods. According to the results in Tab. 1 and Tab. 5, our method
demonstrates a balance between effectiveness and efficiency. It is worth
noting that there is room for speed enhancement with MASH. For example, the
initial training phase to determine the optimal masking could be performed in
parallel rather than sequentially.

Method | Infer. time (s) | Params (M) | FLOPs (G) |
---|---|---|---|---
DIP [24] | 146.2 | 13.4 | 31.06 |
Self2Self [19] | 3546.5 | 1.0 | 9.55 |
NN+denoiser [29] | 897.6 | 13.4 | 31.06 |
APBSN-single [14] | 121.4 | 3.66 | 234.63 |
ScoreDVI [5] | 81.2 | 13.5 | 37.87 |
Baseline | 24.6 | 0.99 | 11.44 |
Ours | 75.3 | 0.99 | 11.44 |
Table 5: Efficiency comparisons of deep learning-based methods under the input
size 256 × 256 × 3.

##  6 Conclusion

We have introduced MASH, a single image denoising method that leverages the
blind spot denoising framework. Our approach includes an analysis to detect
and alleviate the impact of noise correlation. We demonstrated that the
masking ratio plays a critical role in denoising performance, especially in
the presence of correlated noise. Building on this analysis, we proposed a
method to automatically estimate the level of noise correlation. Additionally,
we introduced a technique to directly de-correlate noise in the input image by
shuffling pixels with similar denoised color intensities. As a result, our
method, MASH, achieves state-of-the-art results compared to existing test-time
training approaches across various public benchmarks.

Acknowledgements We acknowledge the support of the SNF project number
200020_200304.

## References

  * Abdelhamed et al. [2018] Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown.  A high-quality denoising dataset for smartphone cameras.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Bao and Ji [2013] Jian-Feng Cai Bao, Chenglong and Hui Ji.  Fast sparsity-based orthogonal dictionary learning for image restoration.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2013.
  * Buades Antoni and Morel [2011] Bartomeu Coll Buades Antoni and Jean-Michel Morel.  Non-local means denoising.  In _Image Processing On Line 1_ , 2011.
  * Burger et al. [2012] Harold C Burger, Christian J Schuler, and Stefan Harmeling.  Image denoising: Can plain neural networks compete with bm3d?  In _2012 IEEE conference on computer vision and pattern recognition_ , pages 2392–2399. IEEE, 2012.
  * Cheng et al. [2023] Jun Cheng, Tao Liu, and Shan Tan.  Score priors guided deep variational inference for unsupervised real-world single image denoising.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12937–12948, 2023.
  * Dabov et al. [2009] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Bm3d image denoising with shape-adaptive principal component analysis.  In _SPARS’09-Signal Processing with Adaptive Sparse Structured Representations_ , 2009.
  * Franzen [1999] Rich Franzen.  Kodak lossless true color image suite.  _source: http://r0k. us/graphics/kodak_ , 4(2):9, 1999.
  * Geonwoon et al. [2021] Jang Geonwoon, Lee Wooseok, Son Sanghyun, and Lee Kyoung Mu.  C2n: Practical generative noise modeling for real-world denoising.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2350–2359, 2021.
  * Gu and Timofte [2019] Shuhang Gu and Radu Timofte.  A brief review of image denoising algorithms and beyond.  _Inpainting and Denoising Challenges_ , pages 1–21, 2019.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Huang et al. [2021] Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu.  Neighbor2neighbor: Self-supervised denoising from single noisy images.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14781–14790, 2021.
  * J. et al. [2018] Lehtinen J., Munkberg J., Hasselgren J., Laine S., Karras T., and T. Aittala M. & Aila.  Noise2noise: Learning image restoration without clean data.  In _arXiv preprint arXiv:1803.04189_ , 2018.
  * Krull Alexander and Jug. [2019] Tim-Oliver Buchholz Krull Alexander and Florian Jug.  Noise2void-learning denoising from single noisy images.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019.
  * Lee et al. [2022] Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee.  Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17725–17734, 2022.
  * Li et al. [2023] Junyi Li, Zhilu Zhang, Xiaoyu Liu, Chaoyu Feng, Xiaotao Wang, Lei Lei, and Wangmeng Zuo.  Spatially adaptive self-supervised learning for real-world image denoising.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9914–9924, 2023.
  * Michael and Aharon [2006] Elad Michael and Michal Aharon.  Image denoising via sparse and redundant representations over learned dictionaries.  In _IEEE Transactions on Image processing 15.12_ , 2006.
  * Nam et al. [2016] Seonghyeon Nam, Youngbae Hwang, Yasuyuki Matsushita, and Seon Joo Kim.  A holistic approach to cross-channel image noise modeling and its application to image denoising.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1683–1691, 2016.
  * Neshatavar et al. [2022] Reyhaneh Neshatavar, Mohsen Yavartanoo, Sanghyun Son, and Kyoung Mu Lee.  Cvf-sid: Cyclic multi-variate function for self-supervised image denoising by disentangling noise from image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17583–17591, 2022.
  * Quan et al. [2020] Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji.  Self2self with dropout: Learning self-supervised denoising from single image.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1890–1898, 2020.
  * S. et al. [2014] Gu S., Zhang L., and X. Zuo W. & Feng.  Weighted nuclear norm minimization with application to image denoising.  In _In Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014.
  * Shayan et al. [2022] Kousha Shayan, Maleky Ali, Brown Michael S, and Brubaker Marcus A.  Modeling srgb camera noise with normalizing flows.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17463–17471, 2022.
  * Talebi and Milanfar [2013] Hossein Talebi and Peyman Milanfar.  Global image denoising.  _IEEE Transactions on Image Processing_ , 23(2):755–768, 2013.
  * Thakur et al. [2019] Rini Smita Thakur, Ram Narayan Yadav, and Lalita Gupta.  State-of-art analysis of image denoising methods using convolutional neural networks.  _IET Image Processing_ , 13(13):2367–2380, 2019.
  * Ulyanov Dmitry and Lempitsky [2018] Andrea Vedaldi Ulyanov Dmitry and Victor Lempitsky.  Deep image prior.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018.
  * Wang et al. [2022] Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han.  Blind2unblind: Self-supervised image denoising with visible blind spots.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2027–2036, 2022.
  * Xie Yaochen and Ji [2020] Zhengyang Wang Xie Yaochen and Shuiwang Ji.  Noise2same: Optimizing a self-supervised bound for image denoising.  In _Advances in Neural Information Processing Systems 33_ , 2020.
  * Zhang et al. [2010] Lei Zhang, Weisheng Dong, David Zhang, and Guangming Shi.  Two-stage image denoising by principal component analysis with local pixel grouping.  _Pattern recognition_ , 43(4):1531–1549, 2010.
  * Zhang et al. [2019] Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard.  A poisson-gaussian denoising dataset with real fluorescence microscopy images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11710–11718, 2019.
  * Zheng et al. [2020] Dihan Zheng, Sia Huat Tan, Xiaowen Zhang, Zuoqiang Shi, Kaisheng Ma, and Chenglong Bao.  An unsupervised deep learning approach for real-world image denoising.  In _International Conference on Learning Representations_ , 2020.
  * Zheng et al. [2022] Dihan Zheng, Xiaowen Zhang, Kaisheng Ma, and Chenglong Bao.  Learn from unpaired data for image restoration: A variational bayes approach.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(5):5889–5903, 2022.
  * Zhou et al. [2020] Yuqian Zhou, Jianbo Jiao, Haibin Huang, Yang Wang, Jue Wang, Honghui Shi, and Thomas Huang.  When awgn-based denoiser meets real noises.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 13074–13081, 2020.
  * Zontak et al. [2013] Maria Zontak, Inbar Mosseri, and Michal Irani.  Separating signal from noise using patch recurrence across scales.  In _proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1195–1202, 2013.