# Data driven weather forecasts trained and initialised directly from
observations

Anthony McNally Christian Lessig Peter Lean Eulalie Boucher Mihai Alexe Ewan
Pinnington Matthew Chantry Simon Lang Chris Burrows Marcin Chrust Florian
Pinault Ethel Villeneuve Niels Bormann Sean Healy European Centre for Medium-
Range Weather Forecasts (ECMWF)

###### Abstract

Skilful Machine Learned (ML) weather forecasts have challenged conventional
approaches to numerical weather prediction (NWP), demonstrating competitive
performance compared to traditional physics-based approaches. Existing data-
driven systems have been trained to forecast future weather by learning from
long historical records of past weather, typically provided by reanalyses such
as ECMWF’s ERA5. These datasets have been made freely available to the wider
research community, including the commercial sector, which has been a major
factor in the rapid rise of ML forecast systems and the impressive levels of
accuracy they have achieved. However, both historical reanalyses used for
training and real-time analyses used for initial conditions are produced by
data assimilation, essentially an optimal blending of observations with a
traditional physics-based forecast model. As such, many ML forecast systems
have an implicit, unknown and unquantified dependence on the physics-based
models they seek to challenge. Here we propose a new and radical approach to
weather forecasting, by training a neural network to predict future weather
purely from historical observations with no dependence on a physics-based
model or reanalysis datasets. We use raw observations (level-1) to initialise
a model of the atmosphere (in observation space) learned directly from the
observations themselves. Forecasts of crucial weather-related parameters (such
as surface temperature and wind) are obtained by predicting weather parameter
observations (such as SYNOP surface data) at future times and arbitrary
locations. We present preliminary results on forecasting observations 12-hours
into the future. These already demonstrate successful learning of the time
evolution of the physical processes captured in real observations. We argue
that this new approach, by staying purely in observation space avoids many of
the challenges of traditional data assimilation, can exploit a wider range of
observations and is readily expanded to simultaneous forecasting of the full
Earth system (atmosphere, land, ocean and atmospheric composition).

Keywords: Numerical weather prediction, observations, machine learning, neural
networks, coupled Earth system, atmospheric composition

## The current use of observations in NWP

The many millions of meteorological observations measured each day require
highly sophisticated data assimilation (DA) systems to transform the raw
measurements made at irregular times and locations into a best estimate of the
state of the atmosphere on a regular grid and into the variables required to
initialise physics-based forecast models. Data assimilation is based upon very
sound theoretical principles and performs this task by combining different
sources of information in an optimal way, respecting the uncertainty in each
element. The European Centre for Medium-range Weather Forecasts (ECMWF) and
others have been extremely successful in the development of advanced global
data assimilation systems (e.g. 4D-Var, [11]) that exploit a large amount of
information from the observations and generate highly skilful initial
conditions. These DA systems, in addition to initialising real-time
operational forecast models, have been used to create state of the art
historical weather records such as ERA5 ([6]) used extensively for climate
research and, more recently, training ML forecast models. However, DA is an
extremely challenging task because forecast models demand initialisation on a
fine spatial grid over the entire globe (e.g. 9 km for the ECMWF Integrated
Forecast System) and many vertical levels, even in regions where no
observations are available. The challenge is further compounded as physics-
based models require initialisation in physical meteorological variables (e.g.
temperature, humidity, and wind) while the overwhelming majority of weather
observations come from passive satellite sensors which measure thermal
infrared and microwave radiance signals emanating from broad vertical layers
of the atmosphere. In data assimilation systems, the extraction of
temperature, humidity or wind requires a detailed understanding of complex
radiative transfer processes in the atmosphere, including a prior knowledge of
the current atmospheric state. These downward viewing satellites, by their
nature, also cannot provide information on fine vertical scales (the existence
of a so-called null space, is discussed in [13], atmospheric structures to
which the radiances are effectively blind). To meet the demands of forecast
model initialisation, the data assimilation system must very carefully blend
the satellite radiances and other observations with gridded fine-scale
information from a previous forecast trajectory, known as a background. This
operation requires a highly exacting characterisation of large, complex and
often poorly known observation and background error covariances. In practice,
and particularly in a real-time operational DA context, many compromises have
to be made in the exploitation of observations. One of the most significant is
that large fractions of the global observing system are not actually used for
operational NWP. Observations typically need to be heavily thinned in both
space and time, discarded in certain meteorological conditions when
uncertainties become inconsistent with our error covariances, or even
discarded completely when the physics of the measurement is extremely
difficult to relate to the model variables.

## Does machine learning offer an alternative?

Much of the data assimilation complexity described above stems from the fact
that physics-based forecast models demand extremely accurate initial
conditions for all meteorological variables on a fine spatial grid. The
problem is effectively ill-posed, in that there is insufficient information
contained in the observations to constrain all parameters of this 3D grid. The
strategy that has traditionally been employed by NWP centres to produce
accurate weather forecasts is to build models which represent the real
atmosphere as comprehensively as possible, explicitly describing a myriad of
physical processes and interactions at every grid point over the entire globe,
from the surface up to the mesosphere [2]. Over recent decades, we have
witnessed impressive accuracy gains in global NWP as ever more powerful
supercomputers have allowed physics-based models to operate at ever higher
spatial resolutions, and the representation of unresolved processes has
improved. And as model resolution has increased, so has the burden placed upon
data assimilation systems to provide the initial conditions with the required
accuracy and detail.

However, more recently it has been demonstrated that ML systems operating with
far fewer variables than physics-based models and on significantly coarser
spatial grids are capable of producing highly skilful medium-range forecasts
of important weather parameters [3, 8, 10, 9]. Given the dependence of these
systems upon reanalysis datasets (for training) and real-time analyses (for
initialisation), both produced by data assimilation (with the aforementioned
challenges and limitations), we ask the intriguing question if ML forecast
models could learn and be initialised directly from observations. This would
obviate the need for DA systems to map raw observations (e.g. satellite
radiances) to a fine grid of unmeasured physical variables (e.g. temperature)
dictated by the requirements of the physics-based model and one could instead
directly use the information in the observations to produce a forecast. If
training of a ML model directly from observations is successful, large
fractions of the data assimilation process described above could be completely
reconsidered. For real-time applications like operational NWP (both physics-
based or the current generation of analysis-based data driven models) this
could allow forecasts to be produced in a significantly timelier manner,
effectively as soon as observations are available. But crucially, observation-
based prediction in observation space could also be more accurate. A model
learned directly from observations might be more primitive than comprehensive
physics-based systems, but if it allows the exploitation of more observations
(more than are used in operational analyses and reanalyses) with fewer
approximations and avoids the null-space problem, the end result has the
potential to be more accurate.

In the prototype experiments described here, a machine learning algorithm has
been trained to predict future observations from long time series of
historical observations. A key challenge is to learn the spatial and temporal
correlations that exist within a given observation type, but also between
different measurement systems, and encapsulate these within the internal
latent space of the machine learning model. Learning the relationships between
different observations (both the complementarities and the redundancies) from
the historical data allows the network at inference time to predict future
observations. However, predicting future values of satellite radiances at
different microwave and infrared wavelengths would not constitute a useful
weather forecast. To be useful, the network must be able to predict future
weather parameter observations such as SYNOP surface measurements and
radiosondes measurements in the upper atmosphere, by learning the correlations
between these sparse in-situ data and the dense global satellite radiance
observations. The spatial and temporal relationships between weather station
data and the satellite radiances, once learned from real observation
locations, must then be generalised to arbitrary (user defined) locations to
allow weather parameter prediction wherever required.

## Observation training data

Our first prototype experiments have used a limited set of observations as
detailed in Table 1. The table highlights the instruments, variables and
period of each dataset used and specifies whether the data is used
prognostically or diagnostically. Prognostic observations are passed both to
the network as inputs and are predicted as an output, whilst diagnostic
observations are only predicted but are not part of the inputs (and are thus
derived from other input sources).

Instrument | Satellite | Period | Variables | Use
---|---|---|---|---
AMSUA | METOP-B | 2013- present | 15 channels | Prognostic
ATMS | NPP, NOAA-20 | 2012 - present | 22 channels | Prognostic
IASI | METOP-B | 2013 - present | 17 channels | Prognostic
AVHRR | METOP-B | 2013 - present | Visible channel | Prognostic
SYNOPS | N/A | 1979 – present | ps, t2m, rh2m, u10, v10 | Diagnostic
Table 1: A description of the current set of observations in the training
database. A glossary of satellite instrument acronyms is contained in appendix
A.

Passive microwave radiance measurements from ATMS and AMSU sensors (see
glossary in appendix A for instrument acronym meanings) onboard low Earth
orbiting (LEO) satellites provide deep layer information on temperature (from
channels around 50 GHz) and on water vapour (from channels around 183 GHz).
Microwave window channels also provide information on surface temperature,
albeit potentially contaminated by highly variable surface emissivity values.
Similar deep layer temperature and humidity information is provided by passive
infrared radiances measured by the IASI instrument, also carried by LEO
satellites. Emissivity over land and ocean is less variable and closer to
unity at infrared wavelengths, providing more direct information on surface
temperature. Infrared radiances display an acute (and highly nonlinear)
sensitivity to the presence of clouds. While this sensitivity significantly
limits the use of infrared data in cloudy regions by conventional data
assimilation systems, it is not an obstacle for a machine learning system. On
the contrary, the spatial and temporal patterns in the cloudy infrared data
provide the network with invaluable information on the trajectory of weather
systems as they move around the globe. Similar information on the movement of
weather systems is provided by AVHRR visible imagery, the reflected solar
illumination providing better thermal contrast for cloud evolutions over cold
higher-latitude surfaces compared to the infrared and free from the
potentially interfering effects of humidity. Figure 1 (a)-(c) shows that all
of the satellite observations provide near global spatial coverage within a
12-hour window and frequent temporal sampling when carried on multiple LEO
satellites. Also, in situ observations of surface temperature and wind from
SYNOP stations are included in the training. Coverage of these data (see
Figure 1 (d)) can be very sparse (e.g. over oceans and uninhabited land areas)
and even over populated areas it can be rather inhomogeneous. However, these
weather parameter observations provide an important strategic resource to the
training process, allowing the network to forecast weather parameters, by
exploiting correlations learned between these and satellite radiance
observations.

(a) ATMS microwave radiances

(b) IASI infrared radiances

(c) AVHRR visible reflectances

(d) SYNOP surface measurements of 2m temperature and 10m wind

Figure 1: Typical examples of the data coverage provided by ATMS microwave
radiances (a), IASI infrared radiances (b), AVHRR visible reflectances (c) and
SYNOP surface measurements of 2m temperature and 10m wind (d) within a 12-hour
window.

## Prototype neural network

The prediction of future observations from historical observations should be
reasonably agnostic to the choice of network architecture, as long as the
training data is sufficiently rich. For our prototype experiments, we use a
transformer-based neural network depicted schematically in Figure 2. The
network takes as input information subdivided into tokens that represent data
in a small space-time neighbourhood. The tabular form of the data are mapped
through learnable embedding networks to a latent vector representation. For
each observation, all channels are treated together and embedded together with
their associated location as well as observation time. For radiosondes,
weather parameters on different standard pressure levels are treated in a
similar way. The set of vectors from all neighbourhoods and all observations
is the input to the transformer ‘backbone’ neural network, in which the data
interacts through the transformer’s attention mechanism [14]. The attention
mechanism is a key component of transformer models, learning how each element
of the representation relates to the other elements. Here it learns how
spatial, temporal and observation-specific information are related.

Figure 2: A schematic representation of the transformer neural network used in
the prototype experiments. On the left we see portions of the satellite
observations being randomly masked, and during training the network being
challenged to predict the values of the masked data.

During training, randomly selected parts of the data are masked with only the
space-time coordinates of the observation being fed to the embedding networks.
The pre-training task is to predict the corresponding withheld observed
values. The network is then fine-tuned to make 12-hour predictions,
essentially using observations in one 12-hour assimilation time window as
input and predicting the observations that would be obtained in the next
12-hour window. This selection of time window is somewhat arbitrary but chosen
here to allow the exploitation of pre-existing infrastructure supporting the
ECMWF operational data assimilation system. It also allows some comparisons to
be made between the network predictions and observations in the 12-hour
4D-Var.

## Results of the prototype experiments

An example of the successful prediction of future observations is shown
qualitatively in Figure 3 for IASI infrared radiances. The future IASI
radiances predicted (b) by the network are clearly physically plausible in
comparison to real measurements (c). The predicted values are a smoothed
representation of the real data, but otherwise capture the large-scale
patterns and indeed many of the finer scale cloud patterns in a very
reasonable way. What is also evident in Figure 3 is that the network is
correctly predicting the advection of weather systems (for example the
easterly movement over 12 hours of the Southern Hemisphere frontal cloud
pattern contained the dashed box), but also more complex atmospheric
evolutions such as those shown within the solid box over the North Pacific.
Here we see more than simple advection with the formation of a hook-shaped
cloud feature that was not in the initial conditions (observations) 12 hours
previously.

(a) Input IASI channel 921 radiances

(b) Predicted IASI channel 921 radiances

(c) Target IASI channel 921 radiances

Figure 3:  An example of the network predicting 12-hours of IASI window
channel radiances (channel 921 around 10 microns) for a case on October 17th
2022 (00z) using the measurements from the previous 12-hour window as input.
In this IASI channel, light shades correspond to cold features (e.g. clouds
and the Antarctic continent) and dark shades to warm features (e.g. the
satellite viewing warm surfaces unobscured by clouds).

Over equatorial Africa there is some evidence that the predictions
successfully capture changes to the deep convection that occur over 12 hours
in the real observations. The realism of the forecasted satellite radiance
evolutions is similarly evident in the microwave radiance and visible
reflectance satellite data (not shown).

In the results of the prototype experiments we also see evidence that the
network is able to make skillful predictions that exploit the correlations
between different observing systems contained in the historical training data.
Figure 4 shows 12 hours of real AVHRR data provided as input to the
prediction, where the western hemisphere is mostly in darkness and the
reflectances are zero (apart from at high summer northern latitudes where
there is solar illumination and some sporadic calibration anomalies). Yet, in
the subsequent 12-hour window the network successfully predicts very plausible
reflectances for the western hemisphere which is now illuminated by the sun.
This prediction skill cannot have originated from the AVHRR reflectances in
the western hemisphere used as input 12 hours previously (as these were zero)
so the network must have drawn upon information from other observing systems.
The most likely explanation is that the network has learned the very high
correlation between clouds causing bright solar reflections in the AVHRR
visible data and clouds causing strong absorption signals in the IASI infrared
measurements (similar to those seen in Figure 3).

(a) Input AVHRR visible reflectances

(b) Predicted AVHRR visible reflectances

(c) Target AVHRR visible reflectances

Figure 4:  Example of input (a), predicted (b) and target (c) METOP-B AVHRR
visible reflectances in the next 12-hour window on October 17th 2022 (00z),
along with the target truth. Note the accurate prediction of visible cloud
features in the western hemisphere despite this hemisphere being in darkness
for the input window and having no AVHRR information.

Finally, we present evidence suggesting that the network can successfully
produce plausible weather parameter forecasts exploiting the learned
correlations between real SYNOP measurements and satellite observations. In
Figure 5 we see an example of predicted 10m SYNOP winds (upper panels)
compared to the real SYNOP wind observations (lower panels) in the target
12-hour window during an event on February 18th 2022 (a deep low-pressure area
causing strong cyclonic winds over continental Europe and even an intense
Medicane system in the Eastern Mediterranean causing strong winds on the North
African coast). While there are clearly some differences in magnitude and
direction between the forecast winds and the real observed wind vectors, the
overall picture is a strikingly realistic representation of a rather complex
synoptic situation. Furthermore, in this rapidly evolving synoptic situation,
it is particularly impressive that the forecasted winds manage to follow the
rapid changes in intensity and direction of the observed winds.

It must be reiterated that SYNOP observations are diagnostic in these
experiments, in that no real wind observations are provided as input to the
forecast. The forecast winds in Figure 5 are completely diagnosed from the
satellite radiances that were observed in the previous 12 hours, exploiting
the correlations between historical SYNOP and satellite radiances that were
learned during the network training. An examination of the input satellite
radiances suggests that during this particular case, much of the wind
information was being inferred from ATMS. The microwave brightness
temperatures provide clear information on the strong thermal gradients and the
passage of the main synoptic features through the region.

Figure 5:  Example of 10m winds inferred from satellite brightness
temperatures alone. Predicted SYNOP 10m wind (upper panels) and verifying
target SYNOP 10m wind (lower panels) at 9UTC and 18UTC for a case on February
18th 2022.

## Discussion and future perspectives

Here we have proposed a new and radical approach to weather forecasting by
training a neural network to perform skilful weather forecasts purely from
observations with no influence of a physics-based model, analysis or
reanalysis dataset. In conventional NWP, we go to extraordinary lengths to
convert many millions of observations to gridded fields of unmeasured
quantities, to initialise conventional forecasting models with an imperfect
representation of physics. The influence of the observations upon the forecast
quality effectively stops at the initial conditions and limits the information
from observations that can be exploited for a forecast. In any time-lapse
animation of satellite cloud imagery, one sees a graphic illustration of
exactly how our atmosphere actually evolves in time. In this new approach we
use observations to initialise a model of the atmosphere (in observation
space) learned directly from the observations themselves. Unlike other recent
studies (e.g. [15, 7, 16]) there is no explicit data assimilation step.

There are a number of examples of extremely interesting work reported in the
literature where observations are predicted with machine learning, but we feel
it is important to point out the differences to this approach. Originally
using feature tracking and extrapolation (e.g. [4]), more recent nowcasting
studies have demonstrated success using machine learning techniques [12, 5,
17]. However, these nowcasting applications only ingest and predict a single
variable, and capturing only a small fraction of the complete atmospheric
state restricts the effective forecast range to just a few hours ahead. [1]
were able to extend the skilful forecast range up to a day ahead by augmenting
the observational data with gridded NWP analysis fields which were used as
both input and targets in the training. But here we use no NWP analysis data,
instead including a far wider range of observational data sources to provide
the network with richer information about the global temperature and humidity
structure of the atmosphere.

The results presented here show that, even with a prototype neural network,
realistic predictions out to 12-hours can be obtained using a rather limited
set of satellite and in-situ training observations. We see evidence that the
network is learning not just simple advection of weather systems, but also
more complex evolutions in time and space. Finally, through learning robust
relationships between different observation types, the approach is able to
generate extremely realistic forecasts of useful weather parameters by
predicting future weather parameter observations such as SYNOP winds and
temperatures.

Of course, on the basis of these very preliminary results we cannot speculate
upon how realistic longer-range predictions directly from observations might
be. Established data driven models (such as the AIFS, GraphCast and Pangu-
Weather) trained from reanalysis data have clearly demonstrated this longer-
range capability so we have some reason for optimism. We are currently working
towards this by building scalable and more sophisticated neural networks
trained and initialised with all operationally used observations.

If successful, a particularly exciting prospect of this approach is that it is
readily (indeed immediately) extendable to forecasts of the full Earth system.
In conventional NWP, completely different physics-based models for each Earth
component are used (e.g. ocean and atmosphere), requiring completely separate
data assimilation systems to support them, and complex coupling technology to
allow communication between different components. In the fully data-driven
approach presented here, the network is completely agnostic (like the
observations themselves) to which Earth component the observations have come
from. Providing they are in the same training data, the network is free to
learn relationships between ocean measurements (such as Argo floats or
drifters) and satellite measurements of the atmosphere and ocean surface (such
as radiances, scatterometry and altimetry). By identifying and exploiting
those correlations which improve forecast accuracy, direct observation
prediction is implicitly and instantly capable of fully coupled forecasting.
In a similar manner, observations of atmospheric composition (e.g. aerosol or
greenhouse gas concentration) can also be simultaneously introduced into the
training data, allowing the network to learn to predict the future evolution
of these, exploiting spatial and time correlations with other (meteorological)
observations in the training data (such as wind observations).

Finally, we see opportunities for direct observation prediction to exploit
additional observations that are currently not used by conventional data
assimilation systems. For example, satellite radiance measurements in the
visible part of the spectrum have such complex radiative transfer that they
are not yet assimilated in most global NWP systems. Yet in any animation of
visible imagery one can clearly see the movement of weather patterns around
the globe, and it seems entirely plausible that the network would readily
exploit this information in its predictions of weather parameters without
needing a detailed physics-based knowledge of the observation origin.

## Appendix A: Glossary of satellite sensor acronyms

AMSU | Advanced Microwave Sounding Unit (carried by NOAA and EUMETSAT satellites)
---|---
ATMS | Advanced Technology Microwave Sounder (carried by NOAA satellites)
IASI | Infrared Atmospheric Sounding Interferometer (carried by EUMETSAT satellites)
AVHRR | Advanced Very High-Resolution Radiometer (carried by NOAA and EUMETSAT satellites)
SYNOP | Surface Synoptic Observations (WMO code type for this measurement)

## References

  * [1] Marcin Andrychowicz, Lasse Espeholt, Di Li, Samier Merchant, Alexander Merose, Fred Zyda, Shreya Agrawal and Nal Kalchbrenner  “Deep learning for day forecasts from sparse observations”  In _arXiv preprint arXiv:2306.06079_ , 2023
  * [2] Peter Bauer, Alan Thorpe and Gilbert Brunet  “The quiet revolution of numerical weather prediction”  In _Nature_ 525.7567 Nature Publishing Group UK London, 2015, pp. 47–55
  * [3] Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu and Qi Tian  “Accurate medium-range global weather forecasting with 3D neural networks”  In _Nature_ 619.7970 Nature Publishing Group UK London, 2023, pp. 533–538
  * [4] Neill E Bowler, Clive E Pierce and Alan W Seed  “STEPS: A probabilistic precipitation forecasting scheme which merges an extrapolation nowcast with downscaled NWP”  In _Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography_ 132.620 Wiley Online Library, 2006, pp. 2127–2155
  * [5] Lasse Espeholt, Shreya Agrawal, Casper Sønderby, Manoj Kumar, Jonathan Heek, Carla Bromberg, Cenk Gazen, Rob Carver, Marcin Andrychowicz and Jason Hickey  “Deep learning for twelve hour precipitation forecasts”  In _Nature communications_ 13.1 Nature Publishing Group, 2022, pp. 1–10
  * [6] Hans Hersbach et al.  “The ERA5 global reanalysis”  In _Quarterly Journal of the Royal Meteorological Society_ 146.730, 2020, pp. 1999–2049  DOI: https://doi.org/10.1002/qj.3803
  * [7] Langwen Huang, Lukas Gianinazzi, Yuejiang Yu, Peter D Dueben and Torsten Hoefler  “Diffda: a diffusion model for weather-scale data assimilation”  In _arXiv preprint arXiv:2401.05932_ , 2024
  * [8] Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen and Weihua Hu  “Learning skillful medium-range global weather forecasting”  In _Science_ 382.6677 American Association for the Advancement of Science, 2023, pp. 1416–1421
  * [9] Simon Lang, Mihai Alexe, Matthew Chantry, Jesper Dramsch, Florian Pinault, Baudouin Raoult, Mariana CA Clare, Christian Lessig, Michael Maier-Gerber and Linus Magnusson  “AIFS-ECMWF’s data-driven forecasting system”  In _arXiv preprint arXiv:2406.01465_ , 2024
  * [10] Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Timo Ewalds, Andrew El-Kadi, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, Remi Lam and Matthew Willson  “GenCast: Diffusion-based ensemble forecasting for medium-range weather”  In _arXiv preprint arXiv:2312.15796_ , 2023
  * [11] F. Rabier, H. Järvinen, E. Klinker, J.-F. Mahfouf and A. Simmons  “The ECMWF operational implementation of four-dimensional variational assimilation. I: Experimental results with simplified physics”  In _Quarterly Journal of the Royal Meteorological Society_ 126.564, 2000, pp. 1143–1170  DOI: https://doi.org/10.1002/qj.49712656415
  * [12] Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem and Sam Madge  “Skilful precipitation nowcasting using deep generative models of radar”  In _Nature_ 597.7878 Nature Publishing Group UK London, 2021, pp. 672–677
  * [13] Clive D Rodgers  “Retrieval of atmospheric temperature and composition from remote measurements of thermal radiation”  In _Reviews of Geophysics_ 14.4 Wiley Online Library, 1976, pp. 609–624
  * [14] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser and Illia Polosukhin  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [15] Anna Vaughan, Stratis Markou, Will Tebbutt, James Requeima, Wessel P Bruinsma, Tom R Andersson, Michael Herzog, Nicholas D Lane, J Scott Hosking and Richard E Turner  “Aardvark Weather: end-to-end data-driven weather forecasting”  In _arXiv preprint arXiv:2404.00411_ , 2024
  * [16] Yi Xiao, Lei Bai, Wei Xue, Kang Chen, Tao Han and Wanli Ouyang  “Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation”, 2024
  * [17] Yuchen Zhang, Mingsheng Long, Kaiyuan Chen, Lanxiang Xing, Ronghua Jin, Michael I Jordan and Jianmin Wang  “Skilful nowcasting of extreme precipitation with NowcastNet”  In _Nature_ 619.7970 Nature Publishing Group UK London, 2023, pp. 526–532


# SUMD: Super U-shaped Matrix Decomposition Convolutional neural network for
Image denoising

###### Abstract

In this paper, we propose a novel and efficient CNN-based framework that
leverages local and global context information for image denoising. Due to the
limitations of convolution itself, the CNN-based method is generally unable to
construct an effective and structured global feature representation, usually
called the long-distance dependencies in the Transformer-based method. To
tackle this problem, we introduce the matrix decomposition module(MD) in the
network to establish the global context feature, comparable to the Transformer
based method performance. Inspired by the design of multi-stage progressive
restoration of U-shaped architecture, we further integrate the MD module into
the multi-branches to acquire the relative global feature representation of
the patch range at the current stage. Then, the stage input gradually rises to
the overall scope and continuously improves the final feature. Experimental
results on various image denoising datasets: SIDD, DND, and synthetic Gaussian
noise datasets show that our model(SUMD) can produce comparable visual quality
and accuracy results with Transformer-based methods.

Index Terms—  denoising, image restoration, CNN

##  1 Introduction

Image denoising is a fundamental and crucial task that is always used in the
other high-level computer vision program as the front steps, receiving the
attention of the academic circles. The main challenge in this task is to
recover the clean image from the noisy image, and the problem can be modeled
as:

$\displaystyle y$ $\displaystyle=$ $\displaystyle x+n$ (1)

Where x, y, and n represent the clean image, noisy image, and noise matrix,
respectively. Previous denoising methods can be divided into two mainstreams,
and one is the traditional algorithm, which usually uses hypothetical prior
information to address the problem, such as BM3D and NLM.

Following the development of deep learning, Methods based on the CNN and
Transformer architecture use the amount of clean-noisy training pairs to
acquire the implicit noise distribution and image priors for denoising.
Recently, the Transformer-based methods have been used in many vision tasks
and acquired notable performance due to advantages in long-distance dependent
information. Combining the advantages of the CNN-based method in local
representations and the Transformer-based methods in long-distance dependent
information to restore high-quality and detailed images becomes a vital issue.

Fig. 1: Overview of our SUMD architecture,containing two stages. We adopt CSFF
and SAM modules from MPRNet [1].

This paper proposes SUMD, an efficient CNN-based architecture, to combine the
advantages of the two methods mentioned in the previous paragraph for image
denoising. We introduce the MD module to acquire the global context
information to achieve these goals. Specifically, we reconstructed the low-
rank information from the image to replace the long-distance dependent
information extracted from the image.

The MD module uses the non-negative matrix factorization method to reconstruct
structural information, where the decomposition process uses the ”one-step
gradient” method to optimize named in [2]. At the same time, the parameters
are initialized with a normal distribution and do not need to be saved that
reduces a significant number of parameters.

In order to further improve the low-rank global feature, we adopt the multi-
stage progressive denoising strategy and build a relative global context
feature within the current stage with different patch sizes through MD.
Thereby the SUMD gradually denoises the more accurate images. Other denoising
methods use the input patch to participate in the training process and add the
image pyramid to obtain high-level information. The SUMD needs the high-level
features and extracts the globally relative information using the non-
overlapping patches as input. For example, their global feature
representations are also different, with an image containing two independent
objects(e.g., hands and T-shirt). It is evident that directly constituting the
low-rank global representation will not pay attention to every detail for
every object that appeared in the input and generate uncertain errors.
Therefore, as mentioned earlier, while introducing multiple stages, the
network at different stages is also focused on constructing global features at
all levels in different scopes to achieve the goal of gradual recovery.

Each stage of our SUMD is built upon Unet [3] and keeps the same hierarchical
encoder-decoder structure and the skip-connection with add MD module. It
guarantees global context information and avoids the shortcomings of
Transformer-based methods that cannot effectively use the local features. It
should be noted that for low-level vision tasks, local features play a vital
role in the final denoising of the image. Overall, we summarize the
contributions of this paper included:

  * •

we propose SUMD, an effective and efficient multi-stage progressive network
for image denoising.

  * •

we assume the global context features as the low-rank information and utilize
the matrix decomposition method to reconstruct it to further eliminate the
interference of noise on the feature representation. The matrix decomposition
approach is capable of acquiring the contextually-enriched and globally
structured outputs.

  * •

SUMD achieves state-of-the-art performance or comparable performance in both
PSNR and SSIM indicators on many popular benchmarks.

##  2 related work

Traditional Methods Traditional noise reduction algorithms have been a hot
topic in the academic community in the past few decades. It has also promoted
the development of methods based on deep learning in recent years. Previous
work mainly used the prior hypothesis of the image or noise to design a model
matching the hypothesis to perform the denoising program. Among them, the use
of image prior includes sparse coding, non-local means, and the use of noise
distribution assumptions includes AWGN denoiser. Although, these classical
approaches have achieved good operating speed and accuracy results. However,
the complexity and robustness of the traditional model are insufficient caused
of the assumptions, and they cannot adapt to tasks in many situations.

Network architecture The deep learning-based model implicitly captures prior
information from the training on the amount of clean-noisy data pairs for
image denoising. Depending on their outstanding performance on the
generalization, DNN based model used in varieties of the complex practical
environment with a single training. With the various assumptions and novel
ideas, MLP, TNRD, DnCNN, RBDN, FFDNet, MIRNet are also developed with
promising denoising performance. MPRNet [1] relied on the multi-stage
progressive recovery strategy, through supervised attention module, cross-
stage feature fusion module, and the state-of-the-art performance had been
achieved in the fields of denoising, deblurring, and de-raining.

Recently, the Transformer architecture has gradually been applied to the field
of computer vision and has made outstanding contributions to the global
feature representation of long-range dependence. The Uformer [4], a
transformer-based model, achieves the best performance in the field of image
denoising. However, because relative position embedding still cannot replace
the accurate relative position information, the extraction of local features
is still poor.

##  3 Method

(a)

(b)

(c)

(d)

Fig. 2: (a) Top: The encoder block. Bottom: The decoder block. (b) The skip-connection between the encoder block and the decoder block. (c) BST Module (d) MD Module Table 1: The PSNR(dB) results of all competing methods on the three groups of test datasets. The best and second best results are highlighted in bold and Italic, respectively. Case | Datasets | Methods |
---|---|---|---
CBM3D | DnCNN-B | MemNet | FFDNet | UDNet | VDN | NBNet | Uformer | Ours
Case1 | Set5 | 27.76 | 29.85 | 30.10 | 30.16 | 28.13 | 30.39 | 30.59 | 29.96 | 30.63
LIVE1 | 26.58 | 28.81 | 28.96 | 28.99 | 27.19 | 29.22 | 29.40 | 29.36 | 29.52
BSD68 | 26.51 | 28.73 | 28.74 | 28.78 | 27.13 | 29.02 | 29.16 | 28.76 | 29.22
Case2 | Set5 | 26.34 | 29.04 | 29.55 | 29.60 | 26.01 | 29.80 | 29.88 | 29.53 | 30.11
LIVE1 | 25.18 | 28.18 | 28.56 | 28.58 | 25.25 | 28.82 | 29.01 | 29.00 | 29.14
BSD68 | 25.28 | 28.15 | 28.36 | 28.43 | 25.13 | 28.67 | 28.76 | 28.48 | 28.88
Case3 | Set5 | 27.88 | 29.13 | 29.51 | 29.54 | 27.54 | 29.74 | 29.89 | 29.62 | 30.01
LIVE1 | 26.50 | 28.17 | 28.37 | 28.39 | 26.48 | 28.65 | 28.82 | 28.71 | 28.93
BSD68 | 26.44 | 28.11 | 28.20 | 28.22 | 26.44 | 28.46 | 28.59 | 28.38 | 28.66

###  3.1 Overview

As shown in Fig 1, the proposed framework is composed of two horizontal stages
while each stage consists of the U-shaped encoder-decoder structure. The MD
module is assimilated into the Encoder/Decoder block, and the basic design of
the original Unet [3] architecture is modified to adapt to the requirement of
denoising tasks. Then, the original noisy image is divided into multiple non-
overlapping patches as the input in different stages. However, unlike the
image pyramid, the input maintains the original resolution, which makes this
stage pay more attention to the information within the patch. It facilitates
the MD module to construct the relative global feature map. Between the two
neighbor stages, we use the supervised attention module(SAM) and cross-stage
feature fusion module(CSFF), where these two modules come from [1].
Specifically, given a noisy image $I\in R^{3\times H\times W}$ at a certain
stage, the SUMD extracts a low-level feature map $F_{in}\in R^{C\times H\times
W}$ through the initial block composed of a combination of a $3\times 3$
convolution and the PReLU stacking two blocks. Subsequently, the low-level
features are delivered to the horizontal U-shaped structure to extract
information from different scales gradually and then restored to the original
resolution by degrees to obtain the decoded feature map $F_{out}\in R^{C\times
H\times W}$. The feature of the upper decoder in each stage is processed by
the SAM module and then merged into the initial feature map of the next stage.
The remaining encoder/decoder features of each scale will be adjusted by the
CSFF module and incorporated into encoder/decoder features of the next stage.
After the multi-stage processing, we apply a $3\times 3$ convolution on final
decoder feature of the last stage to obtain a residual image $F_{R}\in
R^{3\times H\times W}$. Finally, the denoising image is obtained by
$I_{out}=I_{in}+F_{R}$.

###  3.2 Horizontal U-shaped structure

To meet the needs of the denoising task, we modified the components of the
Unet [3] to boost the whole framework more efficient and simplified that
illustrated in Fig 2(a). Above all, we utilize a channel attention block in
[5] to weight the initial feature map $F_{in}\in R^{C\times H\times W}$ with
the channel self-attention mechanism for offering the channel information to
promote the function of the MD module. Then, the weighted feature input to the
MD module to extract the relative global low-rank structural features. The
channel attention module and MD module form a basic encoder block and apply in
the encoder part of each scale. Following the encoder block, we use a $4\times
4$ convolution with stride 2 to down-sample the feature maps and increase the
number of channels. For example, given the upper input feature $F_{0}\in
R^{C_{in}\times H\times W}$, the $k$-th scale of the encoder produces the
feature maps $F_{k}\in R^{(C_{in}+k\times C_{up})},k=0,1,2$. In order to
effectively extract high-level context features, we set three scales for the
image pyramid.

For the skip-connection between the encoder and the decoder at the same scale
shown in Fig 2(b), we adopt stacked basic structure blocks(BST) shown in Fig
2(c) to further process features and maintain the details. For the image
decoder, in addition to the features of the bottom scale that are direct as
the input, the decoder block will not only receive the feature map refined by
the skip-connection, but also the feature map of the previous scale will be
up-sampled using $2\times 2$ deconvolution with stride 2. The up-sampled
feature decreases the channels and doubles the size of the feature maps.
Unlike the encoder block, we switch the position of the channel attention
block and the MD module, which stimulate the relative global feature
selectively, to obtain better performance.

18.25 dB

Noisy

36.39 dB

VDN

36.72 dB

CycleISP

36.98 dB

MPRNet

36.97 dB

MIRNet

36.89 dB

Uformer

37.11 dB

Ours

Reference

Noisy

32.09 dB

VDN

32.27 dB

CycleISP

32.63 dB

MPRNet

32.54 dB

MIRNet

33.27 dB

NBNet

33.04 dB

Uformer

33.70 dB

Ours

Fig. 3: Visual results from SIDD and DND datasets. Top row: sample from the
SIDD datasets. Bottom row: sample from the DND datasets. Our results preserve
the details and reconstruct the textures.

###  3.3 Matrix Decomposition Module

Matrix decomposition is to factorize a matrix into the product of several sub-
matrices. The helpful information is usually assumed as low-rank information
for a noisy image and could be factorized into the low-rank matrix and the
noise matrix using matrix decomposition. For the CNN-based denoising method,
the convolution has pretty performance in capturing local features, but tough
to construct the global feature maps limited by the convolution kernel size
and receptive field. Therefore, the characteristic information obtained after
matrix decomposition perfectly complements the convolution. The formula of the
matrix decomposition can be expressed as:

$\displaystyle I_{n}$ $\displaystyle=$ $\displaystyle I_{l}+N=DC+N$ (2)

where $I_{n},I_{l},N\in R^{d\times hw}$ are the noisy image, the low-rank
reconstruction and the noisy matrix, respectively. Among them, the factor of
$I_{l}$ can be expressed as the product of the dictionary matrix $D\in
R^{d\times r}$ and the corresponding codes $C\in R^{r\times hw}$,$h,w$ is the
height and the width of input features.

The MD module is composed of preprocessing and matrix decomposition, shown in
Fig 2(d). We use the BST to process the input features further to adapt to the
matrix decomposition program as the pre-component. We firstly shallow the
channel from the $F_{in}$ to $F_{d}$. Then, using a series of linear
transformations consisting of the $1\times 1$ convolution and ReLU activation,
we map the output of the preprocessing to split the $F_{d}$ into the $k$
subspaces of the features following the assumption that a union of multiple
subspaces is hidden in $I_{l}$. After that, we utilize the Multiplicative
Update rules [6] to solve the non-negative matrix factorization(NMF) [7],
decomposing and reconstructing the features. During the optimization step, we
apply the one-step gradient to conquer the gradient vanishing, or gradient
explosion described in [2]. After that, we concat each decomposed feature and
put it into another $1\times 1$ convolution to recover the original channels.
The whole process can be expressed as:

$\displaystyle F_{k}=W_{k}F_{in},k=1,...,K$ $\displaystyle
D_{k}=MD(F_{k}),k=1,...,K$ (3) $\displaystyle
F_{out}=W_{out}(Concat(D_{1},...,D_{k})),k=1,...,K$

where $F_{in}\in R^{C_{in}\times h\times w}$, $W_{k}\in R^{C_{mid}\times
C_{in}}$ represents the input feature and the linear transformation for the
$k$-th subspace, respectively. MD is the process of matrix decomposition. The
input feature of the MD should be flattened along the H, W dimension. The
$D_{k}\in R^{C_{mid}\times hw}$ of sub results are concated and then be
transformed by the $W_{out}\in R^{C_{out}\times C_{mid\times K}}$. The
$F_{out}$ is the reconstructed feature of the MD module.

##  4 Experiments

Table 2: The PSNR(dB) result of all competing methods on AWGN noise cases of three test datasets. The best and second best results are highlighted in bold and Italic, respectively. Case | Datasets | Methods
---|---|---
CBM3D | DnCNN-B | MemNet | FFDNet | UDNet | VDN | NBNet | Uformer | Ours
$\sigma$=15 | Set5 | 33.42 | 34.04 | 34.18 | 34.30 | 34.19 | 34.34 | 34.64 | 34.32 | 34.74
LIVE1 | 32.85 | 33.72 | 33.84 | 33.96 | 33.74 | 33.94 | 34.25 | 34.11 | 34.38
BSD68 | 32.67 | 33.87 | 33.76 | 33.85 | 33.76 | 33.90 | 34.15 | 33.88 | 34.24
$\sigma$=25 | Set5 | 30.92 | 31.88 | 31.98 | 32.10 | 31.82 | 32.24 | 32.51 | 32.13 | 32.60
LIVE1 | 30.05 | 31.23 | 31.26 | 31.37 | 31.09 | 31.50 | 31.73 | 31.65 | 31.86
BSD68 | 29.83 | 31.22 | 31.17 | 31.21 | 31.02 | 31.35 | 31.54 | 31.24 | 31.61
$\sigma$=50 | Set5 | 28.16 | 28.95 | 29.10 | 29.25 | 28.87 | 29.47 | 29.70 | 28.98 | 29.74
LIVE1 | 26.98 | 27.95 | 27.99 | 28.10 | 27.82 | 28.36 | 28.55 | 28.42 | 28.67
BSD68 | 26.81 | 27.91 | 27.91 | 27.95 | 27.76 | 28.19 | 28.35 | 27.78 | 28.40
Table 3: Denoising results on the SIDD and DND datasets. Datasets | Method | BM3D | CBDNet | RIDNet | VDN | DANet | MPRNet | MIRNet | NBNet | Uformer | ours
---|---|---|---|---|---|---|---|---|---|---|---
SIDD | PSNR | 25.65 | 30.76 | 38.71 | 39.28 | 39.47 | 39.71 | 39.72 | 39.75 | 39.77 | 39.77
SSIM | 0.685 | 0.754 | 0.914 | 0.909 | 0.918 | 0.958 | 0.959 | 0.959 | 0.959 | 0.959
DND | PSNR | 34.51 | 38.06 | 39.26 | 39.38 | 39.59 | 39.80 | 39.88 | 39.89 | 39.96 | 39.86
SSIM | 0.851 | 0.942 | 0.953 | 0.952 | 0.955 | 0.954 | 0.956 | 0.955 | 0.956 | 0.955

###  4.1 Training details

The proposed architecture adopts the end-to-end strategy during the training
process and requires no extra component for pre-training. In the experiment,
we use 2 stage horizontal U-shaped structures as a fundamental model and set
the initial channel number as 128. After upsampling or downsampling in the
image pyramid, the number of channels will increase or decrease by 48. In the
MD module, we set the $F_{d}$ to 3/4 $F_{in}$, the number of subspace $k$ as
2, and $r=\sqrt{hw}/l,l=1,2,3$, $l$ is the level of the Horizontal U-shaped
structure. We use the Charbonnier loss to train our model, the formula as :

$\mathcal{L}(D,\textbf{{x}},\textbf{{y}})=\sqrt{\left\|D(\textbf{{x}})-\textbf{{y}}\right\|^{2}+\epsilon^{2}},$
(4)

where $\textbf{{x}},\textbf{{y}},D(\cdot)$ represent noisy image, clean image
and SUMD, respectively.

In the training phase, we use Adam optimizer with the momentum term
(0.9,0.999) and the weight decay of $1e-8$. At the same time, set the initial
learning rate to $2\times 10^{-4}$ gradually reduced to $1\times 10^{-6}$ with
the cosine annealing. We crop the patches of size $128\times 128$ from the
training pairs as a sample with a batch size of 32 for 300K iterations. We
augment the training sample with random horizontal flip and random rotation
with $90^{\circ}\times k,k=1,2,3$.

###  4.2 Synthetic Gaussian denosing

Following the [8, 9], we use the same experimental setup, in which the
training samples are part of BSD, ImageNet validation datasets, Waterloo
Exploration Database. The validation test datasets are composed of Set5,
LIVE1, and BSD68. In the generation of the noise data, we are consistent with
the method in [8, 9] to facilitate fair comparison. We generate four noise
masks, one of which is used for training and the other three types used for
testing. The specific non-$i.i.d.$ Gaussian noise generation formula is as
follows:

n $\displaystyle=$
$\displaystyle\textbf{{n}}^{1}\odot\textbf{{M}},n^{1}_{ij}\sim\mathcal{N}(0,1),$
(5)

where M is a spatially variant mask. We can demonstrate the denoising results
for multiple types of synthetic Gaussian noise under the training of a
category of noise data.

Table 1 lists the denoising performance of 4 types of non-$i.i.d.$ Gaussian
noise and compares them with the previous methods. From the table 1, our
algorithm exceeds the previous arithmetic VDN [8] and NBNet [9] and acquires
an average improvement of more than 0.1 dB on the PSNR compared with the NBNet
in every test case. Table 2 illustrates the PSNR results in the scene of
additive white Gaussian noise(AWGN) and accomplishes the better
performance(average increase of 0.1 dB).

Our method uses the MD module to reconstruct the context features without
prior noise data distribution gradually. On the one hand, it separates the
signal and the noise. On the other hand, it also makes the recovery of the
detailed information of the image more accurate within the relative ranges.

###  4.3 Real Image denoising

SIDD Dataset To justify that our model is competent for real noise scenarios,
we use the SIDD [10], a commonly used mobile phone real-noise benchmark, to
evaluate the performance. We use 320 high-resolution images to train our model
and evaluate the performance of the 1280 validation patches offered by the
SIDD. We compare with the previous methods, such as VDN [8], MPRNet [1],
MIRNet, NBNet [9] and Uformer[4], and quantitatively demonstrated the
denoising capacity of various models in Table 3. We achieve the comparable
performance of 39.77 dB with Uformer on PSNR and surpass the other methods at
least 0.02 dB. In addition, we also provide a visual denoising comparison
example from different models in the top row of Fig 3.

DND Dataset We further utilize the DND [11] datasets to demonstrate the
capability of our model. Unlike the NBNet training strategy, we only train the
model on the SIDD training set and directly use the best model from the SIDD,
evaluating the performance on the DND benchmark that contained 1000 patches
for testing. We present the visual comparisons of our results with the other
state-of-the-art methods in Fig 3, and our model obtains 39.86 dB shown in
Table 3, the comparable results without using any extra datasets. For
instance, our method achieves 0.66 dB improvement over Uformer on DND.

Table 4: Ablation study on MD and other modules of the proposed in SUMD. #Stages | Stage Combination | PSNR
---|---|---
2 | Unet | 39.56
2 | Unet+HUS | 39.73
2 | Unet+HUS+MD | 39.77
1 | Unet+HUS+MD | 39.68
Table 5: Effect of the middle dimension of the MD module. The rate represents the original dimensions proportioned to the middle dimensions. ratio | w/o | 1:4 | 2:4 | 3:4 | 4:4
---|---|---|---|---|---
PSNR | 39.49 | 39.55 | 39.58 | 39.60 | 39.55

###  4.4 Ablation Studies

Multi-stages with MD Module To demonstrate the performance of our MD module
and HUS structure, we test each component on the various network settings.
Table 4 shows that the HUS merged into Unet has improved the 0.17 dB higher
than the simple Unet structure in 2 stages on PSNR. Then, we compare our
regular version to the model that replaced the MD module with a BST to keep a
fair comparison. The results demonstrate a substantial drop in PSNR from 39.77
dB to 39.73 dB when we remove the MD module. From Table 4, we can also justify
the effectiveness of the multi-stages architecture in our model. Compared with
the single-stage and the two stages design. The PSNR has improved from 39.68
dB to 39.77 dB.

Integrated into simple Unet with HUS In order to examine the performance of
the design on a simple CNN denoising model, We shallow the channels from 128
to 80 and test the model with different middle dimensions on a single-stage
Unet with HUS structure. In Table 5, We notice that the CNN with MD module can
achieve about 0.1 dB higher than the w/o MD module.

Influence about the reducing rate In Table 5, We also explore the appropriate
K, a reducing rate of the middle dimensions of the MD module, boosting our
model achieving the best performance. It is evident that when we set the rate
k between 1/2 and 3/4, the MD can realize a low-ranked, compact, and
representative feature to tackle the shortcoming of the CNN-based model that
lacks global information.

##  5 Conclusion

In this work, we present a multi-stage progressive CNN-based model with the MD
module to tackle the problem of global feature extraction. To compare with the
Transformer long-distance dependent feature, we assume that the low-ranked
features can substitute the global features and achieve comparable
performance. In order to boost the MD module, we also adopt the multi-stages
design that can limit the receptive field in a definite scope on the current
stage and the channel attention block to weight feature from the different
channels for promoting the critical feature maps decided in the final process.
We also believe that how to utilize the MD module to extract the global
feature will be a vital issue that can facilitate the CNN model to acquire the
global structured features.

## References

  * [1] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao,  “Multi-stage progressive image restoration,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14821–14831.
  * [2] Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin,  “Is attention better than matrix decomposition?,”  arXiv preprint arXiv:2109.04553, 2021.
  * [3] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
  * [4] Zhendong Wang, Xiaodong Cun, Jianmin Bao, and Jianzhuang Liu,  “Uformer: A general u-shaped transformer for image restoration,”  arXiv preprint 2106.03106, 2021.
  * [5] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu,  “Image super-resolution using very deep residual channel attention networks,”  in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 286–301.
  * [6] Daniel D.Lee and H.Sebastian Seung,  Algorithms for non-negative matrix factorization. In Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference, vol. 13,  MIT Press, 2001.
  * [7] Daniel D Lee and H Sebastian Seung,  “Learning the parts of objects by non-negative matrix factorization,”  Nature, vol. 401, no. 6755, pp. 788–791, 1999.
  * [8] Zongsheng Yue, Hongwei Yong, Qian Zhao, Lei Zhang, and Deyu Meng,  “Variational denoising network: Toward blind noise modeling and removal,”  arXiv preprint arXiv:1908.11314, 2019.
  * [9] Shen Cheng, Yuzhi Wang, Haibin Huang, Donghao Liu, Haoqiang Fan, and Shuaicheng Liu,  “Nbnet: Noise basis learning for image denoising with subspace projection,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4896–4906.
  * [10] Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown,  “A high-quality denoising dataset for smartphone cameras,”  in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [11] Tobias Plotz and Stefan Roth,  “Benchmarking denoising algorithms with real photographs,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1586–1595.