# GAMA-IR: Global Additive Multidimensional Averaging for Fast Image
Restoration

Youssef Mansour and Reinhard Heckel
School of Computation  Information and Technology  Technical University of
Munich
Munich Center for Machine Learning

###### Abstract

Deep learning-based methods have shown remarkable success for various image
restoration tasks such as denoising and deblurring. The current state-of-the-
art networks are relatively deep and utilize (variants of) self attention
mechanisms. Those networks are significantly slower than shallow convolutional
networks, which however perform worse. In this paper, we introduce an image
restoration network that is both fast and yields excellent image quality. The
network is designed to minimize the latency and memory consumption when
executed on a standard GPU, while maintaining state-of-the-art performance.
The network is a simple shallow network with an efficient block that
implements global additive multidimensional averaging operations. This block
can capture global information and enable a large receptive field even when
used in shallow networks with minimal computational overhead. Through
extensive experiments and evaluations on diverse tasks, we demonstrate that
our network achieves comparable or even superior results to existing state-of-
the-art image restoration networks with less latency. For instance, we exceed
the state-of-the-art result on real-world SIDD denoising by 0.11dB, while
being 2 to 10 times faster.

##  1 Introduction

$10^{1}$$10^{1.2}$$10^{1.4}$$10^{1.6}$$10^{1.8}$$10^{2}$$39.6$$39.8$$40$$40.2$$40.4$GAMA-
IRNAFNetRestormerMAXIMUformerHINetMPRNetLatency/msPSNR/dBSIDD -
Denoising$10^{1}$$10^{1.2}$$10^{1.4}$$10^{1.6}$$10^{1.8}$$10^{2}$$32.5$$33$GAMA-
IRGAMA-
IRNAFNetNAFNetRestormerMAXIMUformerHINetMPRNetMIMOUNet+Latency/msPSNR/dBGoPro
- Deblurring Figure 1: Performance and speed of our GAMA-IR network in
comparison to other popular and state-of-the-art networks on two common
restoration tasks: Image denoising and deblurring. GAMA-IR achieves slightly
better performance than state of the art networks, while being significantly
faster.

Deep learning gives state-of-the-art performance for a variety of image
restoration tasks like denoising, deblurring, and deraining. Image restoration
is often formulated as an image-to-image regression problem, i.e., to map a
noisy image or a coarse reconstruction of an image to a clean image [Zha+17,
Bro+19].

The network architectures that map the noisy or coarse reconstruction to the
clean images play a pivotal role for performance. For many years, the
prominent UNet [RFB15] has been the go-to network for different reconstruction
tasks, since it performs well and has low computational requirements. The UNet
is therefore a common choice for image reconstruction problems where speed is
a concern, and it is also frequently used as a building block in more
complicated reconstruction approaches. For example, recent self-supervised
denoising methods [Leh+18, KBJ19, Mor+20, Hua+21, Wan+22] as well as the
Variational Network [Sri+20], a state-of-the art network for accelerated MRI,
all utilize the UNet as a building block.

In the recent years, several deep networks that use a self attention [Vas+17]
mechanism or variants of it have been proposed for image reconstruction, in
particular SwinIR [Lia+21], Restormer [Zam+22], and MAXIM [Tu+22]. Those
networks provide better reconstructions than the UNet at the cost of
significantly higher compute, due to their depth, and the computational cost
of the attention or attention-like mechanisms used in those networks.

There is large demand for efficient and simple network architectures that
enable fast inference times, consume little memory, while maintaining state-
of-the-art image restoration performance. We focus on latency and memory at
inference instead of MACs (multiply-accumulate operations) or number of
parameters, since in many practical applications, speed and memory consumption
are the metrics of interest.

In this paper, we propose a network with state-of-the-art performance but with
low latency and memory consumption. The network is shallow and only uses
convolutions, non-linearities, skip connections [He+16], and LayerNorm
[BKH16]. A key component of the network is a simple global additive
multidimensional averaging (GAMA) block that efficiently captures non-local
information and enables a large receptive field even when used in shallow
networks. We integrate this block into a relatively standard convolutional
neural network architecture, and the resulting network is called GAMA-IR (IR
for image restoration). GAMA-IR can compete with state-of-the-art models at
higher speed as seen in Figure 1.

State-of-the-art networks like Restormer[Zam+22] and NAFNet[Che+22] are
usually deep. The network’s depth enables good performance, but causes the
network to be slow, since computations must be performed sequentially across
multiple layers. Therefore, the challenge in designing a fast network lies in
enabling a shallow network to perform well, which our proposed GAMA block
achieves.

Up to our knowledge, the only recent image restoration network to focus on
minimizing the latency instead of the MACs is MalleNet [Jia+22]. MalleNet is a
lightweight network that utilizes an efficient variant of dynamic convolutions
[Che+20] to create a fast network. Although MalleNet reaches impressive speed,
its performance is notably below the state of the art.

Recently, several other efficient networks have been proposed that focus on
minimizing the number of parameters or FLOPs (floating point operations).
However, the number of parameters and FLOPs are not necessarily well
correlated with a network’s latency [Din+21, Deh+22, Vas+23] and/or with a
network’s memory footprint as shown in Figure 2. That is because metrics like
parameter count and FLOPs do not consider the memory access cost and degree of
parallelism, which can significantly affect a network’s speed and memory
expenditure [Ma+18].

For instance, sharing a model’s weights decreases the model’s number of
parameters, but does not change the latency. Moreover, adding skip connections
[He+16] does not change the parameter count and has a negligible effect on the
FLOPs, but requires significantly more memory. This weak correlation between
the metrics has been manifested by SwinIR [Lia+21], which is very parameter-
efficient. However, the network is slow and consumes a lot of memory in
comparison with other baseline networks as seen in Figure 2.

Contrary to the previous restoration works, our work is the first to directly
target the compute metrics of interest on a GPU: Memory and speed. The main
contribution of our work, illustrated in Figure 1, is:

  * •

We optimise the network’s latency and memory expenditure instead of the number
of trainable parameters or FLOPs as in previous restoration works.

  * •

We propose a global additive multidimensional averaging block that efficiently
captures global dependencies, and enables a shallow network to have a large
receptive field.

  * •

We design a simple yet effective image restoration network, that performs
similar to the state of the art models, but at much higher speed.

##  2 Related Work

### Image-to-image Networks

Convolutional neural networks are a powerful choice for image reconstruction.
A simple stack of convolutional layers, known as the the DnCNN [Zha+17]
architecture, works well for denoising. Today, the encoder-decoder based UNet
[RFB15] is a popular choice for its good performance-speed trade off. The UNet
was initially proposed for medical image segmentation, but became popular for
image reconstruction tasks, since it is faster and performs better than DnCNN
[Bro+19]. Later, several variants were proposed such as Attention-UNet
[Okt+18] and UNet++ [Zho+20].

With the rise of the transformer architecture and the introduction of the
vision transformer [Dos+21], several networks started incorporating attention
mechanisms, which improve imaging performance at a price of computational
cost. The vision transformer itself performs well for image reconstruction,
better than the UNet, but at a significantly higher computational cost [LH22].
The transformer based SwinIR [Lia+21], that builds on the Swin Transformer
[Liu+21], exhibits excellent performance without any convolutions. Later,
Uformer [Wan+22a] and Restormer [Zam+22] showed even better performance when
utilizing convolutions with attention based architectures.

Inspired by the MLP-Mixer [Tol+21], which only uses MLPs for image
classification, several MLP-based reconstruction networks were proposed. The
Image-to-Image Mixer [MLH22] stacks several MLP layers without any
convolutions or attention gates. MixerGAN [CG21] is an MLP based architecture
for Unpaired Image-to-Image Translation. MAXIM [Tu+22] is a multi-axis
spatially gated MLP-based network that achieves remarkable performance. It is
no surprise that MLP based networks work well; plain MLPs [BSH12] can compete
with BM3D [Dab+07], a classical denoising algorithm.

### Self Attention-based architectures

Transformers and attention modules exhibited success in vision tasks [Vas+17,
Dos+21], but self attention mechanisms exhibit quadratic complexity with the
input size, which raises the cost significantly for high resolution inputs.
Non-local blocks [Wan+18] compute the response at a position as a weighted sum
of the features at all positions, which performs well, but is expensive to
execute. To reduce the cost, Squeeze-and-Excite blocks [HSS18] capture channel
attention only by performing global average pooling across the channel axis.
Efficient Channel Attention Net [Wan+20] simplifies Squeeze-and-Excite by
removing the intermediate convolution layers, and utilizing 1D convolutions.
Convolutional Block Attention Modules [Woo+18] perform global max pooling in
addition to average pooling across both channel and spatial axes. We devise a
block that builds on previous works, and efficiently captures global
information, and enables a shallow network to have a large receptive field.

### Efficient Networks in other Domains

The efficiency of neural networks has been widely studied in other domains,
such as classification and segmentation. MobileNet [How+17], is a lightweight
convolutional neural network that achieved impressive accuracy, while keeping
the computational complexity low, even on mobile devices. Similarly,
EfficientNet [TL19] is designed to be a scalable and efficient network, and
achieves good performance across various resource-constrained settings.
Recently, MobileOne [Vas+23] was proposed, and was shown to outperform several
high performing models in under one millisecond runtime.

Instead of designing lightweight networks, one can change the network’s
structure from train to test time, such that the network can benefit from
large capacity during training, while being efficient at inference. For
example, in knowledge distillation [HVD15], a larger teacher network is used
during training, and a smaller, often simpler, student network is used during
inference. ExpandNets [GAS20] are linearly over-parameterized networks with a
larger teacher network guiding a smaller more efficient student network.
RepVGG [Din+21] combines multiple parallel branches during training into a
single branch at inference, achieving better performance than the original VGG
nets [SZ15] without extra compute at inference.

##  3 GPU Efficiency Metrics

$10^{1.5}$$10^{2}$$0$$50$$100$$150$Latency/ms$r$ =
0.650$10^{0}$$10^{1}$$10^{2}$$0$$50$$100$$150$$r$ =
0.153UNet[RFB15]DnCNN[Zha+17]UNet++[Zho+20]Img2Img- Mixer
[MLH22]NAFNet[Che+22]SwinIR[Lia+21]MAXIM[Tu+22]Restormer[Zam+22]$10^{1.5}$$10^{2}$$0$$5$$10$FLOPS/GMemory/GB$r=0.698$$10^{0}$$10^{1}$$10^{2}$$0$$5$$10$Parameters/M$r=0.131$
Figure 2: Correlation of FLOPs and parameter count with latency and memory of
image restoration algorithms run on a NVIDIA384RTX A600 GPU. Latency and
memory are measured as the time and memory required for a forward pass through
the network at inference. All metrics are considered for an image input size
of $1\times 3\times 256\times 256$, which is a single 3-channel RGB image
(i.e., the batch size is one). The networks’ parameters are varied to create
different sizes of the same network. Note the moderate to weak correlation
between FLOPS and Parameters with Memory and Latency.

We propose a network that reaches a good trade-off between image quality and
computational efficiency, specifically we focus on latency and memory
consumption on a GPU. Note that a network can be computationally efficient
with respect to one metric but inefficient with respect to another [Deh+22].
Several works measure a network’s efficiency with FLOPs/MACs or number of
parameters. However, a network with a small FLOP count or with few parameters
can still be significantly slower and consume significantly more memory when
run on a GPU, relative to a network with more FLOPs.

For many applications, image restoration methods run on a GPU, and in such a
setup, speed and memory consumption are the metrics of interest, since they
provide a more accurate measure for the actual cost spent to deploy the
network on a GPU.

In Figure 2 we plot the correlation between FLOPs and parameter count
(commonly reported efficiency metrics) with speed and memory consumption on a
GPU (metrics of interest when run on a GPU). We plot the FLOPs and number of
parameters against the memory and latency for several image reconstruction
networks ranging from classical models to newer state-of-the-art ones. We also
calculate Spearman’s rank correlation coefficient $r$ for every plot. By
inspecting the plots and correlation factors, we can see that FLOPs are
moderately correlated with memory and latency, whereas parameter count is only
very weakly correlated.

##  4 GAMA-Image Restoration (IR) Network

In this section we introduce the GAMA-IR network. The key to achieving state-
of-the-art performance at significantly higher speed than competing networks
is to limit the depth of the network. This poses a challenge, since high
performance is usually attained by deep networks which often have a high
receptive field.

Our GAMA block enables a large receptive field even when used in a shallow
network, therefore enabling state-of-the-art performance at higher speed. We
start with defining the overall architecture of our network, and then
introduce the GAMA block.

Figure 3:  The architecture of our proposed network. Similar to the UNet, the
network has an encoder-decoder structure with skip connections adding the
feature maps of equal resolution elementwise. The multi-resolution layout is
achieved by downsampling and upsampling operations. The network has only 2
hyperparameters, the depth and width. The depth is the total number of
building blocks, and the width is the number of channels $C$.  Figure 4:  a)
Non-Local Block [Wan+18]. b) Squeeze-and-Excite Block [HSS18] c) Our proposed
GAMA Block. $\otimes$ denotes matrix multiplication, $\oplus$ and
{\raisebox{-2.7pt}{*}}⃝ denote element wise addition and multiplication,
respectively. The circular arrows in c) denote a transpose operation. The
blocks aid in capturing long-range dependencies in data. Non-local blocks
calculate interactions between all pixels (matrix multiplications), while the
other blocks rely on global averaging to summarize the entire feature map
across a specific dimension.

GAMA-IR is is displayed in Figure 3. The input image is first converted to a
feature map with $C$ channels using a $3\times 3$ convolution. The feature map
is then processed by a series of building blocks and downsampling and
upsampling layers. Skip connections from the encoder to the decoder add the
feature maps of equal resolution element wise. The last feature map is mapped
back to the image space via a final $3\times 3$ convolution.

We follow the down and upsampling scheme of Pixel Shuffle [Shi+16], but use
$1\times 1$ convolutions instead of $3\times 3$ to reduce cost. When
downsampling, the number of channels are doubled and the height and the width
are halved. This is achieved by a $1\times 1$ convolutional layer that first
maps a feature map of size $C\times H\times W$ to $\frac{C}{2}\times H\times
W$, which is then reshaped to $2C\times\frac{H}{2}\times\frac{W}{2}$. When
upsampling, those computations are reversed, specifically, a feature map of
size $C\times H\times W$ is mapped via $1\times 1$ convolutions to $2C\times
H\times W$, which is then reshaped to $\frac{C}{2}\times 2H\times 2W$.

The main operations are carried out by a building block that consists of our
proposed GAMA block and standard convolutional network components, such as
convolutional operations, non-linearities, skip connections [He+16], and
LayerNorm [BKH16].

Following Restormer [Zam+22], we use $1\times 1$ convolutions followed by
$3\times 3$ depthwise convolutions [Cho17] in the building block. This
combination of convolutions can be thought of as a sparse standard pointwise
$3\times 3$ convolution, that performs equally well, but with substantially
less parameters and memory consumption.

We adopt NAFNet’s [Che+22] non-linearity, which was shown to slightly
outperform classical activation functions such as ReLU [Aga18] or GeLU [HG16].
The non-linearity splits the input feature map along the channel axis in two
halves, and then multiplies the two halves element wise.

The key difference to existing restoration networks is the GAMA block that
models global dependencies. We first describe the global blocks of other
state-of-the-art networks, and then explain the GAMA block, and finally
provide a comparison for the receptive field of the different blocks.

###  4.1 Global Blocks

Convolutional networks operate locally only, and have a limited receptive
field, except when concatenated several times which leads to deep networks.
This limits their ability to capture interactions between far-away pixels,
making them inherently local in their processing. To enable global
dependencies, Restormer [Zam+22] utilizes a variant of the Non-Local Block
[Wan+18], illustrated in Figure 4.

Non-local blocks are designed to capture long-range dependencies in data by
utilizing self attention [Vas+17]. They perform matrix multiplications which
allow each position of the input feature map to attend to every other
position. This helps capture global context and dependencies in the data,
making them useful for imaging tasks. However, the execution time of these
matrix multiplications is high, making non-local blocks resource intensive and
challenging to scale for large inputs.

Instead of calculating all pairwise interactions, NAFNet [Che+22] utilizes a
Squeeze-and-Excite block [HSS18], that executes 2D global averaging, which
squeezes a 3D feature map of size $C\times H\times W$ to a 1D channel vector
of length $C$, i.e., each spatial feature $H\times W$ is reduced to a scalar
value. The channel vector is passed through an MLP (multi-layer perceptron) to
output a new vector of same length, which is multiplied element-wise with the
input feature map. This is equivalent to multiplying each channel of the input
feature map by a learnable scalar weight, i.e., giving more weight to more
relevant channels, while suppressing less informative ones. In summary, a
Squeeze-and-Excite block captures global dependencies across the channel axis.

A Squeeze-and-Excite block is significantly faster than a Non-local block,
since it does not perform matrix multiplications, but only global averaging
and element-wise multiplication. However, a major drawback is that it only
captures channel-wise dependencies, which significantly increases the number
of parameters. The number of parameters in the MLP is $C^{2}$. Since in deep
layers of a network, $C$ exceeds $1000$, this significantly increases the
parameters per block. Our proposed GAMA block aids in overcoming the
aforementioned drawbacks.

###  4.2 GAMA Block

The input to the GAMA block, a feature map of size $C\times H\times W$, is
globally averaged across each dimension separately, resulting in three feature
maps of sizes $1\times H\times W$, $C\times 1\times W$, and $C\times H\times
1$. The latter two are transposed to $1\times C\times W$ and $1\times H\times
C$ respectively. The three feature maps are convolved with three separate 2D
convolutions with input and output channels = 1, and kernel size = 7. Finally,
the resulting feature maps are transposed and then expanded to match the
initial input size ($C\times H\times W$) before being added together with the
input feature map to generate the block’s output as seen in figure 4. The
expansion occurs by simply replicating the values across the appropriate
dimensions.

Unlike Squeeze-and-Excite, we capture global information from the entire
feature map, since we average along all three dimensions. We also execute
global averaging across one dimension at a time, which results in 2D feature
maps instead of one dimensional vectors, and therefore less loss of
information. Moreover, our block has minimal parameter overhead. Each 2D
convolution operator has $7\times 7\times 1\times 1=49$ learnable parameters,
which results in only $3\times 49=147$ parameters per block. The number of
parameters is fixed, and independent of the size of the input feature map.

###  4.3 Receptive Field

Clean Image Noisy Image Plain CNN  Squeeze&Excite GAMA

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 5: The ability of the underparameterized network illustrated above with
different choices for the block to memorize an image with more pixels than it
has parameters. The block in the network is either Squeeze&Excite, or GAMA, or
identity in case of the Plain CNN. The total number of parameters is the same
for each network variant. The chess board has a repetitive pattern that the
networks with the larger receptive fields can exploit, whereas the baboon has
less such structure, which is why we see similar denoising performance.

The receptive field of a convolutional neural network is the spatial extent of
the input volume that influences a given pixel in the output volume. A large
receptive field allows an output pixel to have long range dependencies on far
away pixels. Deep convolutional networks have a large receptive field [ZF14,
Kuo16], but are slow since information flows sequentially from layer to layer.

The GAMA block enables a relatively shallow and therefore potentially fast
network to have a large receptive field relative to existing blocks. To
demonstrate this, we perform the following experiment.

We consider three variants of a shallow network, as illustrated in Figure 5.
Plain CNN: The plain CNN with only one hidden layer, as displayed in Figure 5
with Block=Identity. Squeeze&Excite:  The same network, but with the Squeeze-
and-Excite block inserted in the hidden layer, i.e., block=Squeeze&Excite.
GAMA: The same network again, but block=GAMA block.

We do not consider the Non-Local blocks in this comparison, since they are
slow and memory consuming as seen in the next section with Restormer [Zam+22].

We train each of the three networks to map a single noisy image to its clean
counterpart, and measure how well it can recover the clean image. Note that we
train and evaluate on the same single image, but since the network is
underparameterized (has much less parameters than the image’s pixels), it is
not able to memorise the exact clean image (achieve 0 training loss). For a
fair comparison, the number of channels in the network’s hidden layer is
adapted so that each of the three networks has the same number of parameters.

The comparison is shown in figure 5. We consider two different images: The
first, a chess board, features repeated patterns, while the second, a baboon,
lacks such a pattern. A network with a small receptive field operates at a
local level, unable to recognize the global patterns present in the chess
image. Conversely, a larger receptive field enables the network to perceive
the entire chess board as a composition of several identical smaller segments,
significantly enhancing pattern recognition and feature extraction. This is
evidenced by our GAMA block achieving markedly higher PSNR on the chess image.
In contrast, for the Baboon image, where recognisable patterns are absent, the
benefits of an enlarged receptive field diminish, resulting in a less
pronounced difference in PSNR. The comparative performance on these two images
suggests that the improvement observed with the GAMA block on the chess image
might be attributed to its larger receptive field.

In this experiment, the blocks were inserted in a small network with one
hidden layer, and trained and tested on the same single image to study the
receptive field of those blocks. When a deeper network is trained on a large
training set and tested on a separate test set, as indented, we also find that
a GAMA-IR block works well in a shallow network, but a Squeeze-and-Excite
block (as used in NAFNet) does not not work well in a shallow network, as
shown in the ablation studies of the original NAFNet paper.

##  5 Experiments

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 6:  Sample denoised images from SIDD [ALB18]. Our method produces sharp
images that are almost indistinguishable from the ground truths.

| MPRNet | SRMNet | HINet | Uformer | MAXIM | Restormer | NAFNet | GAMA-IR
---|---|---|---|---|---|---|---|---
| [Zam+21] | [Fan+22] | [Che+21] | [Wan+22a] | [Tu+22] | [Zam+22] | 32/64 [Che+22] | S/L (ours)
Memory/GB | 6.50 | 3.95 | 2.83 | 4.22 | 13.60 | 13.85 | 2.96/4.52 | 2.98/4.57
Latency/ms | 35.6 | 29.5 | 27.0 | 34.6 | 133.7 | 92.1 | 30.1/35.2 | 15.1/20.7
PSNR | 39.71 | 39.72 | 39.99 | 39.89 | 39.96 | 40.02 | 39.96/40.30 | 40.02/40.41
SSIM | 0.958 | 0.959 | 0.958 | 0.960 | 0.960 | 0.960 | 0.960/0.962 | 0.960/0.962

Table 1: Denoising results on SIDD [ALB18]. Our large model boosts the state-
of-the-art PSNR score by 0.11 dB at higher speed.

We evaluate GAMA-IR on four common image restoration tasks: real-world camera
denoising, synthetic Gaussian denoising, motion deblurring, and deraining. For
all tasks, we use the AdamW optimizer [LH19] with initial learning rate
$1e^{-3}$ and $\beta_{1}=\beta_{2}=0.9$ and a cosine annealing scheduler
[LH17] to reduce the learning rate to $1e^{-7}$.

We consider a small and large network. The small network GAMA-IR-S has depth
18 and width 42, and the large network GAMA-IR-L has depth 19 and width 80.
The small network is trained for 6.4 million iterations and the large network
for 25.6 million iterations.

As performance metrics, we report the PSNR in dB and the SSIM [Wan+04] scores.
As efficiency metrics, we measure the latency and memory on a NVIDIA RTX A6000
GPU at inference for input size $1\times 3\times 256\times 256$. Latency and
memory are measured as the average of 100 runs after ignoring the first 10
runs, that might fluctuate in speed and memory usage. All networks are run in
Pytorch.

###  5.1 Real-world Camera Noise

We consider the Smartphone Image Denoising Dataset (SIDD) [ALB18], which is
the most commonly used dataset for naturally occurring camera noise. It
consists of noisy images captured by several smartphone cameras under
different lighting conditions and noise patterns. The ground truths are
estimated by averaging multiple noisy instances of the same static scene. We
take the medium dataset as the training set, and the validation set as the
test set.

The results are in table 1 and figure 6. GAMA-IR improves over the previous
state of the art Restormer and NAFNet, while being faster, while being 7 and 2
times faster respectively.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 7: Example deblurred results for GoPro [NKL17]. Our GAMA-IR network is
able to deblur the images well and restores fine details.

| MIMO-UNet+ | Uformer | MPRNet | HINet | MAXIM | Restormer | NAFNet 32/64 | GAMA-IR
---|---|---|---|---|---|---|---|---
| [Cho+21] | [Wan+22a] | [Zam+21] | [Che+21] | [Tu+22] | [Zam+22] | [Che+22] | S/L (ours)
Memory/GB | 3.11 | 4.22 | 6.50 | 2.83 | 13.60 | 13.85 | 2.96/4.52 | 2.98/4.57
Latency/ms | 17.0 | 34.6 | 35.6 | 27.0 | 133.7 | 92.1 | 30.1/35.2 | 15.1/20.7
GoPro | 32.45 | 32.97 | 32.66 | 32.77 | 32.86 | 32.92 | 32.29/33.08 |  32.44/33.15
[NKL17] | 0.957 | 0.967 | 0.959 | 0.959 | 0.961 | 0.961 |  0.956/0.963 |  0.958/0.963
HIDE | 29.99 | 30.83 | 30.96 | 30.32 | 32.83 | 31.22 | - | 30.57/31.14
[She+19] | 0.930 | 0.952 | 0.939 | 0.932 | 0.956 | 0.942 | - | 0.936/0.943

Table 2: Deblurring results in terms of PSNR and SSIM for training on GoPro
and testing on GoPro and HIDE.

###  5.2 Deblurring

We train on the GoPro [NKL17] training set and test on the GoPro test set and
the HIDE [She+19] dataset. The model trained on GoPro is tested on HIDE
without any finetuning. We test on the full resolution test images without any
test time refinements such as testing by patches [Che+21] or test time local
conversion [Chu+21] for a fair comparison with the competing methods.

The results are reported in table 2 and figure 7. On GoPro [NKL17], our
network achieves best results. On HIDE [She+19], only MAXIM [Tu+22]
outperforms our network, but at the cost of significantly more compute.

###  5.3 Deraining

(a)

(b)

(c)

(d)

Figure 8:  Derained samples from Rain100H [Yan+17]. GAMA-IR is able to recover
most of the details despite the input image being extremely noisy.

| SPAIR | MPRNet | HINet | MAXIM | Restormer | GAMA-IR-L
---|---|---|---|---|---|---
| [Pur+21] | [Zam+21] | [Che+21] | [Tu+22] | [Zam+22] | (ours)
Memory/GB | - | 6.50 | 2.83 | 13.60 | 13.85 | 4.57
Latency/ms | - | 35.6 | 27.0 | 133.7 | 92.1 | 20.7
Rain 100H | 30.95 | 30.41 | 30.63 | 30.81 | 31.46 | 33.23
[Yan+17] | 0.892 | 0.890 | 0.893 | 0.903 | 0.904 | 0.922
Rain 100L | 36.93 | 36.40 | 37.20 | 38.06 | 38.99 | 37.67
[Yan+17] | 0.969 | 0.965 | 0.969 | 0.977 | 0.978 | 0.972

Table 3: PSNR and SSIM for image deraining. Memory and latency measurements
for SPAIR are missing, since the code is not available online.

Following recent works on image deraining [Zam+21, Tu+22, Zam+22], we set up a
training set collected from several deraining datasets [Li+16, ZSP20, ZP18,
Fu+17]. We evaluate on the two common test sets: Rain100H and Rain100L
[Yan+17]. We calculate the PSNR/SSIM scores by utilizing the Y channel within
the YCbCr color space, which is the common evaluation approach in deraining.

As seen in table 3 and figure 8, our method achieves competitive performance
with the state of the art networks, and significantly outperforms them on the
Rain100H dataset.

###  5.4 Gaussian Noise

Following [Lia+21, Zam+22], we use images from the DIV2K [AT21], Flickr2K,
BSD500 [Arb+11], and WaterlooED [Ma+16] datasets as training images. As test
sets, we take the standard benchmarks Kodak24 [Fra99], and CBSD68 [Mar+01].

We consider three noise levels with $\sigma=15,25,50$. We show the results in
table 4. On average, GAMA-IR’s performance is within state-of-the-art, while
being significantly faster and requiring less memory.

As seen from the results on the various tasks, our network’s performance is
within state-of-the-art, and sometimes even slightly better. Most of the
recently proposed state-of-the-art restoration networks achieve minor
differences in PSNR/SSIM scores, which are hard to notice when looking at the
restored images (see figures 6,7,8). Therefore, what makes our network stand
out is its low latency, and memory consumption.

Method | Memory | Latency | Kodak24 [Fra99] | CBSD68 [Mar+01] | Average
---|---|---|---|---|---
/GB | /ms | $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | $\sigma=15$ | $\sigma=25$ | $\sigma=50$
DnCNN[Zha+17] | 2.10 | 5.1 | 34.60 | 32.14 | 28.95 | 33.90 | 31.24 | 27.95 | 34.25 | 31.69 | 28.45
BRDNet[TXZ20] | 2.20 | 6.2 | 34.88 | 32.41 | 29.22 | 34.10 | 31.43 | 28.16 | 34.49 | 31.92 | 28.69
SwinIR[Lia+21] | 13.64 | 149.7 | 35.34 | 32.89 | 29.79 | 34.42 | 31.78 | 28.56 | 34.88 | 32.34 | 29.18
Restormer[Zam+22] | 13.85 | 92.1 | 35.47 | 33.04 | 30.01 | 34.40 | 31.79 | 28.60 | 34.94 | 32.42 | 29.31
GRL-T[Li+23] | 7.62 | 72.7 | 35.24 | 32.78 | 29.67 | 34.30 | 31.66 | 28.45 | 34.77 | 32.22 | 29.10
GRL-B [Li+23] | 41 | 503 | 35.43 | 33.02 | 29.93 | 34.45 | 31.82 | 28.62 | 34.94 | 32.42 | 29.28
GAMA-IR-L | 4.57 | 20.7 | 35.47 | 33.04 | 30.02 | 34.42 | 31.79 | 28.61 | 34.95 | 32.42 | 29.32

Table 4: PSNR scores for denoising synthetic Gaussian noise.

##  6 Ablation Studies

We perform ablation studies with our small model on SIDD [ALB18] denoising and
GoPro [NKL17] deblurring. We train the model for 6.4 million iterations for
every ablation. Results are reported in table 5.

Our GAMA block performs global averaging individually across each dimension,
leading to three 2D feature maps (figure 4 c). We test with a variant which
averages along two dimensions simultaneously as in Squeeze-and-Excite (figure
4 b) resulting in three one-dimensional vectors of length $C,H,W$. Each vector
is then processed with a separate MLP. This variant is denoted by 1D Vectors,
and performs worse than the default one, specially on GoPro. It also comes
with a signifcant increase in parameter count as discussed in section 4.1.

Another variant we test is a multiplicative instead of an additive operation
within the GAMA block. The default version adds all branches together with the
input (figure 4 c). We replace the addition with a multiplication, and denote
this variant with Multiplicative. The default version performs better, is 10%
faster, and consumes 7% less memory.

We also test a version of our network without the GAMA block, which we denote
by w/o GAMA. By comparing its performance to the default version with the GAMA
block, we see the importance of increasing the receptive field, and capturing
long range dependencies. This variant without the GAMA block is 5% faster, and
consumes 5% less memory. This shows that the GAMA block can improve
performance with minimal computational overhead.

Additionally, we compare to a UNet [RFB15], which is a simple encoder-decoder
network consisting of only convolutions, instance normalization [UVL17], and
ReLU [Aga18] non-linearities. We create a version of the UNet that consumes
the same memory as our default small model, but is twice as fast. As seen from
the scores, it performs much worse.

Finally, we design a faster version of our small model, by decreasing the
depth. We increase the width to keep memory consumption constant. This yields
a network that is similar to the UNet in both memory, and speed. This variant,
denoted by Default (fast), outperforms the UNet despite consuming the same
compute (memory and speed).

| Default | 1D Vectors | Multiplicative | w/o GAMA | UNet | Default(fast)
---|---|---|---|---|---|---
SIDD | 40.02/0.960 | 40.00/0.960 | 39.67/0.958 | 39.79/0.958 | 39.43/0.957 | 39.85/0.959
GoPro | 32.41/0.957 | 32.26/0.956 | 31.04/0.943 | 32.21/0.956 | 30.28/0.934 | 31.56/0.949

Table 5: PSNR and SSIM scores of the ablation studies. Most variants have
similar latency and memory requirements. Only the UNet and Default(fast) are
twice as fast, with same memory consumption.

##  7 Conclusion

In this paper we proposed an image restoration network, GAMA-IR, that achieves
state-of-the-art restoration performance with low memory consumption and
significantly better latency. The GAMA block of GAMA-IR mitigates the long
latency and high memory consumption associated with deep networks and
attention mechanisms used in competing architectures. By implementing global
additive multidimensional averaging (GAMA) operations, we efficiently capture
long range dependencies, and increase the receptive field of a shallow network
with minimal computational overhead. The github page for our project is here.

## Acknowledgements

The authors are supported by the German Federal Ministry of Education and
Research and the Bavarian State Ministry for Science and the Arts. The authors
of this work take full responsibility for its content. The authors also
received funding by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) - 456465471, 464123524.

## References

  * [ALB18] Abdelrahman Abdelhamed, Stephen Lin and Michael S. Brown  “A High-Quality Denoising Dataset for Smartphone Cameras”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018
  * [Aga18] Abien Fred Agarap  “Deep Learning using Rectified Linear Units (ReLU)”  In _arXiv preprint_ , 2018  arXiv:1803.08375
  * [AT21] Eirikur Agustsson and Radu Timofte  “NTIRE 2017 challenge on single image super resolution: Dataset and study”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2021
  * [Arb+11] Pablo Arbelaez, Michael Maire, Charless Fowlkes and Jitendra Malik  “Contour Detection and Hierarchical Image Segmentation”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)_ , 2011
  * [BKH16] Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton  “Layer Normalization”  In _arXiv preprint_ , 2016  arXiv:1607.06450
  * [Bro+19] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet and Jonathan T. Barron  “Unprocessing Images for Learned Raw Denoising”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019
  * [BSH12] Harold C. Burger, Christian J. Schuler and Stefan Harmeling  “Image denoising: Can plain neural networks compete with BM3D?”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012
  * [CG21] George Cazenavette and Manuel Ladron De Guevara  “MixerGAN: An MLP-Based Architecture for Unpaired Image-to-Image Translation”  In _arXiv preprint_ , 2021  arXiv:2105.14110
  * [Che+22] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang and Jian Sun  “Simple Baselines for Image Restoration”  In _European Conference on Computer Vision (ECCV)_ , 2022
  * [Che+21] Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu and Chengpeng Chen  “HINet: Half Instance Normalization Network for Image Restoration”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2021
  * [Che+20] Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan and Zicheng Liu  “Dynamic Convolution: Attention Over Convolution Kernels”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [Cho+21] Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung and Sung-Jea Ko  “Rethinking Coarse-To-Fine Approach in Single Image Deblurring”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [Cho17] François Chollet  “Xception: Deep Learning with Depthwise Separable Convolutions”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017
  * [Chu+21] Xiaojie Chu, Liangyu Chen, , Chengpeng Chen and Xin Lu  “Improving Image Restoration by Revisiting Global Information Aggregation”  In _arXiv preprint_ , 2021  arXiv:2112.04491
  * [Dab+07] K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian  “Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering”  In _IEEE Transactions on Image Processing (TIP)_ , 2007
  * [Deh+22] Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani and Yi Tay  “The Efficiency Misnomer”  In _International Conference on Learning Representations (ICLR)_ , 2022
  * [Din+21] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding and Jian Sun  “RepVGG: Making VGG-style ConvNets Great Again”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021
  * [Dos+21] Alexey Dosovitskiy et al.  “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”  In _International Conference on Learning Representations (ICLR)_ , 2021
  * [Fan+22] Chi-Mao Fan, Tsung-Jung Liu, Kuan-Hsien Liu and Ching-Hsiang Chiu  “Selective Residual M-Net for Real Image Denoising”  In _European Signal Processing Conference (EUSIPCO)_ , 2022
  * [Fra99] Rich Franzen  “Kodak lossless true color image suite”  In _http://r0k.us/graphics/kodak/_ , 1999
  * [Fu+17] Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding and John Paisley  “Removing Rain From Single Images via a Deep Detail Network”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017
  * [GAS20] Shuxuan Guo, Jose M. Alvarez and Mathieu Salzmann  “ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks”  In _Neural Information Processing Systems (NeurIPS)_ , 2020
  * [He+16] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep Residual Learning for Image Recognition”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [HG16] Dan Hendrycks and Kevin Gimpel  “Gaussian Error Linear Units (GELUs)”  In _arXiv preprint_ , 2016  arXiv:1606.08415
  * [HVD15] Geoffrey Hinton, Oriol Vinyals and Jeff Dean  “Distilling the Knowledge in a Neural Network”  In _arXiv preprint_ , 2015  arXiv:1503.02531
  * [How+17] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto and Hartwig Adam  “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications”  In _arXiv preprint_ , 2017  arXiv:1704.04861
  * [HSS18] Jie Hu, Li Shen and Gang Sun  “Squeeze-and-Excitation Networks”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018
  * [Hua+21] Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu and Jianzhuang Liu  “Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021
  * [Jia+22] Yifan Jiang, Bartlomiej Wronski, Ben Mildenhall, Jonathan T. Barron, Zhangyang Wang and Tianfan Xue  “Fast and High-Quality Image Denoising via Malleable Convolutions”  In _European Conference on Computer Vision (ECCV)_ , 2022
  * [KBJ19] Alexander Krull, Tim-Oliver Buchholz and Florian Jug  “Noise2Void - Learning Denoising From Single Noisy Images”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019
  * [Kuo16] C. Kuo  “Understanding Convolutional Neural Networks with A Mathematical Model”  In _Journal of Visual Communication and Image Representation_ , 2016
  * [Leh+18] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala and Timo Aila  “Noise2Noise: Learning Image Restoration without Clean Data”  In _International Conference on Machine Learning (ICML)_ , 2018
  * [Li+23] Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte and Luc Van Gool  “Efficient and Explicit Modelling of Image Hierarchies for Image Restoration”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023
  * [Li+16] Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu and Michael S. Brown  “Rain Streak Removal Using Layer Priors”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [Lia+21] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool and Radu Timofte  “SwinIR: Image Restoration Using Swin Transformer”  In _IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)_ , 2021
  * [LH22] Kang Lin and Reinhard Heckel  “Vision Transformers Enable Fast and Robust Accelerated MRI”  In _Medical Imaging with Deep Learning_ , 2022
  * [Liu+21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin and Baining Guo  “Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [LH19] Ilya Loshchilov and Frank Hutter  “Decoupled Weight Decay Regularization”  In _International Conference on Learning Representations (ICLR)_ , 2019
  * [LH17] Ilya Loshchilov and Frank Hutter  “SGDR: Stochastic Gradient Descent with Warm Restarts”  In _International Conference on Learning Representations (ICLR)_ , 2017
  * [Ma+16] KEde Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li and Lei Zhang  “Waterloo exploration database: New challenges for image quality assessment models”  In _IEEE Transacions on Image Processing (TIP)_ , 2016
  * [Ma+18] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng and Jian Sun  “ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design”  In _European Conference on Computer Vision (ECCV)_ , 2018
  * [MLH22] Youssef Mansour, Kang Lin and Reinhard Heckel  “Image-to-Image MLP-Mixer for Image Reconstruction”  In _arXiv preprint_ , 2022  arXiv:2202.02018
  * [Mar+01] David Martin, Charless Fowlkes, Doron Tal and Jitendra Malik  “A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2001
  * [Mor+20] Nick Moran, Dan Schmidt, Yu Zhong and Patrick Coady  “Noisier2Noise: Learning to Denoise From Unpaired Noisy Data”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [NKL17] Seungjun Nah, Tae Hyun Kim and Kyoung Mu Lee  “Deep multi-scale convolutional neural network for dynamic scene deblurring”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017
  * [Okt+18] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y. Hammerla and Bernhard Kainz  “Attention U-Net: Learning Where to Look for the Pancreas”  In _Medical Imaging with Deep Learning_ , 2018
  * [Pur+21] Kuldeep Purohit, Maitreya Suin, A. N. Rajagopalan and Vishnu Naresh Boddeti  “Spatially-Adaptive Image Restoration Using Distortion-Guided Networks”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021
  * [RFB15] Olaf Ronneberger, Philipp Fischer and Thomas Brox  “U-Net: Convolutional Networks for Biomedical Image Segmentation”  In _Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , 2015
  * [She+19] Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu and Ling Shao  “Human-Aware Motion Deblurring”  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019
  * [Shi+16] Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert and Zehan Wang  “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [SZ15] Karen Simonyan and Andrew Zisserman  “Very Deep Convolutional Networks for Large-Scale Image Recognition”  In _International Conference on Learning Representations (ICLR)_ , 2015
  * [Sri+20] Anuroop Sriram, Jure Zbontar, Tullie Murrell, Aaron Defazio, C. Lawrence Zitnick, Nafissa Yakubova, Florian Knoll and Patricia Johnson  “End-to-End Variational Networks for Accelerated MRI Reconstruction”  In _arXiv preprint_ , 2020  arXiv:2004.06688
  * [TL19] Mingxing Tan and Quoc Le  “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”  In _International Conference on Machine Learning (ICML)_ , 2019
  * [TXZ20] Chunwei Tian, Yong Xu and Wangmeng Zuo  “Image Denoising Using Deep CNN with Batch Renormalization”  In _Neural Networks_ , 2020
  * [Tol+21] Ilya O. Tolstikhin et al.  “MLP-Mixer: An all-MLP Architecture for Vision”  In _Neural Information Processing Systems (NeurIPS)_ , 2021
  * [Tu+22] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik and Yinxiao Li  “MAXIM: Multi-Axis MLP for Image Processing”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022
  * [UVL17] Dmitry Ulyanov, Andrea Vedaldi and Victor Lempitsky  “Instance Normalization: The Missing Ingredient for Fast Stylization”  In _arXiv preprint_ , 2017  arXiv:1607.08022
  * [Vas+23] Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel and Anurag Ranjan  “MobileOne: An Improved One Millisecond Mobile Backbone”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023
  * [Vas+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin  “Attention is All you Need”  In _Neural Information Processing Systems (NeurIPS)_ , 2017
  * [Wan+20] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo and Qinghua Hu  “ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020
  * [Wan+18] Xiaolong Wang, Ross Girshick, Abhinav Gupta and Kaiming He  “Non-Local Neural Networks”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018
  * [Wan+22] Zejin Wang, Jiazheng Liu, Guoqing Li and Hua Han  “Blind2Unblind: Self-Supervised Image Denoising With Visible Blind Spots”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022
  * [Wan+22a] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu and Houqiang Li  “Uformer: A General U-Shaped Transformer for Image Restoration”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022
  * [Wan+04] Zhou Wang, A.C. Bovik, H.R. Sheikh and E.P. Simoncelli  “Image quality assessment: from error visibility to structural similarity”  In _IEEE Transactions on Image Processing_ , 2004
  * [Woo+18] Sanghyun Woo, Jongchan Park, Joon-Young Lee and In So Kweon  “CBAM: Convolutional Block Attention Module”  In _European Conference on Computer Vision (ECCV)_ , 2018
  * [Yan+17] Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo and Shuicheng Yan  “Deep Joint Rain Detection and Removal From a Single Image”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017
  * [Zam+22] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan and Ming-Hsuan Yang  “Restormer: Efficient Transformer for High-Resolution Image Restoration”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022
  * [Zam+21] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang and Ling Shao  “Multi-Stage Progressive Image Restoration”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021
  * [ZF14] Matthew D. Zeiler and Rob Fergus  “Visualizing and Understanding Convolutional Networks”  In _European Conference on Computer Vision (ECCV)_ , 2014
  * [ZP18] He Zhang and Vishal M. Patel  “Density-Aware Single Image De-Raining Using a Multi-Stream Dense Network”  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018
  * [ZSP20] He Zhang, Vishwanath Sindagi and Vishal M. Patel  “Image De-Raining Using a Conditional Generative Adversarial Network”  In _IEEE Transactions on Circuits and Systems for Video Technology_ , 2020
  * [Zha+17] K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang  “Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising”  In _IEEE Transactions on Image Processing_ , 2017
  * [Zho+20] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh and Jianming Liang  “UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation”  In _IEEE Transactions on Medical Imaging_ , 2020


# Comments on ”Necessary optimality conditions of an optimization problem
governed by a double phase PDE”

Omar Benslimane and Nazih Abderrazzak Gadhi 111LAMA, FSDM, Sidi Mohamed Ben
Abdellah University, Fes, Morocco. Emails<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract. This note concerns the paper by Benslimane et Gadhi (JMAA. doi:
10.1016/j.jmaa.2023.127117) where the authors established necessary optimality
conditions for an optimization problem $\left(\mathcal{P}\right)$ governed by
a double phase partial differential equation $\left(\mathcal{P}_{f}\right).$
Having noticed inconsistencies between the investigated problem
$\left(\mathcal{P}_{f}\right)$ and the weak formulation of the equation that
is associated with it, we propose some modifications that, from our
perspective, correct the forthmentioned issue. Some careless mistakes and
typos that caught our interest are also underlined and rectified.

Keywords Double phase partial differential equation; Necessary optimality
conditions; Optimal control problem; $p-$hyperconvexity; Lebesgue space;
Sobolev space.
AMS Subject Classifications: 49J52; 49K20; 90C29; 35J69.

##  1 Introduction

The double-phase operator has piqued the interest of several academics during
the last decade [1, 3, 4, 5, 6, 10, 11, 12, 13]. Let
$\Omega\subseteq\mathbb{R}^{n}$ be an open, bounded and regular set with a
smooth boundary $\partial\Omega$. Let $C_{0}^{\infty}\left(\Omega\right)$ be
the set of infinitely differentiable functions with compact supports on
$\Omega$ [7]. Let $L^{p}\left(\Omega\right)$ and
$\mathcal{W}^{1,p}\left(\Omega\right)$ be the Lebesgue and the Sobolev spaces
defined by

$L^{p}\left(\Omega\right)=\left\\{u:\Omega\rightarrow\mathbb{R}/\int_{\Omega}|u(x)|^{p}\
dx<\infty\right\\}$

and

$\mathcal{W}^{1,p}\left(\Omega\right)=\left\\{u:u\in L^{p}\left(\Omega\right)\
\mbox{and}\ |\nabla u|\in L^{p}\left(\Omega\right)\right\\}.$

In [3], Benslimane and Gadhi investigated the following optimal control
problem

$\left(\mathcal{P}\right):\left\\{\begin{array}[]{rcl}\underset{u,f}{\min}\
\mathcal{E}\left(f,u\right)&&\\\ \mbox{subject to}:u\in\psi\left(f\right),\
u\in\mathcal{W}_{0}^{1,p}\left(\Omega\right),\ f\in
L^{q}\left(\Omega\right)&&\end{array}\right.$

where, for each $f\in L^{q}\left(\Omega\right),\ \psi\left(f\right)$ is the
set of all non-trivial solution of the following double phase problem

$\left(\mathcal{P}_{f}\right):\left\\{\begin{array}[]{rcl}-\mbox{div}\left(|\nabla
u|^{p-2}\nabla u+\mu\left(x\right)\ |\nabla u|^{q-2}\nabla u\right)=f\ \
\mbox{in}\ \ \Omega&&\\\ u=0\ \ \mbox{on}\ \
\partial\Omega.&&\end{array}\right.$ (1)

where $\mathcal{W}_{0}^{1,p}\left(\Omega\right)$ is the closure of
$C_{0}^{\infty}\left(\Omega\right)$ in $W^{1,p}\left(\Omega\right),$
$\mu:\overline{\Omega}\rightarrow\mathbb{R}_{+}^{\ast}$ is a Lipschitz
continuous function and
$\mathcal{E}:L^{q}\left(\Omega\right)\times\mathcal{W}_{0}^{1,p}\left(\Omega\right)\rightarrow\mathbb{R}$
is a weakly lower continuous function, which is continuously Fréchet-
differentiable and bounded from below. The spaces $L^{q}\left(\Omega\right)$
and $\mathcal{W}_{0}^{1,p}\left(\Omega\right)$ are equipped with the norms

$\left\|u\right\|_{L^{q}\left(\Omega\right)}=\left[\int_{\Omega}|u|^{q}\
dx\right]^{\frac{1}{q}}\text{ and
}\left\|u\right\|_{\mathcal{W}_{0}^{1,p}\left(\Omega\right)}=\left[\sum_{i=1}^{n}\int_{\Omega}\left|\frac{\partial
u\left(x\right)}{\partial x_{i}}\right|^{p}dx\right]^{\frac{1}{p}}.$

An element $u\in\mathcal{W}_{0}^{1,p}\left(\Omega\right)$ was said to be a
non-trivial solution of $\left(P\right)$ if

$\sum_{i=1}^{n}\int_{\Omega}\left(\bigg{|}\frac{\partial
u\left(x\right)}{\partial x_{i}}\bigg{|}^{p-2}\frac{\partial
u\left(x\right)}{\partial x_{i}}+\mu\left(x\right)\bigg{|}\frac{\partial
u\left(x\right)}{\partial x_{i}}\bigg{|}^{q-2}\frac{\partial
u\left(x\right)}{\partial x_{i}}\right)\ \frac{\partial\varphi(x)}{\partial
x_{i}}\ dx=\int_{\Omega}f\left(x\right)\varphi\left(x\right)\ dx$ (2)

is satisfied for all test function
$\varphi\in\mathcal{W}_{0}^{1,p}\left(\Omega\right).$ In order to get
necessary optimality conditions for the optimal control problem
$\left(\mathcal{P}\right),$ the authors made use of an energy function
$\mathcal{J}:\mathcal{W}_{0}^{1,p}\left(\Omega\right)\rightarrow\mathbb{R}$
defined by

$\mathcal{J}\left(u\right)=\frac{1}{p}\sum_{i=1}^{n}\int_{\Omega}\bigg{|}\frac{\partial
u\left(x\right)}{\partial x_{i}}\bigg{|}^{p}\
dx+\frac{1}{q}\sum_{i=1}^{n}\int_{\Omega}\mu\left(x\right)\
\bigg{|}\frac{\partial u\left(x\right)}{\partial x_{i}}\bigg{|}^{q}\
dx-\int_{\Omega}f\left(x\right)u\left(x\right)\ dx.$

Their method involved proving first the energy function $\mathcal{J}$’s
$p$-hyperconvexity and Gâteaux differentiability. This allowed them to
conclude that the solution operator
$\psi:L^{q}\left(\Omega\right)\longrightarrow\mathcal{W}_{0}^{1,p}\left(\Omega\right)$
associated with $\left(\mathcal{P}_{f}\right)$ is both well-defined and
Gâteaux-differentiable. Following that, they proved that
$\left(\mathcal{P}_{f}\right)$ admits a unique non-trivial solution
$\mathbb{\psi}\left(f\right)\in\mathcal{W}_{0}^{1,p}\left(\Omega\right),$
which enables them to determine necessary optimality conditions of the optimal
control problem $\left(\mathcal{P}\right)$ of interest.

In this note, we underline the fact that energy function $\mathcal{J}$ is not
well defined and that the weak formulation of the equation
$\left(\ref{33}\right)$ does not correspond to the double phase problem
$\left(\ref{12}\right),$ but corresponds in fact to the pseudo-double phase
problem $\left(\ref{bbg}\right),$ defined in the next section. Adjustments
that we believe will fix these awkward circumstances are then proposed. Some
typos and careless mistakes that drew our attention are also highlighted and
then corrected.

##  2 Comments and corrections

First and foremost, simply replacing

$p<q\text{ and }\frac{1}{q}=\frac{1}{p}-\frac{1}{n}$

with

$q<p\text{ and }\frac{1}{p}=\frac{1}{q}-\frac{1}{n}$

renders the energy functional $\mathcal{J}$ well-defined. Accordingly, to
retain the accuracy of the proof of [3, Lemma 7], it is clear that

$\mathcal{X}_{i}\left(x\right)=\frac{1}{p}\left(\left|\frac{\partial
u}{\partial x_{i}}+\frac{\partial v}{\partial
x_{i}}\right|^{p}+\mu\left(x\right)\left|\frac{\partial u}{\partial
x_{i}}+\frac{\partial v}{\partial x_{i}}\right|^{q}\right),\ i\in I,$

should be replaced with

$\mathcal{X}_{i}\left(x\right)=\frac{1}{q}\left(\left|\frac{\partial
u}{\partial x_{i}}+\frac{\partial v}{\partial
x_{i}}\right|^{p}+\mu\left(x\right)\left|\frac{\partial u}{\partial
x_{i}}+\frac{\partial v}{\partial x_{i}}\right|^{q}\right),\ i\in I,$

within this proof. On the other hand, in order to address the unconstency
between $\left(\ref{12}\right)$ and $\left(\ref{33}\right),$ we propose to
replace the standard $p$-Laplacian in $\left(\mathcal{P}_{f}\right)$ by the
pseudo-$p$-Laplacian. By doing this, the optimal control problem
$\left(\mathcal{P}\right)$ becames

$\left(\mathcal{P}\right):\left\\{\begin{array}[]{rcl}\underset{u,f}{\min}\
\mathcal{E}\left(f,u\right)&&\\\ \mbox{subject to}:u\in\psi\left(f\right),\
u\in\mathcal{W}_{0}^{1,p}\left(\Omega\right),\ f\in
L^{q}\left(\Omega\right)&&\end{array}\right.$

where, for each $f\in L^{q}\left(\Omega\right),\ \psi\left(f\right)$ is the
set of all non-trivial solution of the following pseudo-double phase problem

$\left(\mathcal{P}_{f}\right):\left\\{\begin{array}[]{rcl}-\sum_{i=1}^{n}\frac{\partial}{\partial
x_{i}}\left(\left|\frac{\partial u}{\partial x_{i}}\right|^{p-2}\frac{\partial
u}{\partial x_{i}}+\mu\left(x\right)\left|\frac{\partial u}{\partial
x_{i}}\right|^{q-2}\frac{\partial u}{\partial x_{i}}\right)=f\ \ \mbox{in}\ \
\Omega&&\\\ u=0\ \ \mbox{on}\ \ \partial\Omega.&&\end{array}\right.$ (3)

This adjustment has the advantage of preserving the relationship, which should
tie the Euler-Lagrange equation $\left(\ref{33}\right)$ to the pseudo-double
phase problem $\left(\mathcal{P}_{f}\right).$ It is worth mentionning that the
pseudo-$p$-Laplacian [2, 8] defined by

$\widetilde{\Delta}_{p}u:=\sum_{i=1}^{n}\frac{\partial}{\partial
x_{i}}\left(\left|\frac{\partial u}{\partial x_{i}}\right|^{p-2}\frac{\partial
u}{\partial x_{i}}\right)$

has some different features compared to the standard $p$-Laplacian

$\Delta_{p}u:=\mathop{\mathrm{d}iv}\left(\left|\nabla u\right|^{p-2}\nabla
u\right)$

appearing in $\left(\ref{12}\right),$ in particular concerning regularity
results. However, it is worth mentioning that these two operators coincide in
the case $p=1,$ because

$\widetilde{\Delta}_{p}u=\Delta_{p}u=\left(\left|u^{\prime}\right|^{p-2}u^{\prime}\right)^{\prime}=\left(p-1\right)\left|u^{\prime}\right|^{p-2}u^{\prime\prime}$

for $C^{2}$-functions. Taking into account the previously indicated
modification,

$-\mbox{div}\left(|\nabla u|^{p-2}\nabla u+\mu\left(x\right)\ |\nabla
u|^{q-2}\nabla u\right)$

should be replaced with

$-\sum_{i=1}^{n}\frac{\partial}{\partial x_{i}}\left(\bigg{|}\frac{\partial
u}{\partial x_{i}}\bigg{|}^{p-2}\frac{\partial u}{\partial
x_{i}}+\mu\left(x\right)\bigg{|}\frac{\partial u}{\partial
x_{i}}\bigg{|}^{q-2}\frac{\partial u}{\partial x_{i}}\right)$

throughout the document for all
$u\in\mathcal{W}_{0}^{1,p}\left(\Omega\right).$

Following that, we suggest that in the definition of the
$\gamma$-hyperconvexity [3, Definition 1], the reel $c$ be strictly positive.
In addition, to have a factually double phase problem, the codomain of the
weight-function $\mu$ should be $\mathbb{R}_{+}$ and [3, Assumption 1] should
be revised as follows.

$\text{There exists }\mu_{1}>0\text{ such that for all
}x\in\mathcal{W}_{0}^{1,q}\left(\Omega\right),\text{ we have
}0\leq\mu\left(x\right)\leq\mu_{1}.$

It should be noted that the aforementioned changes have no effect on the
proofs or findings presented in [3], but it does allow for the energy
functional to exhibit two different kinds of growth on the
$\left\\{\mu=0\right\\}$ and $\left\\{\mu>0\right\\}.$

Our final comment concerns the proof of [3, Lemma 5]. After some true steps,
the authors have concluded that

$\left(h+g\right)\left(\theta x+\left(1-\theta\right)y\right)\leq\theta\
\left(h+g\right)\left(x\right)+\left(1-\theta\right)\
\left(h+g\right)\left(y\right)-\min\left(\theta,\left(1-\theta\right)\right)\
\left[c\ \|x-y\|^{p}+c^{\prime}\ \|x-y\|^{q}\right].$

From our perspective, the remaining of the proof should be replaced by the
following: since $c^{\prime}\ \|x-y\|^{q}\geq 0,$ we get

$\left(h+g\right)\left(\theta x+\left(1-\theta\right)y\right)+c\
\min\left(\theta,\left(1-\theta\right)\right)\ \|x-y\|^{p}\leq\theta\
\left(h+g\right)\left(x\right)+\left(1-\theta\right)\
\left(h+g\right)\left(y\right),$

which implies the $p-$hyperconvexity of the sum-function $h+g.$

##  3 Conclusion

In this note, we highlight certain inconsistencies in [3] and then propose
some modifications that, from our perspective, correct them.

## References

  * [1] Baroni, P., Colombo, M. and Mingione, G.: Regularity for general functionals with double phase. Calc. Var. Partial Differential Equations 57, 1-48 $\left(2018\right).$
  * [2] Belloni, M. and Kawohl, B.: The pseudo-$p$-Laplace eigenvalue problem and viscosity solutions as $p\rightarrow\infty.$ ESAIM: COCV 10, 28-52 $\left(2004\right).$
  * [3] Benslimane, O. and Gadhi, N.A.: Necessary optimality conditions of an optimization problem governed by a double phase PDE. J. Math. Anal. Appl. 524 $\left(2023\right)$ 127117\. https://doi.org/10.1016/j.jmaa.2023.127117
  * [4] Colasuonno, F. and Squassina, M.: Eigenvalues for double phase variational integrals. Annali di Matematica Pura ed Applicata 195, 1917-1959 $\left(2016\right).$
  * [5] Farkas, C. and Winkert, P.: An existence result for singular Finsler double phase problems. Journal of Differential Equations 286, 455-473 $\left(2021\right).$
  * [6] Gasiński, L. and Winkert, P.: Existence and uniqueness results for double phase problems with convection term. J. Differential Equations 268, 4183-4193 $\left(2020\right).$
  * [7] Harjulehto, P. and Hästö, P.: Orlicz Spaces and Generalized Orlicz Spaces. Springer, Cham, 47-78 $\left(2019\right).$
  * [8] Lions, J.L.: Quelques méthodes de résolutions des problèmes aux limites non linéaires. Dunod, Gauthier-Villars, Paris $\left(1969\right).$
  * [9] Liu, W. and Dai, G.: Existence and multiplicity results for double phase problem. Journal of Differential Equations 265, 4311-4334 $\left(2018\right).$
  * [10] Liu, Z. and Papageorgiou, N.: A double phase equation with convection. Electronic Journal of Qualitative Theory of Differential Equations 2021, 1-11 $\left(2021\right).$
  * [11] Tam, VM.: Upper-Bound Error Estimates for Double Phase Obstacle Problems with Clarke’s Subdifferential. Numerical Functional Analysis and Optimization 43, 463-485 $\left(2022\right).$
  * [12] Zeng, S., Bai. Y., Gasiński, L. and Winkert, P. : Existence results for double phase implicit obstacle problems involving multivalued operators. Calc. Var. Partial Differential Equations 59, 1-18 $\left(2020\right).$
  * [13] Zeng, S., Bai. Y., Yao, JC. and Nguyen, V.T : A Class of Double Phase Mixed Boundary Value Problems: Existence, Convergence and Optimal Control. Applied Mathematics & Optimization 86, 1-29 $\left(2022\right).$