(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Hangzhou Dianzi University,
11email: 211330013, yinxs<EMAIL_ADDRESS>

# Restorer: Solving Multiple Image Restoration Tasks with One Set of
Parameters

Jiawei Mao 11 0000-1111-2222-3333 Xuesong Yin 11 1111-2222-3333-4444 Yuanqi
Chang 11 2222–3333-4444-5555

###### Abstract

Although there are many excellent solutions in image restoration, the fact
that they are specifically designed for a single image restoration task may
prevent them from being state-of-the-art (SOTA) in other types of image
restoration tasks. While some approaches require considering multiple image
restoration tasks, they are still not sufficient for the requirements of the
real world and may suffer from the task confusion issue. In this work, we
focus on designing a unified and effective solution for multiple image
restoration tasks including deraining, desnowing, defogging, deblurring,
denoising, and low-light enhancement. Based on the above purpose, we propose a
Transformer network Restorer with U-Net architecture. In order to effectively
deal with degraded information in multiple image restoration tasks, we need a
more comprehensive attention mechanism. Thus, we design all-axis attention
(AAA) through stereo embedding and 3D convolution, which can simultaneously
model the long-range dependencies in both spatial and channel dimensions,
capturing potential correlations among all axis. Moreover, we propose a
Restorer based on textual prompts. Compared to previous methods that employ
learnable queries, textual prompts bring explicit task priors to solve the
task confusion issue arising from learnable queries and introduce
interactivity. Based on these designs, Restorer demonstrates SOTA or
comparable performance in multiple image restoration tasks compared to
universal image restoration frameworks and methods specifically designed for
these individual tasks. Meanwhile, Restorer is faster during inference. The
above results along with the real-world test results show that Restorer has
the potential to serve as a backbone for multiple real-world image restoration
tasks.

###### Keywords:

Image Restoration Severe Weather Removal Textual Prompt

##  1 Introduction

Environments such as severe weather (rain, fog, and snow) and low light can
reduce image visibility. Cameras produce issues such as noise and blurring
when shooting. All these negative factors dramatically affect many computer
vision algorithms especially object detection, semantic segmentation, and
depth estimation which are crucial components of surveillance systems,
autonomous vehicle systems, or edge devices [37, 42, 43, 44]. This appears to
be particularly important for image restoration tasks for these degradations.

Figure 1: Overview of existing all-in-one image restoration methods and
Restored images of TransWeather and Restorer based on different textual
prompts on the RealBlur dataset. Restorer achieves promising performance in
more image restoration tasks with shorter inference time. TransWeather
confuses the low-light enhancement task when performing the deblurring task,
resulting in poor deblurring results. In contrast, Restorer at different
textual prompts strictly performed the corresponding image restoration task.

To tackle these tasks, various image restoration algorithms like image
deraining [49, 65, 17], desnowing [38, 8, 73], defogging [63, 45, 12],
deblurring [41, 74, 29], denoising [48, 72, 22], and low-light image
enhancement [15, 60] have been widely explored nowadays. Despite their
promising performance, they may not be optimal in other types of image
restoration tasks because they are specifically designed for a single type of
degradation. A simple idea is to switch between a range of different types of
image restoration algorithms. However, this is undoubtedly costly. Li et al.
[35] searched for appropriate weights in several severe weather frameworks
with neural architecture search (NAS) technique to address this issue.
Although this approach has achieved better or comparable results in several
severe weather image restoration tasks, it requires multiple encoders to be
selected based on the type of image restoration task, making it inefficient.
To address the above issue, Chen et al. [9] proposed a single encoder
framework TKL through two-stage knowledge distillation and multi-contrastive
regularization achieving promising results on different severe weather image
restoration tasks simultaneously. Valanarasu et al. [56] proposed
TransWeather, a Transformer network with only one encoder and one decoder,
which adapts to a specific severe weather image restoration task by learnable
weather type embeddings.

Although TransWeather and TKL achieve encouraging results in different types
of image restoration tasks, they are still insufficient for the requirements
of real-world applications because degradations such as low light, noise, and
blurring in the real world also affect the computer vision algorithm
performance. Meanwhile, it is complex for TKL to find multiple appropriate
specific task image restoration models for two-stage distillation. While
TransWeather works well in spatially removing weather degradation at different
scales, its network design ignores channel information that is equally
important for image restoration tasks resulting in its performance still
falling short on some image restoration tasks. Furthermore, the learnable
weather type embeddings in Transweather lack interpretability and clear
definitions. This may lead to confusion between different image restoration
tasks (see Figure 9).

Figure 2: Illustration of spatial attention, channel attention, omni self-
attention, and all-axis attention.

In this work, we propose the Transformer network Restorer with U-Net [51]
architecture to face the above challenges. It can efficiently address multiple
image restoration tasks including image deraining, desnowing, defogging,
deblurring, denoising, and low-light image enhancement. To handle the
different degradation information in image restoration tasks mentioned above,
Restorer’s Transformer module needs a more comprehensive attention mechanism.
Thus, we propose the all-axis attention (AAA) module in Restorer by design of
stereo embedding and 3D convolution. All channels of the feature map in
spatial attention share the attention weights obtained by calculating the
cross-covariance of the spatial directions. This attention modeling scheme
ignores the channel information in the feature map. However, [68, 58]
experimental results show that attention modeling schemes for channel
dimensions are also crucial in image restoration tasks, they are
computationally more compact than spatial attention. Pixels on all spaces in
the channel attention modeling scheme share channel attention weights. This
scheme ignores spatial information, leading to a significant reduction in
relation modeling capability impairing model performance. Our AAA extends the
interaction to 3D space, fulfilling the complementary strengths of these two
attention modeling schemes. Thus, AAA is more suitable for handling multiple
image restoration tasks. In addition, we introduce learnable task queries in
AAA to predict task-related degenerate feature vectors. In order to preserve
the fine details in the stereo patches, we also design a 3D deep convolutional
feed-forward network (3D-DCFFN) for the stereo patches in the AAA module to
perform local modeling of the stereo patches in both space and channel at
once. Experiments show that 3D-DCFFN facilitates Restorer to obtain high-
fidelity restored images. To address the task confusion issue caused by
learnable task queries, we propose a version of Restorer based on textual
prompts. Compared to learnable task queries that lack interpretability,
textual prompts have a deterministic task prior thus effectively solving the
task confusion issue and introducing interactivity for Restorer to let the
user decide what to restore for the image. Additionally, we find that Restorer
with text prompts also solves the issue of multiple degradations in the same
image.

Experiments on numerous standard datasets show that Restorer achieves state-
of-the-art (SOTA) or competitive results on multiple image restoration tasks
compared to universal image restoration frameworks and methods specifically
designed for these individual tasks. Meanwhile, the results on the real-world
dataset show that Restorer is robust enough. Figure 9 indicates that Restorer
is more favorable in efficiency. These results together demonstrate the
potential of Restorer as a real-world application for image restoration tasks.

Figure 3: The overall architecture of the proposed Restorer and the structure
of each Restorer component. The upsampling and downsampling operations are
realized by transposed convolution and convolution with a stride of 2,
respectively.

##  2 Related Work

###  2.1 Universal Image Restoration Tasks

Zamir et al. [70] proposed MPRNet with a multi-stage architecture, which
utilizes in-situ supervised attention to progressively learn the restoration
function for degraded inputs. Mou et al. [39] integrated a gradient estimation
strategy into the gradient descent step of the Proximal Gradient Descent (PGD)
algorithm to propose DGUNet for the image restoration task. The Uformer
proposed by Wang et al. [61] has achieved success in image restoration through
the design of a localized enhancement window transformer structure. Tu et al.
[55] proposed MAXIM, the first general-purpose MLP architecture for low-level
vision, which is more efficient and flexible in image restoration tasks. Chen
et al. [6] proposed a simple baseline NAFNet for the image restoration task
through extensive network design experiments. Li et al. [36] proposed GRL
performing image restoration tasks that achieve a balance between the spatial
and temporal complexity and modeling capabilities of self-attention through
the design of anchored stripe self-attention.

Although these methods achieve satisfactory image restoration results,
however, degradation residuals may exist not only in the feature map space
dimension but also in the channel dimension when performing image restoration
tasks. This means that image restoration needs to consider both spatial and
channel modeling, especially for unified image restoration models for multiple
low-level visual tasks. While some work [58] has explored this, they typically
employ two stages of modeling for both space and channel, and this asynchrony
from different stages prevents image restoration algorithms from fully
exploiting potential correlations between all axis of a degraded image. Thus,
the stereo spatial attention design that combines channel dimensions in the
AAA Module is better suited to the requirements of a unified image restoration
task.

###  2.2 All-in-One Image Restoration Tasks

Li et al. [35] designed All in One for image restoration with a scheme of
neural architecture search (NAS) and adversarial training based on
classification. However, All in One requires multiple specific encoders to be
selected based on the type of degradation, which definitely affects the
efficiency. To address this problem, Valanarasu et al. [56] were inspired by
DETR [3] to design transformers with learnable weather type embeddings to
adapt to the weather degradation at hand. Chen et al. [9] learned crucial
features for different types of degradation from multiple encoders through
two-stage knowledge distillation and multi-contrast knowledge regularization.
Li et al. [34] restored various degraded images in a single network by
contrastive-based degradation encoder and degradation-guided restoration
network.

The main challenge of current unified image restoration methods lies in the
mutual interference between different tasks, while contrastive learning and
learnable weather type embeddings can alleviate this issue to a certain
extent, in some composite degradation scenes (e.g., low-light blurring
scenes), the interference between different tasks will be more serious, which
may lead to the failure of these designs (see Figure 9). The introduction of
text prompts in Restorer is a favorable solution to the above problem. Users
can explicitly perform specific image restoration tasks by instructing the
model, thus eliminating interference between tasks. In addition, Restorer can
remove compound degradation in real scenes based on iterative prompts.

Figure 4: Visualization of feature maps for each channel before and after the
AAA module for different image restoration tasks. It can be observed that
there are significant degradation residuals in the inter-channel feature maps
before the AAA module, while the degradation residuals among the channels are
eliminated after the AAA module. Please zoom in to see the details better.

##  3 Proposed Method

Our main goal is to develop an efficient Transformer model that meets the
needs of real-world image restoration, which can be quickly applied to
multiple image restoration tasks. In this section, we first describe the
overall pipeline of Restorer (see Figure 3). Then, we discuss the core
components related to the all-axis attention module in Restorer: the stereo
embedding, the all-axis attention, 3D deep convolutional feedforward networks,
and Restorer based on textual prompts. Finally, we introduce the loss
function.

###  3.1 Overall Pipeline

Given a degraded image, Restorer transforms it from low-level features to
high-level features via four-level symmetric encoder-AAA-decoder architecture.
As shown in Figure 3, Restorer’s encoder uses a compact convolutional
architecture to extract low-level feature embeddings for computational
efficiency. The design of the encoder-AAA-decoder in different stages
guarantees Restorer’s ability to effectively capture visual information at
different granularities like U-Net [51]. For different stages of the encoder
output, we first transform it into a sequence of stereo token embeddings by
stereo embedding. Then AAA mines potential dependencies between the learnable
task queries representing task-related degraded feature vectors and the
degraded feature in the all-axis based on stereo patches and filters out the
degenerate features from the feature embedding in both spatial and channel
dimensions via negative affinity matrices. Next, the AAA module performs fine-
grained feature extraction of stereo patches through 3D-DCFFN to preserve the
high-frequency information in the image. For the stereo patch output from the
AAA module, the convolutional decoder converts them into feature maps through
the patch reversion layer and progressively restores a high-resolution
representation by stacked convolution and transposed convolution while
preserving the fine details of the image. Convolution has a natural advantage
over Transformer in local modeling. Eventually, a high-fidelity restored image
is obtained from the $3\times 3$ convolutional projection block.

###  3.2 All-Axis Attention Module

As illustrated in Figure 2, the one-dimensional attention operator is limited
by space or channel dimension thus failing to exploit the full potential of
the attention mechanism in image restoration tasks. For the spatial attention
operator [5, 57], since the channels share the same spatial weights, it can
only mine potential correlations in the degraded image space, ignoring the
explicit use of channel information. Recent studies [68, 58] have shown that
the channel attention operator is more compact than the spatial attention
operator, it also plays an important role in image restoration tasks. However,
the channel attention operator ignores critical spatial information in the
image restoration task leading to a drastic reduction in the relational
modeling capability compromising the accuracy of aggregation (see Figure 6),
especially for unified image restoration tasks. Although Wang et al. proposed
OSA [58] to lift the limitation of the one-dimensional attention operator,
this approach requires two stages of modeling through both spatial attention
and channel attention. This asynchrony prevents OSA from fully mining the
potential dependencies between the all-axis of degraded images. In contrast,
our AAA truly extends the interaction to 3D space through stereo embedding,
which enjoys the complementary spatial and channel advantages of the one-
dimensional attention operator, making it more suitable for image restoration
tasks.

#### Stereo Embedding.

For the AAA module in different stages, Restorer employs stereo embedding to
divide the feature map
$\rm{F}\in{\mathbb{R}^{\rm{h}\times\rm{w}\times\rm{c}}}$ ($\rm{h}\times\rm{w}$
denotes the spatial dimension and $\rm{c}$ denotes the number of channels)
into stereo token embeddings. Specially, the stereo embedding first reshapes
the feature map into a sequence of stereo tokens
$\rm{E}\in{\mathbb{R}^{\rm{p}\times\rm{p}\times\rm{d}\times(\rm{n_{p}^{2}n_{d}})}}$;
where $\rm{p^{2}}$ and $\rm{d}$ denote the stereo token space dimension and
the channel dimension, respectively,
$\rm{N}=\rm{n_{p}}\times\rm{n_{p}}\times\rm{n_{d}}=\rm{hwc}/\rm{p^{2}d}$ is
the resulting number of stereo tokens. The stereo embedding then replaces the
time dimension in the 3D convolution with the channel dimension to map the
stereo token, thus introducing stereo spatial information for each stereo
token. Next, we flatten the stereo token sequence and project it to a constant
potential vector size D with linear layer to obtain the stereo token
embeddings $\rm{E}\in{\mathbb{R}^{\rm{N}\times\rm{D}}}$ like ViT [14]. To
stabilize the learning process and improve convergence during training, the
stereo embedding is treated with layer normalization [2] for stereo token
embeddings. Finally, we add learnable position embeddings $\rm{E_{pos}}$ for
stereo token embeddings to preserve stereo position information (see Equation
1). Compared to other token embedding division ways, stereo token embeddings
divided in 3D space allow for the modeling of long-range dependencies in both
space and channel through attention.

$\begin{split}&{\rm{F}}=[{\rm{f^{(1)}}};{\rm{f^{(2)}}};...;{\rm{f^{(i)}}};...;{\rm{f^{(N)}}}],{\rm{f^{(i)}}}\in{\mathbb{R}^{\rm{p}\times\rm{p}\times\rm{d}}},\\\
&{\rm{Z}}=\rm{Flat(3DConv(F))},\\\
&{\rm{Z}}=[{\rm{z^{(1)}W}};{\rm{z^{(2)}W}};...;{\rm{z^{(i)}W}};...;{\rm{z^{(N)}W}}],{\rm{z^{(i)}}}\in{\mathbb{R}^{\rm{p^{2}d}}},{\rm{W}}\in{\mathbb{R}^{\rm{(p^{2}d)}\times\rm{D}}},\\\
&{\rm{Z}}={\rm{LN(Z)+E_{pos}}},{\rm{E_{pos}}}\in{\mathbb{R}^{\rm{N}\times\rm{D}}},\end{split}$
(1)

where $\rm{3DConv(.)}$ denotes 3D convolution, $\rm{Flat(.)}$ is the
flattening operation, $\rm{W}$ is the linear layer weights, and $\rm{LN(.)}$
is the layer normalization operation.

#### All-Axis Attention.

For the stereo patches $\rm{z}\in{\mathbb{R}^{\rm{N}\times\rm{D}}}$ obtained
by stereo token embeddings, we perform a multi-head attention computation
between them and the learnable task queries
$\rm{t}\in{\mathbb{R}^{\rm{N}\times\rm{D}}}$ used to fit the task-related
degenerate feature vectors:

$\begin{split}&{\rm{q}}={\rm{tW_{q}}},{\rm{k}}={\rm{zW_{k}}},{\rm{v}}={\rm{zW_{v}}},\\\
&{\rm{o^{(n)}}}=\rm{\theta}(0-\rm{q^{(n)}k^{(n)\intercal}}/\sqrt{d}){\rm{v^{(n)}}},{\rm{n}}=1,...,{\rm{H}},\\\
&{\rm{o}}=[{\rm{o^{(1)}}};...;{\rm{o^{(H)}}}]{\rm{W_{o}}},\end{split}$ (2)

where $\rm{W_{q}},\rm{W_{k}},\rm{W_{v}},\rm{W_{o}}$ represent the mapping
matrices, respectively. $\rm{H}$ indicates the number of heads.
$\rm{\theta}(.)$ is a softmax function, and $\rm{d}$ is the embedding
dimension of each header to avoid gradient vanishing. Unlike vanilla multi-
head attention, AAA adopts negative affinity matrices. Based on these negative
affinity matrices, the Restorer can assign attention weights that are not
related to the degradation in all axis through latent dependencies between the
learnable task queries and the stereo patches thus effectively removing the
degradation information from the stereo patches in both spatial and channel
(see Figure 4). In addition, to further enhance the robustness of AAA to
scale, we have also established connections for AAA modules at different
stages.

#### 3D Deep Convolutional Feedforward Networks.

A Feedforward network with deep convolution facilitates Transformer to extract
fine-grained features. This contributes to the restoration image retaining the
fine details in the input image thus the AAA module designs the 3D deep
convolutional feedforward network (3D-DCFFN). Unlike other work on
convolutional feedforward networks [24, 19, 20], the 3D-DCFFN in our work does
not capture the local information of the whole feature map but rather the
fine-grained features of each stereo token. In contrast, convolution acting on
a feature map may lack sensitivity to fine-grained location information to
some extent. Therefore, for image restoration tasks that emphasize fine
restoration of exact locations, 3D-DCFFN may be more suitable. Meanwhile,
3D-DCFFN not only considers the spatial information in the tokens but also
pays attention to the channel information of the tokens at the same time,
which has not been concerned by previous work. First, we map the AAA output to
the hidden dimensions with a linear layer and reshape it back to stereo tokens
$\rm{E}\in{\mathbb{R}^{\rm{p}\times\rm{p}\times\rm{d}\times(\rm{n_{p}^{2}n_{d}})}}$.
For stereo tokens, we use 3D deep convolution to extract fine-grained features
in 3D space. Finally, 3D-DCFFN reshapes the stereo tokens back to 2D stereo
patches and maps them back to the input dimensions by linear layer. Appendix D
indicates that the 3D-DCFFN based on the above design effectively improves the
quality of restored images by performing local modeling of the stereo patches
in both spatial and channel dimensions.

###  3.3 Restorer based on textual prompts

Figure 5: Restorer’s image restoration results at different textual prompts on
SIDD.

Although TransWeather has shown satisfactory results in multiple severe
weather image restoration tasks, the lack of a clear definition of learnable
task queries makes it difficult to achieve an accurate distinction between
image restoration task types in the face of various degradations leading to
task confusion issue. To address this issue, we propose Restorer based on
textual task prompts. Specifically, we represent degraded types of text (e.g.,
"low light", "rain", "blur", etc.) with the text encoder from CLIP [47] to
obtain textual prompts. Textual prompts would replace learnable task queries
into the AAA module for each stage of Restorer. As shown in Figure 9, compared
with learnable task queries, this textual prompt with the task prior provides
clear instructions for Restorer’s image restoration effectively solving the
task confusion issue caused by learnable task queries thus further improving
Restorer’s performance in each image restoration task. At the same time,
textual prompts introduce significant interactivity into Restorer, allowing
the user to decide what restoration tasks need to be performed on the image by
themselves. In addition, we found that Restorer can remove complex compound
degenerates from an image by stacking textual prompts in the real world (see
Figure 9). To the best of our knowledge, this is what previous image
restoration methods fail to do.

Furthermore, we also show the performance of Restorer on SIDD based on
different textual prompts in Figure 5. It can be observed that Restorer based
on different prompts also exhibits similar performance on SIDD as on the
RealBlur-R dataset. Restorer also strictly follows the textual prompts to
perform the appropriate image restoration tasks. Moreover, by stacking text
prompts Restorer also removes the complex degradation of noise and low-light
composites. This suggests that the success of Restorer based on textual
prompts is no accident. We hope that this finding will be useful for composite
degraded restoration tasks.

Figure 6: Visual comparison with SOTA image restoration algorithms in
different image restoration tasks.

###  3.4 Overall Loss

Restorer uses smooth L1-loss and perceptual loss in TransWeather to restore
the pixels and content of the input image. The total loss is formulated as
follows:

${L_{total}}={L_{\rm{smoothL_{1}}}}+{\lambda}{L_{\rm{perceptual}}},$ (3)

where ${\lambda}=\rm{0.04}$ is used to control the contribution of perceptual
loss in the total loss of Restorer.

##  4 Experiments

We evaluated Restorer on six different image restoration tasks: deraining,
desnowing, defogging, deblurring, denoising, and low-light image enhancement.
Ablation study, more specific experimental details, and more comprehensive
results can be found in the Appendix.

###  4.1 Experimental Setup

| deraining | desnowing | defogging | denoising | low-light enhancement | deblurring |
---|---|---|---|---|---|---|---
| rain1400 | CSD | SOTS | SIDD | LOL | GoPro | RealBlur-R | Average
Method | PSNR $\uparrow$ | SSIM($\%$) $\uparrow$ | PSNR $\uparrow$ | SSIM($\%$) $\uparrow$ | PSNR $\uparrow$ | SSIM($\%$) $\uparrow$ | PSNR $\uparrow$ | SSIM($\%$) $\uparrow$ | PSNR $\uparrow$ | SSIM($\%$) $\uparrow$ | PSNR $\uparrow$ | SSIM($\%$) $\uparrow$ | PSNR $\uparrow$ | SSIM($\%$) $\uparrow$ | PSNR $\uparrow$ | SSIM($\%$) $\uparrow$
GRL [36] | 20.56 | 79.71 | 20.45 | 80.01 | 21.71 | 84.97 | 33.81 | 93.53 | 7.09 | 3.82 | 18.50 | 77.94 | 27.58 | 49.15 | 21.42 | 67.01
Uformer [61] | 31.74 | 95.49 | 27.46 | 92.61 | 29.85 | 97.82 | 38.35 | 96.26 | 8.50 | 26.84 | 25.89 | 91.38 | 37.13 | 95.86 | 28.41 | 85.18
DGUNet [39] | 33.05 | 96.52 | 28.81 | 92.28 | 31.04 | 96.22 | 39.61 | 97.04 | 13.04 | 55.37 | 30.06 | 94.04 | 37.74 | 96.36 | 30.47 | 89.69
MPRNet [70] | 33.47 | 96.59 | 28.03 | 92.57 | 30.96 | 97.40 | 38.94 | 96.75 | 15.78 | 71.15 | 28.39 | 94.10 | 37.50 | 96.75 | 30.43 | 92.18
Restormer [68] | 24.38 | 87.60 | 14.30 | 72.67 | 15.94 | 82.92 | 25.30 | 65.35 | 7.79 | 19.79 | 26.22 | 87.15 | 34.42 | 96.83 | 21.19 | 73.18
MAXIM [55] | 32.63 | 96.80 | 27.81 | 92.20 | 32.20 | 98.14 | 27.70 | 95.95 | 13.48 | 78.33 | 25.78 | 93.39 | 32.52 | 91.87 | 27.44 | 92.38
NAFNet [6] | 32.87 | 97.32 | 28.31 | 92.65 | 32.34 | 96.74 | 29.89 | 96.18 | 21.57 | 89.37 | 25.58 | 93.46 | 32.91 | 95.18 | 29.06 | 94.41
TransWeather [56] | 34.27 | 96.25 | 29.38 | 92.22 | 32.96 | 96.55 | 36.14 | 95.69 | 15.72 | 62.90 | 29.03 | 93.78 | 39.00 | 93.06 | 30.92 | 90.06
SYENet [18] | 30.02 | 89.11 | 22.55 | 87.35 | 26.37 | 95.66 | 33.19 | 91.03 | 17.64 | 81.57 | 26.78 | 93.30 | 35.90 | 94.92 | 27.49 | 90.42
All in One [34] | 24.63 | 83.67 | 18.40 | 72.28 | 18.14 | 76.86 | 35.10 | 87.30 | 11.24 | 32.10 | 24.18 | 85.88 | 26.94 | 85.57 | 22.66 | 74.80
Ours | 33.61 | 96.33 | 29.93 | 92.71 | 34.19 | 98.21 | 37.74 | 96.36 | 21.60 | 89.86 | 30.12 | 94.85 | 41.68 | 95.20 | 32.69 | 94.78
Ours+Prompts | 34.21 | 96.70 | 30.28 | 93.09 | 34.46 | 98.25 | 37.75 | 96.36 | 21.44 | 90.53 | 30.92 | 95.48 | 43.10 | 97.55 | 33.16 | 95.42
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Table 1: Quantitative Comparison with SOTA unified image restoration methods
and all-in-one image restoration methods.

#### Datasets and metrics.

We employ PSNR and SSIM ($\%$) metrics to evaluate Restorer. We unify several
datasets as "mixed training set" of image restoration tasks for Restorer
training and validate Restorer performance with official test sets, including
CSD [8], rain1400 [16], OTS [33], SIDD [1], LOL [62], GoPro [40], and
RealBlur-R [50]. See the Appendix A for details on the setting of the dataset.

#### Implementation details.

Restorer employs the PyTorch framework. Restorer is trained for 250 epochs
with a batch size of 26. The rest of the settings followed TransWeather. The
architecture configuration and model complexity for Restorer can be found in
the Appendix B.

###  4.2 Main Results

#### Comparison with unified image restoration methods.

In this section, we compare Restorer with current SOTA unified image
restoration methods [36, 61, 39, 70, 68, 55, 6, 56, 34, 18]. All algorithms
are trained uniformly on mixed training sets according to the official
settings and tested under the official test set for each image restoration
task. We report numerical results for Restorer with above baselines on
multiple image restoration tasks in Table 4. Compared to current SOTA methods,
our method exhibits SOTA or competitive PSNR and SSIM results on multiple
image restoration tasks. The visualization in Figure 6 shows that compared to
other baselines our method can be successfully applied to multiple image
restoration tasks, closer to the ground truth while maintaining clarity.

Figure 7: Qualitative comparison results of Restorer with expert networks on
individual image restoration tasks.

#### Deraining.

We compare Restorer to expert networks designed for deraining tasks. We cite
results in [9] where these networks are trained and tested on rain1400. The
improvements of our Restorer method in image deraining are shown in Table
LABEL:tab:deraining. In Figure 7, both our method and most baselines obtain
high-quality restored images in deraining. Restorer restores more fine detail
of the original image compared to the deraining baselines (see green-boxed
area).

#### Desnowing.

Table LABEL:tab:desnowing shows the numerical comparison results between
Restorer and desnowing expert networks trained on CSD. It can be seen that
Restorer’s PSNR and SSIM on the CSD dataset are 0.87 dB PSNR and 1.71$\%$ SSIM
higher than the previous SOTA method. Additionally, the Restorer with text
prompts has a higher gain of 1.18dB PSNR and 2.09$\%$ SSIM. As shown in Figure
7, Restorer successfully removes snow particles from the input image and does
not suffer from the artifact issue in HDCW-Net [8] compared to the desnowing
baselines.

#### Defogging.

Restorer is compared with defogging networks trained on the OTS dataset. We
mainly cite the results from [9]. Although Restorer is not achieving SOTA
results on SOTS, its 34.19dB PSNR and 98.21$\%$ SSIM metrics in Table
LABEL:tab:defogging are still competitive. Figure 7 shows that Restorer
achieves effective defogging while restoring the global hue of the input
image. Thus, the visual results of Restorer restored images are more
satisfactory.

#### Deblurring.

To validate the performance of Restorer on deblurring, we compare it with
methods trained on GoPro and RealBlur-R, respectively. Compared to these
expert networks, Our SSIM metric on the GoPro is highly competitive. See the
Appendix E for comparison results on the RealBlur-R dataset. Figure 7 shows a
visual comparison of the evaluated models on GoPro. It can be observed that
Restorer shows satisfactory results on the deblurring task.

#### Denoising.

As shown in Table LABEL:tab:denoising, our method achieves the best SSIM
values on the denoising task compared to the expert network trained on SIDD
alone. This indicates that our approach is competitive. Compared to baselines,
Restorer effectively removes real noise and produces sharper restored results
(see Figure 7).

#### Low-light Enhancement.

We report the comparison results of Restorer with the low-light enhancement
expert networks trained on the LOL dataset in Table LABEL:tab:low-
light_enhancement. Our PSNR achieves competitive results and our SSIM is
increased by 4.66$\%$ compared to the previous SOTA method. And Restorer based
on textual prompts even achieves 90.53$\%$ SSIM. In Figure 7, while the
baselines all achieve low-light enhancement well, they suffer from a degree of
overexposure and chromatic aberration (see green-boxed area). In comparison,
Restorer restores images closer to the target.

Method | PSNR |  SSIM ($\%$)
---|---|---
JORDER [64] | 31.28 | 92.00
PReNet [49] | 31.88 | 93.00
DRD-Net [11] | 29.65 | 88.00
MSPFN [26] | 29.24 | 88.00
EfficientDeRain [21] | 32.30 | 92.72
JRGR [65] | 31.18 | 91.00
RCDNet [59] | 33.04 | 94.72
Ours | 33.61 | 96.33
Ours+Prompts | 34.21 | 96.70
(a)

Method | PSNR |  SSIM ($\%$)
---|---|---
EPDN [46] | 23.82 | 87.00
PFDN [13] | 31.45 | 97.00
KDDN [23] | 29.16 | 94.00
MSBDN [12] | 33.79 | 98.00
FFA-Net [45] | 34.98 | 99.00
AECRNet [63] | 35.61 | 98.00
DehazeFormer [52] | 34.29 | 98.30
Ours | 34.19 | 98.21
Ours+Prompts | 34.46 | 98.25
(b)

Method | PSNR |  SSIM ($\%$)
---|---|---
DesnowNet [38] | 20.13 | 81.00
CycleGAN [76] | 20.98 | 80.00
JSTASR [7] | 27.96 | 88.00
DDMSNet [73] | 27.24 | 82.00
HDCW-Net [8] | 29.06 | 91.00
DesnowGAN [25] | 28.63 | 90.00
Ours | 29.93 | 92.71
Ours+Prompts | 30.28 | 93.09
(c)

Method | PSNR |  SSIM ($\%$)
---|---|---
GLADNet [60] | 19.71 | 70.30
EnlightenGAN [27] | 17.48 | 65.70
KinD [75] | 20.37 | 80.40
MIRNet [69] | 24.14 | 83.00
night-enhancement [28] | 21.52 | 76.30
HWMNet [15] | 24.24 | 85.20
Ours | 21.60 | 89.86
Ours+Prompts | 21.44 | 90.53
(d)

Method | PSNR |  SSIM ($\%$)
---|---|---
CBDNet [22] | 30.78 | 80.10
AINDNet [30] | 39.08 | 95.40
VDN [66] | 39.28 | 95.60
SADNet [4] | 39.46 | 95.70
DANet+ [67] | 39.47 | 95.70
DeamNet [48] | 39.47 | 95.70
Ours | 37.74 | 96.36
Ours+Prompts | 37.75 | 96.36
(e)

Method | PSNR |  SSIM ($\%$)
---|---|---
DeblurGAN [31] | 28.70 | 85.80
DeblurGAN-v2 [32] | 29.55 | 93.40
DBGAN [74] | 31.10 | 94.20
MT-RNN [41] | 31.15 | 94.50
DMPHN [71] | 31.20 | 94.00
Stripformer [54] | 33.08 | 96.20
Ours | 30.12 | 94.85
Ours+Prompts | 30.92 | 95.48
(f)

Table 2: Quantitative comparison of Restorer with several different types of
image restoration expert networks. Figure 8: Sample qualitative image
restoration results on the real-world rain, haze, and snow degradations.

###  4.3 Real-world Tests

In this section, we test Restorer based on textual prompts performance on
real-world datasets to verify that it can be applied in real scenarios. Since
Restorer’s performance on image restoration tasks such as denoising,
deblurring, and low-light enhancement are all verified on real-world datasets,
we only need to select real-world image restoration tasks for severe weather
to evaluate. Figure 8 shows the comparison results of our algorithm with
several SOTA algorithms for image restoration on real datasets. Compared to
each baseline, Our method performs more robustly on real-world datasets. We
also provide visual comparison results between Restorer and the baselines on
real-world severe weather image restoration in Appendix F.

| Module | Metric
---|---|---
Setting | AAA | 3D-DCFFN | TQ | PSNR/SSIM
Baseline |  |  |  | 14.18 / 0.705
s1 | ✓ |  |  | 25.44 / 0.902
s2 | ✓ | ✓ |  | 28.26 / 0.919
s3 | ✓ | ✓ | ✓ | 29.93 / 0.927
|  |  |  |

Table 3: Ablation of Core Designs. TQ represents Task Queries.

###  4.4 Ablation Study

We conduct extensive ablation studies to validate the contribution of AAA,
3D-DCFFN, and learnable task queries to Restorer. The U-Net with Restorer’s
four-stage encoder-decoder framework is used as the baseline for our ablation
study. Then we gradually add AAA, 3D-DCFFN, and learnable task queries to the
baseline. Table 6 shows that each design of this work contributes to the
performance of Restorer. See Appendix D for more ablation results.

##  5 Conclusion

In this work, we propose Restorer to handle multiple image restoration tasks.
With this aim, we introduce an all-axis attention module that has
complementary advantages of spatial attention and channel attention.
Furthermore, the proposed 3D-DCFFN preserves the fine details in the image in
both spatial and channel dimensions. Meanwhile, Restorer based on textual
prompts solves the task confusion issue and introduces an interaction element.
Extensive experiments have shown that Restorer has the potential to serve as
the backbone of real-world image restoration applications.

##  Appendix 0.A Datasets

Task | Dataset | #Train | #Test | Test Dubname
---|---|---|---|---
Desnowing |  CSD [8] | 5000 | 2000 | CSD
Deraining |  rain1400 [16] | 5000 | 1400 | rain1400
Dehazing |  OTS [33] | 5000 | 500 | SOTS
Denoising |  SIDD [1] | 5000 | 1280 | SIDD
Debluring |  GoPro [40] | 2103 | 1111 | GoPro
RealBlur-R [50] | 3758 | 980 | RealBlur-R
Enhancement |  LOL [62] | 485 | 15 | LOL
Table 4: Dataset configuration on six image restoration tasks.

To train Restorer which can be applied to multiple image restoration tasks,
our mixed training set is selected from several standard image restoration
datasets including CSD [8], rain1400 [16], OTS [33], SIDD [1], LOL [62], GoPro
[40], and RealBlur-R [50]. For the desnowing task, we randomly selected 5K
images from the CSD training set containing 10K synthetic snow images for the
mixed training set. In the testing phase, we use the official CSD test set for
testing. Regarding the deraining task, we randomly selected 5000 images from
the rain1400 training set containing 12600 synthetic rainfall images to the
mixed training set and validated Restorer with the official rain1400 test set.
For the training of Restorer defogging, the mixed training set has 5000 images
randomly selected from the OTS dataset according to the settings of Chen et
al. [9]. We applied SOTS for the defogging test. Regarding denoising, the
mixed training set consists of 5K images randomly selected from a SIDD
containing 30K real noisy images. For the testing phase, we chose the official
SIDD test set. For the low-light enhancement task, we select the LOL dataset
containing 485 training images and 15 test images for training and testing.
Regarding the deblurring task, the mixed training set includes 2103 GoPro and
3758 RealBlur-R training images. Regarding the deblurring tests, we used the
official test sets of GoPro and RealBlur-R respectively. Detailed dataset
settings are shown in Table 4.

##  Appendix 0.B Complexity

Architecture
---
Stage | Input shape | Output Shape | Layers
Stage1 | $256^{2}\times 3$ | $128^{2}\times 128$ | $\text{N}_{1}=3,\text{p}=16,$
$\text{d}=16,\text{D}=512,\text{H}=2$
Stage2 | $128^{2}\times 128$ | $64^{2}\times 128$ | $\text{N}_{2}=4,\text{p}=8,$
$\text{d}=16,\text{D}=512,\text{H}=2$
Stage3 | $64^{2}\times 128$ | $32^{2}\times 256$ | $\text{N}_{3}=6,\text{p}=4,$
$\text{d}=32,\text{D}=512,\text{H}=4$
Stage4 | $32^{2}\times 256$ | $16^{2}\times 512$ | $\text{N}_{4}=3,\text{p}=4,$
$\text{d}=16,\text{D}=512,\text{H}=8$
Table 5: Detailed architectural configurations for the different stages of Restorer. Task | Dataset | Model | PSNR | FLOPs
---|---|---|---|---
Denoise | SIDD [1] |  Restormer [68] | 25.30 | 141G
MAXIM [55] | 27.70 | 216G
Ours | 37.74 | 147G
Ours+Prompts | 37.75 | 147G
Deblur | GoPro [40] |  Restormer [68] | 26.22 | 141G
MAXIM [55] | 25.78 | 216G
Ours | 30.12 | 147G
Ours+Prompts | 30.92 | 147G
Deblur | RealBlur-R [50] |  Restormer [68] | 34.42 | 141G
MAXIM [55] | 32.52 | 216G
Ours | 41.68 | 147G
Ours+Prompts | 43.10 | 147G
Derain | rain1400 [16] |  Restormer [68] | 24.38 | 141G
MAXIM [55] | 32.63 | 216G
Ours | 33.61 | 147G
Ours+Prompts | 34.21 | 147G
Dehaze | SOTS |  Restormer [68] | 15.94 | 141G
MAXIM [55] | 32.20 | 216G
Ours | 34.19 | 147G
Ours+Prompts | 34.46 | 147G
Enhance | LOL [62] |  Restormer [68] | 7.79 | 141G
MAXIM [55] | 13.48 | 216G
Ours | 21.60 | 147G
Ours+Prompts | 21.44 | 147G
Table 6: Model performance _vs_. complexity comparison of our model with SOTA
baselines.

The detailed specifications of the Restorer architecture configuration are
shown in Table 5. We have provided the input shapes and output shapes for each
stage of the Restorer encoder for better understanding. We reduce the
computational overhead of AAA module by adjusting the spatial dimension p and
the channel dimension d of the stereo token embedding at each stage. Also for
this purpose, we unify the embedding dimension D of each stage to 512
dimensions. See Table 6 for a comparison of performance and complexity between
our method and competing methods. Moreover, we show the complexity of each
component in Restorer in Table 7. It can be noticed that our AAA module and
3D-DCFFN do not lead to a huge amount of computation.

| Module |
---|---|---
Setting | AAA | 3D-DCFFN | TQ | FLOPs
Baseline |  |  |  | 147G
s1 |  | ✓ | ✓ | 121G
s2 | ✓ |  | ✓ | 144G
s3 | ✓ | ✓ |  | 147G
|  |  |  |

Table 7: Ablation of Core Designs. TQ represents Task Queries.

##  Appendix 0.C Experimental Details

Restorer mainly follows the [56] settings. Apart from the training settings
described in the main text, Restorer uses 256-resolution images as input and
performs normalization operations on the input data. In addition, for training
after 60 epochs, the learning rate decays by half every 50 epochs. We
implemented Restorer training with two NVIDIA RTX 3090.

##  Appendix 0.D Ablation Study

#### Effectiveness of AAA.

To further validate the effectiveness of AAA, we also test the performance of
Restorer paired with different attention mechanisms. Table LABEL:tab:ablation3
shows that AAA is more suitable for Restorer to handle multiple image
restoration tasks at the same time than other attention mechanisms.

#### Effectiveness of 3D-DCFFN.

We compare the results of different feedforward networks on Restorer. Table
LABEL:tab:ablation2 shows that our 3D-DCFFN better improves the quality of
restored images compared to other feedforward network backbones. Therefore,
3D-DCFFN is the key for Restorer to get high-quality restored images.

#### AAA Connection.

We removed the connection between the AAA modules. As shown in Table
LABEL:tab:ablation4, after removing the connection between the AAA modules,
Restorer’s effectiveness on the desnowing task decreases to some extent. This
suggests that the connections between the AAA modules we designed to enhance
Restorer’s multiscale play a role.

#### Encoder Module and Decoder Module Structure.

Regarding the ablation design of the structure of Restorer’s encoder and
decoder modules, we choose Restormer’s [68] Transformer block to replace them
respectively. In Table LABEL:tab:ablation4, each of these ablation designs
resulted in a decrease in the Restorer metric. One possible reason for this is
that convolution inherently has local modeling advantages that Transformer
does not have. This facilitates Restorer’s restored image to retain fine
details in the original image.

|  Metric
---|---
Method | PSNR | SSIM
w/o AAA connection | 28.75 | 0.924
w/ Restormer encoder module | 28.51 | 0.923
w/ Restormer decoder module | 28.26 | 0.920
|  |

(a)

|  Metric
---|---
Method | PSNR | SSIM
w/ vanilla affinity matrices | 29.47 | 0.922
w/ projection affinity matrices | 29.19 | 0.924
w/ negative affinity matrices | 29.93 | 0.927
|  |

(b)

| Metric
---|---
Method | PSNR | SSIM ($\%$)
w/ vanilla FFN | 28.14 | 92.07
w/ DCFFN | 28.65 | 92.38
w/ 3D-DCFFN | 29.93 | 92.71
|  |

(c)

| Metric
---|---
Method | PSNR | SSIM ($\%$)
w/ spatial attention | 25.61 | 90.33
w/ channel attention | 24.85 | 89.72
w/ OSA | 27.92 | 92.26
w/ AAA | 29.93 | 92.71
|  |

(d)

Table 8: Ablation results for each component of Restorer. We report ablation
results using desnowing experiments on CSD.

#### Negative Affinity Matrices.

We test the effectiveness of negative affinity matrices coupled with learnable
task queries in all-axis attention in this section. We compare the negative
affinity matrices with two variants, the vanilla affinity matrices, and the
projection affinity matrices. With Table LABEL:tab:ablation1 we observe that
the negative affinity matrices are more favorable to Restorer. We believe this
is because the negative affinity matrices explicitly model the relationship
between the learnable task queries and the image restoration tasks. Compared
to other types of affinity matrices, this explicit modeling of negative
affinity matrices provides an explicit goal for learnable task queries, i.e.,
fitting the degradation features of the corresponding image restoration task.
As a result, Restorer with negative affinity matrices can achieve more
accurate image restoration.

##  Appendix 0.E More Experiments

In this section, we provide quantitative comparisons and qualitative
comparison results between Restorer and the deblurring expert networks trained
and tested on RealBlur-R alone. We choose multiple current SOTA deblurring
baselines for comparison. Figure 9 and Table 9 demonstrate the results of
qualitative and quantitative comparisons, respectively.

|  | Metric
---|---|---
Method | Venue | PSNR | SSIM
DeblurGAN  [31] | CVPR | 33.79 | 0.9030
DeblurGAN-v2  [32] | ICCV | 35.26 | 0.9440
SRN  [53] | CVPR | 35.66 | 0.9470
DBGAN  [74] | CVPR | 33.78 | 0.9090
MT-RNN  [41] | ECCV | 35.79 | 0.9510
DMPHN  [71] | ECCV | 35.70 | 0.9480
MIMO-UNet+  [10] | ICCV | 35.54 | 0.9470
Stripformer  [54] | ECCV | 39.84 | 0.9740
MSSNet  [29] | ICCV | 39.76 | 0.9720
Ours | - | 41.68 | 0.9520
Ours+Prompts | - | 43.10 | 0.9755
|  |  |

Table 9: Quantitative comparison results of Restorer with multiple deblurring
baselines on RealBlur-R. Figure 9: Visual comparison results between Restorer
and SOTA deblurring baselines on RealBlur-R. Figure 10: Restorer’s image
restoration results at different textual prompts on RealBlur-R. Figure 11:
Qualitative comparison results between Restorer and baselines in the real
world. Figure 12: Qualitative comparison results between Restorer and SOTA
unified image restoration algorithms and all-in-one image restoration
algorithms.

#### Quantitative Results.

As shown in Table 9, our Restorer achieves state-of-the-art performance for
the deblurring task on RealBlur-R. Compared to the current SOTA deblurring
baselines, Restorer offers a large improvement over PSNR. Specifically,
Restorer improved by 1.92 dB compared to MSSNet [29] and 1.84 dB compared to
Stripformer [54]. Meanwhile, Restorer based on textual prompts achieves state-
of-the-art performance as evaluated by PSNR and SSIM metrics.

#### Qualitative Results.

Figure 9 shows the qualitative comparison results between Restorer and the
current SOTA deblurring baselines on RealBlur-R. It can be observed that our
model restores the blurred text well. This may be attributed to the fact that
the use of the AAA module at each stage allows the module to globally
aggregate repeating patterns at different scales.

##  Appendix 0.F Real-world Comparisons

In this section, we show the visual comparison results between the Restorer
and baselines under different degradation conditions in the real world. Figure
11 shows that Restorer better removes degradation from the image and produces
sharper results than many excellent baselines. This shows that Restorer is
robust enough to degradations in real scenarios.

##  Appendix 0.G Restorer Based on Textual Prompts

In this section, we provide more visual results of Restorer employing
different textual prompts for the degraded image. We also demonstrate the
effectiveness of Restorer’s restoration of complex composite degraded images
by making it stack several textual prompts.

As shown in Figure 10, Restorer successfully performs different image
restoration tasks on the RealBlur-R dataset following different textual
prompts. Meanwhile, by stacking textual prompts, Restorer successfully
restores input images with complex composite degradation.

##  Appendix 0.H More Visual Results

#### Comparison with Unified Image Restoration Algorithms.

We show the visual comparison results of Restorer with several SOTA unified
image restoration algorithms and all-in-one image restoration algorithms in
Figure 12. Compared to these excellent baselines, our method successfully
removed various types of degradation and obtained pleasing visual results.

#### Desnowing.

As shown in Figure 13, our method successfully removes snow particles of
different sizes from the input image without artifacts and chromatic
aberration problems compared to multiple SOTA desnowing expert networks that
were individually trained and tested on CSD dataset.

In Figure 14, most of the baselines fail to remove the snow particles from the
degraded image and cause severe impairments to the restored image. In
contrast, with the text "snow" prompt, Restorer successfully removed the snow
particles in the image and obtained a pleasant restored result.

#### Deraining.

Figure 15 shows the visual comparison results between Restorer and the current
state-of-the-art deraining expert networks on rain1400. It can be observed
that our method effectively removes the rain from the input image and
successfully obtains a high-quality restored image.

We compare the performance of Restorer based on textual prompts and expert
networks on deraining in Figure 16. Compared to these excellent baselines,
Restorer based on textual prompts successfully removed rain and had no
problems with artifacts.

Figure 13: Visual comparison results between Restorer and SOTA desnowing
baselines on CSD dataset. Figure 14: Visual comparison results between
Restorer based on textual prompts and SOTA desnowing baselines on CSD dataset.

#### Defogging.

We provide a defogging visual comparison in Figure 17. Restorer restored more
satisfactory results in the defogging task than other methods.

On the qualitative comparison of defogging in Fig 18, although all the
comparison algorithms successfully remove the degradation, EPDN [46], AECR-Net
[63], and FFA-Net [45] produce severe chromatic aberration affecting the
quality of the restored images. On the other hand, DehazeFormer [52] and MSBDN
[12] have some fog residuals in the house area. Compared with them, Restorer
based on textual prompts has a more obvious defogging effect and more
satisfactory image restoration results.

#### Deblurring.

The visualizations on GoPro [40] and RealBlur-R [50] are shown in Figure 19
and Figure 21, respectively, and our model achieves competitive results on
both real-world deblurring benchmarks.

Furthermore, Restorer based on textual prompts also successfully removes
degradation on GoPro and RealBlur-R in Figure 20 and Figure 22. This proves
that both Restorer and Restorer based on textual prompts are robust enough for
blurring.

#### Denoising.

The qualitative comparison results of the denoising task for the SIDD dataset
are shown in Figure 23. Both our method and the comparison baselines
successfully remove the noise from the input image. But Restorer provides
clearer restored results than many baselines.

In the denoising results in Figure 24, Restorer based on textual prompts
successfully removes the noise degradation and restores fine details in the
image. We attribute this to the success of convolution in local modeling in
3D-DCFFN, encoder, and decoder structures in Restorer.

#### Low-light Enhancement.

We show qualitative comparison results of Restorer with current state-of-the-
art low-light enhancement algorithms in Figure 25. It can be observed that our
method successfully achieves low-light enhancement and obtains pleasing visual
effects that are closer to the ground truth.

We can also observe in Figure 26 that Restorer based on low-light textual
prompts achieves a more obvious light enhancement effect.

Figure 15: Visual comparison results between Restorer and SOTA deraining
baselines on rain1400. Figure 16: Visual comparison results between Restorer
based on textual prompts and SOTA deraining baselines on rain1400. Figure 17:
Visual comparison results between Restorer and SOTA defogging baselines on
SOTS. Figure 18: Visual comparison results between Restorer based on textual
prompts and SOTA defogging baselines on SOTS. Figure 19: Visual comparison
results between Restorer and SOTA deblurring baselines on GoPro. Figure 20:
Visual comparison results between Restorer based on textual prompts and SOTA
deblurring baselines on GoPro. Figure 21: Visual comparison results between
Restorer and SOTA deblurring baselines on RealBlur-R. Figure 22: Visual
comparison results between Restorer based on textual prompts and deblurring
baselines on RealBlur-R. Figure 23: Visual comparison results between Restorer
and SOTA denoising baselines on SSIM. Figure 24: Visual comparison results
between Restorer based on textual prompts and SOTA denoising baselines on
SSIM. Figure 25: Visual comparison results between Restorer and SOTA low-light
enhancement baselines on LOL dataset. Figure 26: Visual comparison between
Restorer based on textual prompts and low-light enhancement baselines on LOL.

## References

  * [1] Abdelhamed, A., Lin, S., Brown, M.S.: A high-quality denoising dataset for smartphone cameras. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1692–1700 (2018)
  * [2] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
  * [3] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020)
  * [4] Chang, M., Li, Q., Feng, H., Xu, Z.: Spatial-adaptive network for single image denoising. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16. pp. 171–187. Springer (2020)
  * [5] Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., Gao, W.: Pre-trained image processing transformer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12299–12310 (2021)
  * [6] Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. In: European Conference on Computer Vision. pp. 17–33. Springer (2022)
  * [7] Chen, W.T., Fang, H.Y., Ding, J.J., Tsai, C.C., Kuo, S.Y.: Jstasr: Joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal. In: European Conference on Computer Vision. pp. 754–770. Springer (2020)
  * [8] Chen, W.T., Fang, H.Y., Hsieh, C.L., Tsai, C.C., Chen, I., Ding, J.J., Kuo, S.Y., et al.: All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4196–4205 (2021)
  * [9] Chen, W.T., Huang, Z.K., Tsai, C.C., Yang, H.H., Ding, J.J., Kuo, S.Y.: Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: Toward a unified model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17653–17662 (2022)
  * [10] Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4641–4650 (2021)
  * [11] Deng, S., Wei, M., Wang, J., Feng, Y., Liang, L., Xie, H., Wang, F.L., Wang, M.: Detail-recovery image deraining via context aggregation networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14560–14569 (2020)
  * [12] Dong, H., Pan, J., Xiang, L., Hu, Z., Zhang, X., Wang, F., Yang, M.H.: Multi-scale boosted dehazing network with dense feature fusion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2157–2167 (2020)
  * [13] Dong, J., Pan, J.: Physics-based feature dehazing networks. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16. pp. 188–204. Springer (2020)
  * [14] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [15] Fan, C.M., Liu, T.J., Liu, K.H.: Half wavelet attention on m-net+ for low-light image enhancement. In: 2022 IEEE International Conference on Image Processing (ICIP). pp. 3878–3882. IEEE (2022)
  * [16] Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3855–3863 (2017)
  * [17] Fu, X., Qi, Q., Zha, Z.J., Zhu, Y., Ding, X.: Rain streak removal via dual graph convolutional network. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 1352–1360 (2021)
  * [18] Gou, W., Yi, Z., Xiang, Y., Li, S., Liu, Z., Kong, D., Xu, K.: Syenet: A simple yet effective network for multiple low-level vision tasks with real-time performance on mobile device. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12182–12195 (2023)
  * [19] Grainger, R., Paniagua, T., Song, X., Cuntoor, N., Lee, M.W., Wu, T.: Paca-vit: Learning patch-to-cluster attention in vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18568–18578 (2023)
  * [20] Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., Xu, C.: Cmt: Convolutional neural networks meet vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12175–12185 (2022)
  * [21] Guo, Q., Sun, J., Juefei-Xu, F., Ma, L., Xie, X., Feng, W., Liu, Y., Zhao, J.: Efficientderain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 1487–1495 (2021)
  * [22] Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1712–1722 (2019)
  * [23] Hong, M., Xie, Y., Li, C., Qu, Y.: Distilling image dehazing with heterogeneous task imitation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3462–3471 (2020)
  * [24] Huang, H., Zhou, X., Cao, J., He, R., Tan, T.: Vision transformer with super token sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22690–22699 (2023)
  * [25] Jaw, D.W., Huang, S.C., Kuo, S.Y.: Desnowgan: An efficient single image snow removal framework using cross-resolution lateral connection and gans. IEEE Transactions on Circuits and Systems for Video Technology 31(4), 1342–1350 (2020)
  * [26] Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., Jiang, J.: Multi-scale progressive fusion network for single image deraining. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8346–8355 (2020)
  * [27] Jiang, Y., Gong, X., Liu, D., Cheng, Y., Fang, C., Shen, X., Yang, J., Zhou, P., Wang, Z.: Enlightengan: Deep light enhancement without paired supervision. IEEE transactions on image processing 30, 2340–2349 (2021)
  * [28] Jin, Y., Yang, W., Tan, R.T.: Unsupervised night image enhancement: When layer decomposition meets light-effects suppression. In: European Conference on Computer Vision. pp. 404–421. Springer (2022)
  * [29] Kim, K., Lee, S., Cho, S.: Mssnet: Multi-scale-stage network for single image deblurring. In: European Conference on Computer Vision. pp. 524–539. Springer (2022)
  * [30] Kim, Y., Soh, J.W., Park, G.Y., Cho, N.I.: Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3482–3492 (2020)
  * [31] Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., Matas, J.: Deblurgan: Blind motion deblurring using conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8183–8192 (2018)
  * [32] Kupyn, O., Martyniuk, T., Wu, J., Wang, Z.: Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8878–8887 (2019)
  * [33] Li, B., Ren, W., Fu, D., Tao, D., Feng, D., Zeng, W., Wang, Z.: Benchmarking single-image dehazing and beyond. IEEE Transactions on Image Processing 28(1), 492–505 (2018)
  * [34] Li, B., Liu, X., Hu, P., Wu, Z., Lv, J., Peng, X.: All-in-one image restoration for unknown corruption. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17452–17462 (2022)
  * [35] Li, R., Tan, R.T., Cheong, L.F.: All in one bad weather removal using architectural search. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3175–3185 (2020)
  * [36] Li, Y., Fan, Y., Xiang, X., Demandolx, D., Ranjan, R., Timofte, R., Van Gool, L.: Efficient and explicit modelling of image hierarchies for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18278–18289 (2023)
  * [37] Liang, M., Yang, B., Wang, S., Urtasun, R.: Deep continuous fusion for multi-sensor 3d object detection. In: Proceedings of the European conference on computer vision (ECCV). pp. 641–656 (2018)
  * [38] Liu, Y.F., Jaw, D.W., Huang, S.C., Hwang, J.N.: Desnownet: Context-aware deep network for snow removal. IEEE Transactions on Image Processing 27(6), 3064–3073 (2018)
  * [39] Mou, C., Wang, Q., Zhang, J.: Deep generalized unfolding networks for image restoration. In: CVPR (2022)
  * [40] Nah, S., Son, S., Lee, S., Timofte, R., Lee, K.M.: Ntire 2021 challenge on image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 149–165 (2021)
  * [41] Park, D., Kang, D.U., Kim, J., Chun, S.Y.: Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In: European Conference on Computer Vision. pp. 327–343. Springer (2020)
  * [42] Perera, A.G., Wei Law, Y., Chahl, J.: Uav-gesture: A dataset for uav control and gesture recognition. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pp. 0–0 (2018)
  * [43] Prakash, A., Chitta, K., Geiger, A.: Multi-modal fusion transformer for end-to-end autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7077–7087 (2021)
  * [44] Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.: Frustum pointnets for 3d object detection from rgb-d data. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 918–927 (2018)
  * [45] Qin, X., Wang, Z., Bai, Y., Xie, X., Jia, H.: Ffa-net: Feature fusion attention network for single image dehazing. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 11908–11915 (2020)
  * [46] Qu, Y., Chen, Y., Huang, J., Xie, Y.: Enhanced pix2pix dehazing network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8160–8168 (2019)
  * [47] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [48] Ren, C., He, X., Wang, C., Zhao, Z.: Adaptive consistency prior based deep network for image denoising. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8596–8606 (2021)
  * [49] Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: A better and simpler baseline. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3937–3946 (2019)
  * [50] Rim, J., Lee, H., Won, J., Cho, S.: Real-world blur dataset for learning and benchmarking deblurring algorithms. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16\. pp. 184–201. Springer (2020)
  * [51] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. pp. 234–241. Springer (2015)
  * [52] Song, Y., He, Z., Qian, H., Du, X.: Vision transformers for single image dehazing. IEEE Transactions on Image Processing 32, 1927–1941 (2023)
  * [53] Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
  * [54] Tsai, F.J., Peng, Y.T., Lin, Y.Y., Tsai, C.C., Lin, C.W.: Stripformer: Strip transformer for fast image deblurring. In: European Conference on Computer Vision. pp. 146–162. Springer (2022)
  * [55] Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., Li, Y.: Maxim: Multi-axis mlp for image processing. CVPR (2022)
  * [56] Valanarasu, J.M.J., Yasarla, R., Patel, V.M.: Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2353–2363 (2022)
  * [57] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [58] Wang, H., Chen, X., Ni, B., Liu, Y., Liu, J.: Omni aggregation networks for lightweight image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22378–22387 (2023)
  * [59] Wang, H., Xie, Q., Zhao, Q., Meng, D.: A model-driven deep neural network for single image rain removal. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3103–3112 (2020)
  * [60] Wang, W., Wei, C., Yang, W., Liu, J.: Gladnet: Low-light enhancement network with global awareness. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). pp. 751–755. IEEE (2018)
  * [61] Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H.: Uformer: A general u-shaped transformer for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 17683–17693 (June 2022)
  * [62] Wei, C., Wang, W., Yang, W., Liu, J.: Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560 (2018)
  * [63] Wu, H., Qu, Y., Lin, S., Zhou, J., Qiao, R., Zhang, Z., Xie, Y., Ma, L.: Contrastive learning for compact single image dehazing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10551–10560 (2021)
  * [64] Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1357–1366 (2017)
  * [65] Ye, Y., Chang, Y., Zhou, H., Yan, L.: Closing the loop: Joint rain generation and removal via disentangled image translation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2053–2062 (2021)
  * [66] Yue, Z., Yong, H., Zhao, Q., Meng, D., Zhang, L.: Variational denoising network: Toward blind noise modeling and removal. Advances in neural information processing systems 32 (2019)
  * [67] Yue, Z., Zhao, Q., Zhang, L., Meng, D.: Dual adversarial network: Toward real-world noise removal and noise generation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16. pp. 41–58. Springer (2020)
  * [68] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5728–5739 (2022)
  * [69] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Learning enriched features for real image restoration and enhancement. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16. pp. 492–511. Springer (2020)
  * [70] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Multi-stage progressive image restoration. In: CVPR (2021)
  * [71] Zhang, H., Dai, Y., Li, H., Koniusz, P.: Deep stacked hierarchical multi-patch network for image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5978–5986 (2019)
  * [72] Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing 26(7), 3142–3155 (2017)
  * [73] Zhang, K., Li, R., Yu, Y., Luo, W., Li, C.: Deep dense multi-scale network for snow removal using semantic and depth priors. IEEE Transactions on Image Processing 30, 7419–7431 (2021)
  * [74] Zhang, K., Luo, W., Zhong, Y., Ma, L., Stenger, B., Liu, W., Li, H.: Deblurring by realistic blurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2737–2746 (2020)
  * [75] Zhang, Y., Zhang, J., Guo, X.: Kindling the darkness: A practical low-light image enhancer. In: Proceedings of the 27th ACM international conference on multimedia. pp. 1632–1640 (2019)
  * [76] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)