# Towards Vision Transformer Unrolling Fixed-Point Algorithm: a Case Study on
Image Restoration

Peng Qiao, Sidun Liu, Tao Sun, Ke Yang, Yong Dou P. Qiao, S. Liu, T. Sun, Y.
Dou are with the Science and Technology on Parallel and Distributed
Laboratory, School of Computer, National University of Defense Technology,
Changsha, China. e-mail: (pengqiao@nudt.edu.cn). K. Yang is with National
Innovation Institute of Defense Technology, Beijing, China.

###### Abstract

The great success of Deep Neural Networks (DNNs) has inspired the algorithmic
development of DNN-based Fixed-Point (DNN-FP) for computer vision tasks. DNN-
FP methods, trained by Back-Propagation Through Time or computing the
inaccurate inversion of the Jacobian, suffer from inferior representation
ability. Motivated by the representation power of the Transformer, we propose
a framework to unroll the FP and approximate each unrolled process via
Transformer blocks, called FPformer. To reduce the high consumption of memory
and computation, we come up with FPRformer by sharing parameters between the
successive blocks. We further design a module to adapt Anderson acceleration
to FPRformer to enlarge the unrolled iterations and improve the performance,
called FPAformer. In order to fully exploit the capability of the Transformer,
we apply the proposed model to image restoration, using self-supervised pre-
training and supervised fine-tuning. 161 tasks from 4 categories of image
restoration problems are used in the pre-training phase. Hereafter, the pre-
trained FPformer, FPRformer, and FPAformer are further fine-tuned for the
comparison scenarios. Using self-supervised pre-training and supervised fine-
tuning, the proposed FPformer, FPRformer, and FPAformer achieve competitive
performance with state-of-the-art image restoration methods and better
training efficiency. FPAformer employs only 29.82% parameters used in SwinIR
models, and provides superior performance after fine-tuning. To train these
comparison models, it takes only 26.9% time used for training SwinIR models.
It provides a promising way to introduce the Transformer in low-level vision
tasks.

###### Index Terms:

Image restoration pre-training, Vision Transformer, fixed-point, unrolling.

Figure 1: The number of parameters vs SRx4 performance in terms of PSNR on
Set5. The size of markers is proportional to the number of parameters in each
method. Our proposed methods with † are finetuned for SRx4, while ones with ∗
are pretrained.

##  I Introduction

The popular, exuberant, and efficient Deep Neural Networks (DNNs) techniques
provide a DNN-based routine for the Fixed-Point (FP) method to handle
optimization problems in computer vision [1, 2, 3], decision-making [4] and
other domains, achieve adorable performance due to existing physical
advantages (like GPU computing) or human experiences (like tuning or networks
structure settings).

Conventional DNN-FP methods that directly unroll the FP via Convolutional
Neural Networks (CNNs) or Recurrent Neural Networks (RNNs) are trained by
Back-Propagation (BP) [5, 6] or Back-Propagation Through Time (BPTT) [7, 8].
These methods have limited representation ability [5, 9, 6] and suffer from
high memory consumption [10], and the gradient vanishing/exploding issues [11,
12, 13, 14] as the number of unrolled depth increases. While Deep EQuilibrium
(DEQ) [15, 16] methods unroll FP with implicit depth and seek the equilibrium
point of FP, they are trained by computing the inversion of the Jacobian of
loss w.r.t. the equilibrium point. To alleviate the heavy computation burden
of inversion of the Jacobian, an inexact gradient is proposed. However, it
yields an undesirable solution.

With the great success of Transformer based models in Natural Language
Processing (NLP) [17, 18, 19, 20, 21, 22] and Computer Vision (CV) [23, 24,
25, 26, 27, 28, 29, 30], it has been shown that Transformer-based models are
suitable for model the sequential relation with a powerful representation.
Motivated by this fact, we propose to unroll the FP and approximate each
unrolled process via Transformer called FPformer. Nevertheless, Transformer-
based methods increase the consumption of memory and computation.

To handle this issue, we investigate the parameter sharing [18] in FPformer,
called FPRformer. In this setting, the successive blocks in Transformers share
the parameters, resulting in fewer trainable parameters and maintaining the
unrolled iteration times.

Based on our analysis in Section III-C, we further apply Anderson acceleration
[31, 32] to FPRformer via a simplified ConvGRU [13, 33] module to enlarge the
unrolled iterations, called FPAformer.

To verify the effectiveness of the proposed FPformer and its variants, we
apply it to image restoration task sets as a general image restoration
framework. In order to fully exploit the capability of the Transformer, we
train the FPformer, FPRformer, and FPAformer using self-supervised pre-
training and supervised fine-tuning, widely used in NLP and high-level vision
tasks. In the self-supervised pre-training, fixed-point finding for solving
image restoration problems becomes a natural interpretation for general image
restoration, serving as the self-supervised pre-training problem. We use 161
tasks from 4 categories of image restoration problems to pre-train the
proposed FPformer, FPRformer, and FPAformer. Namely, the image restoration
tasks are Gaussian denoising in grayscale and color space (noise levels
ranging from 0 to 75), single image super-resolution (scale factors are 2, 3,
4, and 8), and JPEG deblocking (quality factors are 10, 20, 30, 40 and 50).
During the supervised fine-tuning, the pre-trained FPformer, FPRformer, and
FPAformer are further fine-tuned for a specific comparison scenario, e.g.,
Gaussian denoising in color space with noise level $\sigma=25$. Using self-
supervised pre-training and supervised fine-tuning, the proposed FPformer,
FPRformer, and FPAformer achieve competitive performance with state-of-the-art
image restoration methods and better efficiency, as shown in Figure 1,
providing a promising way to introduce Transformer in low-level vision tasks.

Figure 2: The architecture of the proposed FPformer, FPRformer, and
FPAformer.CDN and GDN mean Color Gaussian Denoising and Grayscale Gaussian
Denoising, respectively. JPEG stands for image JPEG deblocking. SISR means
Single Image Super-Resolution.

##  II Related works

###  II-A Fixed-Point via DNNs

The fixed-point is formulated as

$z^{*}=\mathcal{F}(z^{*}).$ (1)

The fixed-point finding in Algorithm 1, details in supplementary materials,
generates a series of $\\{{z_{t}}\\}_{t=1}^{T}$ by successively applying the
contraction mapping $\mathcal{F}(.)$, given a initial point $z_{0}$. When we
focus on the states $z_{t}$, we can simplify the Algorithm 1 as

$z_{0}\stackrel{{\scriptstyle\mathcal{F}(.)}}{{\longrightarrow}}z_{1}\longrightarrow\cdots\longrightarrow
z_{T-1}\stackrel{{\scriptstyle\mathcal{F}(.)}}{{\longrightarrow}}z_{T}.$ (2)

Conventional DNN-FP methods that directly unroll the FP via Convolutional
Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), i.e.,
parameterizing $\mathcal{F}(.)$ with $\mathcal{F}_{\theta}$. These methods
either have limited representation ability [5, 9, 6] or suffer from high
memory consumption [10] and the gradient vanishing/exploding issues [11, 12,
13, 14] as the number of unrolled iterations increases. While Deep EQuilibrium
(DEQ) [15] methods unroll the FP with implicit depth and seek the equilibrium
point of the FP, they are trained by computing the inversion of the Jacobian
of loss w.r.t. the equilibrium point. DEQ and its variants [3] suffer from
high computation and inexact gradient to achieve an unacceptable solution. In
[34], DEQ is applied in solving inverse problems in imaging, where
$\mathcal{F}(.)$ is a specific proximal operator and is further parameterized
via $\theta$.

In the fixed-point finding method, the iteration process in Algorithm 1
requires quite a lot of iterations, i.e., a large $T$, to reach a feasible
equilibrium point $z^{*}$. One should also note that the choice of hyper-
parameters of $\epsilon$ and $T$ is vitally important to achieving a good
performance using $\mathcal{F}_{\theta}$, whose contraction property is not
well-guaranteed. When applying DNN-FP methods, repeating the modern DNN a
couple of times is computation-consuming and time-consuming. For example, DEQ
and its variants still consume GPU memory as large as modern DNN, and even
more computational consumption to perform the fixed-point finding algorithm to
achieve a reasonable performance.

Anderson acceleration ([31, 32], AA) is proposed to accelerate the fixed-point
finding utilizing the previous $m$ states $\\{z_{t-m+i}\\}_{i=1}^{m}$ to
estimate the next state $z_{t+1}$, as shown in Algorithm 2 in supplementary
materials. In [35], Anderson acceleration is integrated into DEQ and
parameterized via NNs. In order to explicitly combine the preview $m$ states,
the proposed AA module exploits a bottle-neck-like architecture to produces
real value weights. Thus, the preview $m$ states are needed to be buffered. To
save the storage costs, in [35], the bottle-neck-like network that benefits
NLP but degrades image processing because images are represented in 2d-data
and have vivid context information in spatial domains while natural languages
only need 1d-data.

###  II-B Image Restoration

Conventional image restoration methods can be used to recover various image
degradation problems by minimizing the following energy function,

$\mathcal{E}(u,f)=\mathcal{D}(u,f)+\lambda\mathcal{R}(u),$ (3)

where $\mathcal{D}(u,f)$ is the data term related to one specific image
restoration problem, $f$ is the degraded input image, and $u$ is the restored
image. Taking Gaussian denoising as an example,
$\mathcal{D}(u,f)=\frac{1}{2{\sigma}^{2}}\|u-f\|^{2}$, where $\sigma$ is the
noise level for a specific Gaussian denoising problem. $\mathcal{R}(u)$ is the
regularization term known as the image prior model [36, 37, 38, 39, 40, 5].
Empirically, one can get a minimizer of Equation 3 via gradient descent. It
can be reformulated into a fixed-point finding diagram when
$\mathcal{I}-\nabla\mathcal{E}$ is a contraction mapping.

Benefiting from machine learning methods, the above hand-crafted methods can
be further boosted. For example, the diffusion models in [5, 9] are the
learned counterparts of their conventional ones. With the rapid development of
CNNs and Transformers, the restoration methods are learned in a data-driven
manner and provide very impressive performance in image denoising [41, 6],
single image super-resolution [42, 43], JPEG deblocking [44, 41, 45], image
deblurring [46, 47], et. al.

Most of CNNs based image restoration methods can be regarded as learning the
mapping between the degraded input image $f$ and its corresponding ground-
truth image $u_{gt}$. Therefore, we can summarize that these CNNs based
methods are parameterized in the very first iteration in Equation 2, i.e.,
$z_{1}=\mathcal{F}_{\theta}(z_{0})$. $z_{0}$ is mapped from the degraded image
$f$ via $S_{\phi}$. $z_{T}\big{|}_{T=1}$ is mapped to restored image via
$U_{\psi}$. Note that, both $S_{\phi}$ and $U_{\psi}$ are identical mapping in
[6], and convolution operators in [27]. Due to large parameters and low
training efficiency, approximating each step $z_{t}$ using CNNs is not a
common practice. On the contrary, TNRD [5] and its variants[9] are trained to
approximate the unrolling of Equation 2 in fixed iterations. In each
iteration, the contraction mapping $\mathcal{F}_{\theta_{t}}$ is a learned
diffusion model parameterized by $\theta_{t}$. However, the representation
ability of TNRD is quite limited. So, we need an effective way to approximate
each step $z_{t}$ in Algorithm 1.

###  II-C Vision Transformer

Multi-Head Self Attention (MHSA) introduced in [17] has been widely used in
natural language processing [19, 20, 21, 22]. In [23], MHSA was adopted in
computer vision, resulting in vision transformers. And since then, vision
transformers [23, 24, 26, 25] trained by self-supervised pre-training and
supervised fine-tuning have achieved better performance than CNN models in
high-level vision.

In low-level vision represented by image restoration problems, vision
transformer architecture is adapted [27, 28, 29, 30, 48] and also benefits the
performance compared with the CNN counterparts [6, 41]. These works trained
transformer models via supervised training for one specific task and even one
specific task setting. In [27, 30], the proposed methods were trained for
Gaussian denoising with different noise levels, for single image super-
resolution with different scales, respectively. In [28], IPT was trained for a
couple of image restoration tasks with an auxiliary token, i.e., task-specific
token. In [30], Restormer is proposed and provides a way to efficiently train
for high-resolution image restoration problems. Therefore, the self-supervised
pre-training for image restoration is worth discussing to achieve an efficient
way to utilize a vision transformer.

##  III Methodology

In this section, we first describe the proposed FPformer approximating the
unrolling of the FP. To reduce the memory consumption, we utilize parameter
sharing between the successive blocks in FPformer, called FPRformer. To boost
the restoration performance, we design a module inspired by the Anderson
acceleration algorithm, called FPAformer.

###  III-A Unrolling FP with Transformers: FPformer

As described above, DEQ and its variants benefit from the implicitly infinite
iterations to conduct fixed-point finding with a large $T$ and small
$\epsilon$; they need a huge computational consumption to reach the FP when
$\mathcal{F}$ is approximated via modern DNNs. Directly learning the unrolling
of the fixed-point finding in fixed iterations, e.g., DnCNN with $T=1$ and
TNRD with a larger $T$ less than 10, it is limited by the unrolled times $T$
or the representation ability $\mathcal{F}_{\theta_{t}}$. We need an effective
way to enlarge the unrolled times and strengthen the representation ability.

Having witnessed the success of NLP and CV, the Transformer-based models are
suitable for modeling the sequential relation with a powerful representation.
Motivated by this fact, we propose to unroll the FP and approximate each
unrolled process via Transformer called FPformer. To this end, we resort to
Transformers. To efficiently capture global information, we use Residual Swin
Transformer Block (RSTB) blocks as proposed in [27] to learn each contraction
mapping in Equation 2, $\mathcal{F}_{\theta_{t}},t=1,\cdots,T$.

$z_{0}\stackrel{{\scriptstyle
F_{\theta_{1}}}}{{\longrightarrow}}z_{1}\longrightarrow\cdots\longrightarrow
z_{T-1}\stackrel{{\scriptstyle F_{\theta_{T}}}}{{\longrightarrow}}z_{T}.$ (4)

The resulting architecture for image restoration is called FPformer and is
shown in Figure 2 and Equation 4.

Naturally, the fixed-point finding in Equation 2 of minimizing Equation 3 is
agnostic to the image restoration task. It just tries to recover a degraded
image $f$ to a restored image $u_{T}$, which is close to the ground-truth
clean image $u_{gt}$. Therefore, to fully explore the capability of the
Transformer, we trained the FPformer using multiple image restoration
problems.

The training of FPformer is formulated as

$\displaystyle\begin{aligned} &\mathop{\min}_{\Theta}\sum_{s\in
S}\mathcal{L}(u_{gt}^{s},u_{T}^{s})\\\ z_{0}^{s}&=S_{\phi}(f_{task}^{s})\\\
z_{T}^{s}&=\mathcal{F}_{\theta_{T}}(\cdots(\mathcal{F}_{\theta_{1}}(z_{0}^{s})))=\text{FPformer}_{\Theta}(z_{0}^{s})\\\
u_{T}^{s}&=U_{\psi}(z_{T}^{s})\end{aligned}$ (5)

where $\Theta=\\{\theta_{1},\cdots,\theta_{T}\\}$ is the parameters in
FPformer, $\mathcal{L}$ is the loss function that measures the difference
between the ground-truth image $u_{gt}^{s}$ and the restored image
$u_{T}^{s}$. $u_{gt}^{s},f_{task}^{s}$ are generated using a specific sample
in dataset $S$, details are described in Section IV. Following [27], both
$S_{\phi}$ and $U_{\psi}$ are convolution operators. To be specific,
$S_{\phi}$ represents the degraded image $f_{task}\in\mathbb{R}^{B\times
H\times W\times 3}$ as the first state $z_{0}\in\mathbb{R}^{B\times H\times
W\times C}$. $B$ is the size of the minibatch, $H$ and $W$ are the height and
width of the image (or patch), and $C$ is the channel number of features.
$U_{\psi}$ maps the last state $z_{T}\in\mathbb{R}^{B\times H\times W\times
C}$ to the restored image $u_{T}\in\mathbb{R}^{B\times H\times W\times 3}$.

In FPformer, both $S_{\phi}$ and $U_{\psi}$ are shared among image restoration
problems and specific tasking settings. To handle different upscales in single
image super-resolution, instead of upscaling the features via upsampling
blocks, we upscaled the downscaled images with the corresponding scaling
factor when preparing the minibatch. Details are discussed in Section IV. In
SwinIR[27], $S_{\phi}$ and $U_{\psi}$ are task-specific convolutional
operators. In single image super-resolution, dedicated upsampling modules are
used. In IPT [28], $S_{\phi}$ and $U_{\psi}$ are also task-specific, only
Transformer blocks are shared. In Restormer [30], the degraded and restored
image have the same resolution. It may explain why single image super-
resolution is not discussed. In summary, FPformer can be treated as a general
image restoration solver. Detail performance is discussed in Section IV.

###  III-B Sharing parameters: FPRformer

In ALBERT [18], sharing parameters among Transformer blocks results in fewer
amounts of models parameters and smaller model sizes. In this context, one can
maintain the depth of the Transformers while regularizing the whole
Transformers, resulting in a small model while providing a competitive
performance.

Inspired by this idea, we enforce successive $N_{j}$ blocks in the $T$ blocks
of FPformer to share parameters, coined as FPRformer. Therefore, we have
$\sum_{j=1}^{R}N_{j}=T$, where $R$ is the number of unique RSTB blocks;
$N_{j}$ is the recurrent times of $j_{th}$ unique RSTB block. The amounts of
the parameters are about $R/T$ times less compared with FPformer with $T$ RSTB
blocks. Note that FPformer can be regarded as a special case of FPRformer with
$R=T$ and $N_{j}=1,\forall j\in\\{1,\cdots,R\\}$.

We trained FPRformer with $R=2$, $R=3$ and $R=T$, separately. The details of
this ablation study are shown in Section IV-B.

###  III-C Anderson Acceleration: FPAformer

We present a theorem to characterize the performance of (4), whose details can
be found in supplementary materials.

###### Theorem 1

[Informal] If the trained model $(\theta_{t})_{1\leq t\leq T}$ can fit
$\mathcal{F}$ well. Let $z^{*}$ be the fixed point of $\mathcal{F}$, we have

$\|z_{T}-z^{*}\|=\mathcal{O}\Big{(}\rho^{T}+\frac{\delta}{1-\rho}\Big{)}$ (6)

for some fixed $0<\rho<1$ and $\delta\geq 0$ reflecting how the model fits (a
smaller $\delta$ indicate better fitting).

Based on Theorem 1, we can immediately get two claims.

  * •

As $T$ increases, the bound of $\|z_{T}-z^{*}\|$ gets small. That indicates
when we use a larger $T$, the unrolling yields better results.

  * •

When $T=\frac{\ln\frac{1-\rho}{\delta}}{\ln\frac{1}{\rho}}$, it holds
$\|z_{T}-z^{*}\|=\mathcal{O}(\frac{\delta}{1-\rho})$. That means as $T$ is
fixed as some integer, the performance is only then determined by how the
model fits.

Following the above two claims, we conclude that the performance of FPRformer
may lag behind that of FPformer because of its parameter-sharing setting. To
boost the performance of FPRformer while enjoying the parameter sharing, we
design a module analog to the Anderson acceleration to explicitly enlarge the
iteration times and get $\delta$ smaller.

As described in Algorithm 2, the Anderson acceleration algorithm accelerates
the FP using the previous states. It is complex to directly translate the
Algorithm 2, especially lines 5-7, into CNNs or RNNs. It is because the
forward and backward computation involves the previous outcome of FP and
causes nested dependency. This will take more GPU memory and cause more
complicated computational graphs which harms the GPU performance greatly. To
this end, we simplify the computation in Algorithm 2 into a recurrent module
depending on the current state $z_{t}$ and hidden state $h_{t}$. Let hidden
state $h_{t}$ to maintain and summarize the previous $m$ states,
$\\{\mathcal{G}_{t-m_{t}+1},\cdots,\mathcal{G}_{t}\\}$. In this way, the
simplified Anderson acceleration of the fixed-point finding algorithm is
formulated as

$\displaystyle\begin{aligned} \hat{z}_{t+1}&=\mathcal{F}_{\theta}(z_{t}),\\\
z_{t+1},h_{t+1}&=\mathcal{H}_{\mu}(z_{t},\hat{z}_{t+1},h_{t}),\end{aligned}$
(7)

where $\mathcal{H}_{\mu}$ is parameterized by $\mu$. Line 6 of Anderson
acceleration algorithm 2 determines the weights of the previous states. Then
the weights combine these states to update the next state, as shown in Line 7
of 2. In our simplified version (7), the weights calculation and combination
can be summarized as GRU [13]. Follow [33], we adopt ConvGRU module to
learning $\mathcal{H}_{\mu}$. The proposed simplified ConvGRU module is as
follows,

$\displaystyle\small\begin{aligned}
\mathcal{G}_{t}&=\mathbf{Conv}(\hat{z}_{t+1}-z_{t}),\\\
r_{h}&=\sigma(\mathbf{Conv}(\mathcal{G}_{t})+\mathbf{Conv}(h_{t})),\\\
r_{z}&=\sigma(\mathbf{Conv}(r_{h})),\\\ h_{t+1}&=\mathbf{Norm}((1-r_{h})\odot
h_{t}+r_{h}\odot\mathbf{Conv}(\mathcal{G}_{t})),\\\
z_{t+1}&=\mathbf{Norm}((1-r_{z})\odot\hat{z}_{t+1}+r_{z}\odot\mathbf{Conv}(\mathcal{G}_{t})),\end{aligned}$
(8)

To analog the $m$ previous states setting in Algorithm 2, the hidden states
$h_{t}\in\mathbb{R}^{B\times H\times W\times mC}$ is $m$ times larger than
$z_{t}\in\mathbb{R}^{B\times H\times W\times C}$. $\mathbf{Conv}$ is 2d
convolutional layer. $\sigma(.)$ is the sigmoid function. $\mathbf{Norm}$ is
the layer normalization [49]. Layer normalization is commonly used in
Transformers and benefits the convergence of the training. Therefore, we add
$\mathbf{Norm}$ on the output of $h_{t+1}$ and $x_{t+1}$. $\odot$ is the
element-wise product. $h_{0}$ is initialized as all zero values.

##  IV Experiments

###  IV-A Experimental Setup

Image restoration tasks in training. We use commonly used image restoration
tasks, e.g., color and gray Gaussian denoising, single image super-resolution
(SISR), and image JPEG deblocking in training. For color and gray Gaussian
denoising, we obtain noisy images by adding additive white Gaussian noises
with noise level $\sigma$ ranging from 0 to 75. For SISR, we downscale and
upscale111We use code from https://github.com/fatheral/matlab_imresize/.
images with scale 2, 3, 4, 8. Note that instead of upscaling the features
$z_{t}$ via upsampling blocks in FPformer, we upscaled the downscaled images
$f$ with the corresponding scaling factor. So, the resulting images $u_{T}$
are in the same spatial resolution as the ground-truth images $u_{gt}$. For
image JPEG deblocking, we generate low-quality images using a JPEG encoder
with quality factor $q$=10, 20, 30, 40, 50. Therefore, we train FPformer for
161 tasks from 4 categories of image restoration problems simultaneously.
FPRformer and FPAformer are trained in the same setting.

Training datasets. Following [27, 42], we train FPformer, FPRformer and
FPAformer in the above image restoration tasks using random cropped patches
from 800 images in DIV2K [50], 2560 images in Flickr2K [51], 300 images in
BSD500 [52] and all images in WED [53]. For DIV2K, we use the first 800
images. For Flickr2K, we use the first 2560 images. For BSD500, we have the
whole 300 images in the trainset. For WED, we use the whole dataset.

Pre-Training. All training of FPformer, FPRformer, and FPAformer is run on a
server with 8 NVIDIA GeForce V100 GPUs. The batch size is 16. The patch sizes
are $48\times 48$, $72\times 72$, $120\times 120$ (window size is $8\times
8$). The RSTB block is set as follows. In FPformer, the number of RSTB blocks
is 9 ($T=9$). In FPRformer and FPAformer, the influences of $T$ are discussed
in Section IV-B. In each RSTB block, the number of Swin Transformer Layer is
6; the channel number $C$ is 240. The head number of MHSA is 8.

When preparing the training minibatch, we first sample clean images from the
above training datasets, crop them into patches (with the above patch size,
e.g.$48\times 48$), and augment these patches. Then we randomly choose an
image restoration problem from color and gray Gaussian denoising, SISR, and
image JPEG deblocking for each augmented patch. For the chosen problem, we
randomly choose the task setting, i.e., the noise level $\sigma$, the scale,
or the quality factor $q$. Then we apply the chosen image degradation and
setting to each augmented patch. The resulting degraded patches and the
augmented clean patches consist of training pairs.

Note that the degraded patches in training pairs are generated on-the-fly
instead of generating degraded images offline as [27]. We augment the training
images using color space convert augmentation, flipping, rotating and other
data augmentation methods as [27].

The learning rate is halved at [500K, 750K, 900K, 950K, 1000K], the initial
learning rate is 2e-4. We use Adam [54] optimizer with $\beta_{1}=0.9$ and
$\beta_{2}=0.99$. The training loss $\mathcal{L}(u_{gt},z_{T})$ in Equation 5
is the Charbonnier loss [55].

Fine-Tuning. When the above pre-training of FPformer, FPRformer, and FPAformer
is done, each of them is capable to restore the degraded images from those
image restoration tasks. To boost the performance for a specific image
restoration problem or task setting, we further finetuned FPformer, FPRformer,
and FPAformer. The initial learning rate and schedule are discussed in Section
IV-B.

Evaluation. We pad the image in testing so that the image size is a multiple
of the window size. We compare the proposed FPformer, FPRformer, and FPAformer
with the previous state-of-the-art methods in color and gray Gaussian
denoising, SISR, and image JPEG deblocking. The performance metrics are PSNR
and SSIM [27]. The evaluation details are shown in IV-C.

Figure 3: The $l_{2}$ distances of input $z_{t}$ and output $z_{t+1}$ for each
layer in SRx4. From left to right, each column FPformer, FPRformer, and
FPAformer. The first row is for pre-trained FPformer and its variants. The
second row is for fine-tuned models for SRx4.

###  IV-B Ablation Studies

In ablation studies, we investigate the influence of key hyper-parameters on
the performance of FPformer, FPRformer, and FPAformer. The RSTB block is set
as above. We compare the performance on Set5 [56] in SISR with scale 4 (SRx4).

The recurrent times $T$ and patch size. We trained FPRformer with $R=2$ and
$N_{1}=1$, $N_{2}=T-1$ given $T$ (6, 8, 10, and 14). We trained FPformer with
the default RSTB setting for the different patch sizes listed above. The
result is summarized in Table I.

The number of parameters in FPformer with different patch sizes is the same,
about 27.8M. The number of parameters in FPRformer is 6.5M. Increasing the
patch size in training is able to extend the image context, which in turn
benefits the performance of FPformer and its variants. Therefore, we believe
the performance of FPformer and its variants will continue to increase with
larger patch size, as progressive training in [30]. Due to the limited GPU
memory, we were not able to train FPformer and its variants with patch size
larger than $120\times 120$.

TABLE I: The influence of $T$ and patch size. The evaluation metric is PSNR. patch | FPformer | FPRformer
---|---|---
size | $T=6$ | $T=8$ | $T=10$ | $T=14$
48 | 32.52 | 32.37 | 32.46 | 32.46 | 32.49
72 | 32.68 | 32.54 | 32.62 | 32.64 | 32.61
120 | 32.78 | 32.64 | 32.76 | 32.78 | 32.78

As the increase of $T$, the performance of FPformer, FPRformer, and FPAformer
increases, as shown in Table I and Table II. The performance of FPAformer
improves as $T$ increases, even achieving better performance than FPformer.
One can find that the performance of FPRformer and FPAformer is on-par with
FPformer. The parameter number of FPRformer and FPAformer is reduced to
$23.4\%$ and $30.2\%$ of FPformer’s, respectively.

The number of previous states $m$ in $\mathcal{H}_{\mu}$. We trained FPAformer
with different kernel sizes, $ks$, and various numbers of previous states $m$.
The result is summarized in the upper table of II. When $ks=1$, as the
increase of $m$ the performance peak is presented in $m=3$. $m$ is set to
analog the number of previous states used in Anderson acceleration. Too large
or too small $m$ harms the performance. When $m=3$, as the increase of $ks$,
the performance is degraded. This indicates that spatial fusion via a larger
convolutional kernel is not as important as temporal fusion via larger $m$.

TABLE II: For a given patch size 48 and 120, $R=2$. In the first super block,
$T=10$, the influence of $m$ and kernel size (ks) in $\mathcal{H}_{\mu}$. In
the second super block, $ks=1$, $m=3$, the influence of $T$ in FPAformer. The
evaluation metric is PSNR.

patch | FPformer | FPAformer ($T=10$)
---|---|---
size | $ks=1$ | $ks=1$ | $ks=1$ | $ks=3$
| $m=1$ | $m=3$ | $m=5$ | $m=3$
48 | 32.52 | 32.47 | 32.52 | 32.48 | 32.45
120 | 32.78 | - | 32.82 | - | -

patch | FPRformer | FPAformer
---|---|---
size |  $ks=1$,$m=3$
| $T=2$ | $T=6$ | $T=10$ | $T=14$
48 | 32.46 | 32.06 | 32.47 | 32.52 | 32.52
120 | 32.78 | - | 32.70 | 32.82 | -

Contraction mapping. Parameter sharing in FPRformer and FPAformer benefits the
memory footprint of the models. It is natural to ask why FPRformer and
FPAformer exploit fewer parameters while providing competitive performance.
And what else can we learn from this setting?

As shown in Figure 3, the $l_{2}$ distances of input $z_{t}$ and output
$z_{t+1}$ for each layer of FPformer becomes larger. On the contrary, those of
FPRformer and FPAformer are narrowing down but do not get close to zero.
Meanwhile, the minimum $l_{2}$ distance of FPAformer is smaller than
FPRformer. Intuitively, both FPRformer and FPAformer seem to be contractive.
It may indicate that FPRformer and FPAformer behave like seeking the fixed-
point of image restoration problems. More curves in Gaussian denoising and
image JPEG deblocking are shown in the supplementary materials.

We further investigate the behavior when repeating FPRformer with another
iteration $T=15$, larger than their training setting, $T=10$. The result is
shown in Figure 4. We ran FPRformer in color Gaussian denoising ranging from 5
to 75 with step 5. These noises were added to images in Set5. The denoising
performance for each $\sigma$ was averaged over images in Set5. Figure 4
indicates that the peak performance is achieved around $T=10\pm 2$. As
iteration exceeds $T=10$, the performance almost stays unchanged. Considering
both phenomena, in FPRformer and FPAformer, one can shorten $T$ to balance
between the image restoration performance and the inference speed.

The ablation studies reveal that the ability to provide almost the same
performance while sharing parameters among blocks comes from the increase of
$T$. As $T$ gets larger, the distance in the final $T_{th}$ layer gets
smaller, the performance gets better. It seems that FPRformer is seeking the
contraction point in the feature space. As mentioned in [15], deep equilibrium
models are implicitly seeking the equilibrium point of Equation 2. On the
contrary, FPformer, FPRformer, and FPAformer are trained to find the
equilibrium point of Equation 2 with $T$ steps, each of them learned via
Transformer blocks. Meanwhile, our proposed methods avoid the computation or
approximation of the inverse Jacobian in [15].

Figure 4: The behavior when repeating FPRformer with another iterations $T=15$, larger than their training setting, $T=10$. TABLE III: Comparison with state-of-the-art methods for single image super-resolution on benchmark datasets in terms of average PSNR/SSIM. The best and second-best results are highlighted and underlined, respectively. $\dagger$ means Fine-tuned for a specific task. $*$ means Pre-trained. Dataset | scale | Methods
---|---|---
NLSA | IPT | SwinIR | FPformer$\dagger$ | FPAformer$\dagger$ | FPformer* | FPAformer*
[57] | [28] | [27] | (ours) | (ours) | (ours) | (ours)
Set5 | 2 | 38.34/0.9618 | 38.37/- | 38.42/0.9623 |  38.42/ 0.9620 |  38.39/0.9618 | 38.30/0.9616 | 38.28/0.9614
[56] | 3 | 34.85/0.9306 | 34.81/- |  34.97/0.9318 | 34.97/0.9317 |  34.98/0.9314 | 34.92/0.9311 | 34.89/0.9308
| 4 | 32.59/0.9000 | 32.64/- | 32.92/0.9044 |  32.92/0.9033 |  32.92/0.9031 | 32.78/0.9024 | 32.82/0.9017
Set14 | 2 | 34.08/0.9231 | 34.43/- | 34.46/0.9250 | 34.43/0.9241 | 34.49/0.9251 | 34.08/0.9224 | 34.08/0.9224
[58] | 3 | 30.70/0.8485 | 30.85/- | 30.93/0.8534 | 30.94/0.8523 |  30.99/0.8520 | 30.85/0.8511 | 30.81/0.8503
| 4 | 28.87/0.7891 | 29.01/- |  29.09/0.7950 |  29.10/0.7939 | 29.07/0.7928 | 29.03/0.7922 | 29.00/0.7909
Manga109 | 2 | 39.59/0.9789 | -/- | 39.92/0.9797 | 39.93/0.9795 |  39.95/0.9795 | 39.70/0.9791 | 39.66/0.9789
[59] | 3 | 34.57/0.9508 | -/- | 35.12/0.9537 | 35.15/0.9532 |  35.22/0.9532 | 34.90/0.9522 | 34.96/0.9520
| 4 | 31.27/0.9184 | -/- | 32.03/0.9260 | 32.05/0.9251 |  32.09/0.9247 | 31.80/0.9227 | 31.87/0.9229
TABLE IV: Comparison with state-of-the-art methods for grayscale and color image denoising on benchmark datasets in terms of average PSNR. The best and second-best results are highlighted and underlined, respectively. From left to right, the first 7 methods are task-specific trained models. The last 4 methods are task-agnostic models. $\dagger$ means Fine-tuned for a specific task. $*$ means Pre-trained. Dataset | $\sigma$ | Methods
---|---|---
BM3D | TNRD | DnCNN | IPT | SwinIR | FPformer$\dagger$ | FPAformer$\dagger$ | DRUNet | ResTormer | FPformer* | FPAformer*
[38] | [5] | [6] | [28] | [27] | (ours) | (ours) | [41] | [30] | (ours) | (ours)
BSD68 | 15 | 31.08 | 31.43 | 31.73 | - | 31.97 | 31.95 | 31.93 | 31.91 | 31.95 | 31.91 | 31.89
[60] | 25 | 28.57 | 28.95 | 29.23 | - | 29.50 | 29.48 | 29.47 | 29.48 | 29.51 | 29.45 | 29.43
| 50 | 25.60 | 26.01 | 26.23 | - | 26.58 | 26.56 | 26.55 | 26.59 | 26.62 | 26.52 | 26.52
CBSD68 | 15 | 33.52 | - | 33.90 | - | 34.42 | 34.40 | 34.37 | 34.30 | 34.39 | 34.34 | 34.31
[60] | 25 | 30.71 | - | 31.24 | - | 31.78 | 31.76 | 31.74 | 31.69 | 31.78 | 31.72 | 31.69
| 50 | 27.38 | - | 27.95 | 28.39 | 28.56 | 28.54 | 28.53 | 28.51 | 28.59 | 28.51 | 28.49
Kodak24 | 15 | 34.28 | - | 34.60 | - | 35.34 | 35.34 | 35.30 | 35.31 | 35.44 | 35.27 | 35.22
[61] | 25 | 32.15 | - | 32.14 | - | 32.89 | 32.88 | 32.86 | 32.89 | 33.02 | 32.83 | 32.79
| 50 | 28.46 | - | 28.95 | 29.64 | 29.79 | 29.79 | 29.77 | 29.86 | 30.00 | 29.75 | 29.72

The learning rate schedule in fine-tuning.

Figure 5: The learning rate schedule in fine-tuning. Fine-tuning is performed
on SRx4, and results are summarized from Set5 with PSNR and SSIM.

After pre-training, one can further fine-tune FPformer, FPRformer, and
FPAformer for a specific image restoration task. Meanwhile, fine-tuning
provides an effective way to train, instead of training each task a model from
scratch. We fine-tuned the pre-trained FPformer (patch size is 48) using a
small learning rate, e.g.5e-5, for another 10w steps. The performance improves
from 32.52dB to 32.57$\sim$32.59dB. The improvement is quite marginal. We also
fine-tuned the pre-trained FPformer (patch size is 48) with a large learning
rate of 2e-4 used in pre-training. The learning rate is halved at [5K, 105K,
185K, 245K, 285K] in the additional steps. The performance improves from
32.52dB to 32.70dB.

In fine-tuning the pre-trained FPformer with patch size 120, we performed
three learning rate schedules starting at 2e-4. The results are summarized in
Figure 5. 5-5-3-2 schedule means the learning rate is halved at [5K, 55K,
105K, 135K, 155K]. It is a quick fine-tuning strategy, the improvement is
about 0.1dB. In the 10-8-6-4 schedule, the learning rate is halved at [5K,
105K, 185K, 245K, 285K]. The fine-tuned models are on-par with SwinIR. In the
20-16-12-8 schedule, the learning rate is halved at [5K, 205K, 365K, 485K,
565K]. It takes about half a step for pre-training. The performance exceeds
SwinIR 0.6dB to 32.98dB. It seems that pre-training provides a better
initialization. Large initial learning rate and long fine-tuning benefit the
performance as well. To balance performance and fine-tuning time consumption,
we use a 10-8-6-4 schedule to fine-tune FPformer, FPRformer, and FPAformer. In
this schedule, the fine-tuning runtime is about one-quarter of pre-training.
Meanwhile, color and grayscale Gaussian denoising are joint fine-tuned. Fine-
tuning these pre-trained models for 13 comparison tasks, takes $26.9\%$ time
training each task-specific model from scratch. Pre-train + fine-tune training
strategy provides an effective training of Vision Transformer-based image
restoration models, saving energy.

###  IV-C Comparison with state-of-the-art

In the following comparison, we choose FPformer with patch size 120.
FPRformer222Due to the limited space, the performance of FPRformer is listed
in the supplementary materials. is with $T=10$ and patch size 120. FPAformer
is with $T=10$, patch size 120, $ks=1$, $M=3$. The pre-trained and fine-tuned
FPformer, FPRformer, and FPAformer are compared with other methods in SISR
(x2, x3, and x4), color and grayscale Gaussian denoising ($\sigma$=15, 25, 50)
and image JPEG deblocking ($q$=10, 20, 30, 40)333Due to the limited space, the
comparison results for image JPEG deblocking are listed in the supplementary
materials..

Comparison with DEQ and JFB. We adopted and trained JFB and DEQ framework in
[3] for single image super-resolution with scale 2. The $\mathcal{F}_{\theta}$
was built on the network (38.7M parameters) for CIFAR10 in [3]. We finetuned
key hyper-parameters, i.e., $T$ and $\epsilon$. Comparison is conducted on
Set5 in terms of PSNR and SSIM. JFB achieves 33.67/0.9303 (PSNR/SSIM), and DEQ
achieves 38.15/0.9608. As shown in Table III, JFB and DEQ are behind others a
lot.

SISR. We test comparison methods on Set5 [56], Set14 [58] and Manga109 [59]
for SISR with scale 2, 3 and 4. Following [27], we report PSNR and SSIM on the
Y channel of the YCbCr space, as summarized in Table III. FPformer†
outperforms IPT (115.5M parameters) and is on-par with SwinIR. FPAformer† uses
fewer model parameters and is on-par with SwinIR on Set5, outperforming SwinIR
about 0.02dB on Set14 and 0.06dB on Manga109.

Image denoising. We test comparison methods on BSD68 [60] for grayscale
denoising with noise levels 15, 25, and 50. We compare color denoising with
noise levels 15, 25, and 50 on CBSD68 [60] and Kodak24 [61]. Following [27],
we report the PSNR on the RGB channel and Y channel for color and grayscale
denoising, respectively. The experimental results are summarized in Table IV.
In task-specific methods, the first 7 methods in Table IV, FPformer† and
FPAformer† outperform IPT and are on-par with SwinIR. In task-agnostic
methods, the last 4 methods in Table IV, FPformer∗ and FPAformer∗ use fewer
model parameters and provide competitive performance with DRUNet (32.7M
parameters) and Restormer (25.3M parameters). The number of model parameters
of FPAformer is only $7.3\%$ of those in IPT, $25.7\%$ of DRUNet, $33.2\%$ of
Restormer.

##  V Conclusion

In this work, we propose to learn the unroll of fixed-point via Transformer
based models, called FPformer. By sharing parameters, we achieved a
lightweight model, FPRformer. A module is proposed to analog the Anderson
acceleration to boost the performance of FPRformer, called FPAformer. To fully
exploit the capability of Transformer, we apply the proposed model to image
restoration, using self-supervised pre-training and supervised fine-tuning.
The proposed FPformer, FPRformer, and FPAformer use fewer parameters and
achieve competitive performance with state-of-the-art image restoration
methods and better training efficiency. FPRformer and FPAformer use only
23.21% and 29.82% parameters used in SwinIR models, respectively. To train
these comparison models, we use only 26.9% time used for training from
scratch.

## Acknowledgments

This work was supported by the National Natural Science Foundation of China
under the Grant No.61902415.

## References

  * [1] S. Bai, Z. Geng, Y. Savani, and J. Z. Kolter, “Deep equilibrium optical flow estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 620–630.
  * [2] M. Chang, T. L. Griffiths, and S. Levine, “Object representations as equilibria: Training iterative inference algorithms with implicit differentiation,” in _ICLR 2022 Workshop on Gamification and Multiagent Solutions_ , 2022.
  * [3] S. W. Fung, H. Heaton, Q. Li, D. McKenzie, S. Osher, and W. Yin, “Jfb: Jacobian-free backpropagation for implicit networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2022.
  * [4] H. Heaton, D. McKenzie, Q. Li, S. W. Fung, S. Osher, and W. Yin, “Learn to predict equilibria via fixed point networks,” _arXiv preprint arXiv:2106.00906_ , 2021.
  * [5] Y. Chen and T. Pock, “Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 6, pp. 1256–1272, 2016.
  * [6] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising,” _IEEE transactions on image processing_ , vol. 26, no. 7, pp. 3142–3155, 2017.
  * [7] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” California Univ San Diego La Jolla Inst for Cognitive Science, Tech. Rep., 1985.
  * [8] P. J. Werbos, “Backpropagation through time: what it does and how to do it,” _Proceedings of the IEEE_ , vol. 78, no. 10, pp. 1550–1560, 1990.
  * [9] P. Qiao, Y. Dou, W. Feng, R. Li, and Y. Chen, “Learning non-local image diffusion for image denoising,” in _Proceedings of the 25th ACM international conference on Multimedia_ , 2017, pp. 1847–1855.
  * [10] A. Gruslys, R. Munos, I. Danihelka, M. Lanctot, and A. Graves, “Memory-efficient backpropagation through time,” _Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * [11] Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” _IEEE transactions on neural networks_ , vol. 5, no. 2, pp. 157–166, 1994.
  * [12] R. Jozefowicz, W. Zaremba, and I. Sutskever, “An empirical exploration of recurrent network architectures,” in _International conference on machine learning_. PMLR, 2015, pp. 2342–2350.
  * [13] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _arXiv preprint arXiv:1406.1078_ , 2014.
  * [14] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [15] S. Bai, J. Z. Kolter, and V. Koltun, “Deep equilibrium models,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [16] S. Bai, V. Koltun, and J. Z. Kolter, “Multiscale deep equilibrium models,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 5238–5250, 2020.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [18] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” _arXiv preprint arXiv:1909.11942_ , 2019.
  * [19] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [20] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [21] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” _arXiv preprint arXiv:1909.08053_ , 2019.
  * [22] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu _et al._ , “Exploring the limits of transfer learning with a unified text-to-text transformer.” _J. Mach. Learn. Res._ , vol. 21, no. 140, pp. 1–67, 2020.
  * [23] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021.
  * [24] H. Bao, L. Dong, S. Piao, and F. Wei, “BEit: BERT pre-training of image transformers,” in _International Conference on Learning Representations_ , 2022.
  * [25] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 000–16 009.
  * [26] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 012–10 022.
  * [27] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1833–1844.
  * [28] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-trained image processing transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 12 299–12 310.
  * [29] W. Li, X. Lu, J. Lu, X. Zhang, and J. Jia, “On efficient transformer and image pre-training for low-level vision,” _arXiv preprint arXiv:2112.10175_ , 2021\.
  * [30] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5728–5739.
  * [31] D. G. Anderson, “Iterative procedures for nonlinear integral equations,” _Journal of the ACM (JACM)_ , vol. 12, no. 4, pp. 547–560, 1965.
  * [32] A. Toth and C. T. Kelley, “Convergence analysis for anderson acceleration,” _SIAM Journal on Numerical Analysis_ , vol. 53, no. 2, pp. 805–819, 2015\.
  * [33] Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in _European conference on computer vision_. Springer, 2020, pp. 402–419.
  * [34] D. Gilton, G. Ongie, and R. Willett, “Deep equilibrium architectures for inverse problems in imaging,” _IEEE Transactions on Computational Imaging_ , vol. 7, pp. 1123–1133, 2021.
  * [35] S. Bai, V. Koltun, and J. Z. Kolter, “Neural deep equilibrium solvers,” in _International Conference on Learning Representations_ , 2021.
  * [36] L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” _Physica D: nonlinear phenomena_ , vol. 60, no. 1-4, pp. 259–268, 1992.
  * [37] A. Buades, B. Coll, and J.-M. Morel, “A review of image denoising algorithms, with a new one,” _Multiscale modeling & simulation_, vol. 4, no. 2, pp. 490–530, 2005.
  * [38] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” _IEEE Transactions on image processing_ , vol. 16, no. 8, pp. 2080–2095, 2007.
  * [39] A. Danielyan, V. Katkovnik, and K. Egiazarian, “Bm3d frames and variational image deblurring,” _IEEE Transactions on image processing_ , vol. 21, no. 4, pp. 1715–1728, 2011.
  * [40] S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 2862–2869.
  * [41] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, “Plug-and-play image restoration with deep denoiser prior,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [42] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in _Proceedings of the European conference on computer vision (ECCV) workshops_ , 2018, pp. 0–0.
  * [43] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 14 821–14 831.
  * [44] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 43, no. 7, pp. 2480–2495, 2020.
  * [45] X. Fu, X. Wang, A. Liu, J. Han, and Z.-J. Zha, “Learning dual priors for jpeg compression artifacts removal,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 4086–4095.
  * [46] L. Li, J. Pan, W.-S. Lai, C. Gao, N. Sang, and M.-H. Yang, “Learning a discriminative prior for blind image deblurring,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6616–6625.
  * [47] X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, “Scale-recurrent network for deep image deblurring,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8174–8182.
  * [48] Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li, “Maxim: Multi-axis mlp for image processing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5769–5780.
  * [49] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [50] E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 126–135.
  * [51] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, and L. Zhang, “Ntire 2017 challenge on single image super-resolution: Methods and results,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 114–125.
  * [52] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 33, no. 5, pp. 898–916, 2010.
  * [53] K. Ma, Z. Duanmu, Q. Wu, Z. Wang, H. Yong, H. Li, and L. Zhang, “Waterloo exploration database: New challenges for image quality assessment models,” _IEEE Transactions on Image Processing_ , vol. 26, no. 2, pp. 1004–1016, 2016\.
  * [54] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [55] P. Charbonnier, L. Blanc-Feraud, G. Aubert, and M. Barlaud, “Two deterministic half-quadratic regularization algorithms for computed imaging,” in _Proceedings of 1st International Conference on Image Processing_ , vol. 2. IEEE, 1994, pp. 168–172.
  * [56] M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” 2012\.
  * [57] Y. Mei, Y. Fan, and Y. Zhou, “Image superresolution with non-local sparse attention,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2021, pp. 3517–3526.
  * [58] R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in _International conference on curves and surfaces_. Springer, 2010, pp. 711–730.
  * [59] Y. Matsui, K. Ito, Y. Aramaki, A. Fujimoto, T. Ogawa, T. Yamasaki, and K. Aizawa, “Sketch-based manga retrieval using manga109 dataset,” _Multimedia Tools and Applications_ , vol. 76, no. 20, pp. 21 811–21 838, 2017.
  * [60] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , vol. 2. IEEE, 2001, pp. 416–423.
  * [61] R. Franzen, “Kodak lossless true color image suite,” _http://r0k.us/graphics/kodak_ , 1999.
  * [62] A. Foi, V. Katkovnik, and K. Egiazarian, “Pointwise shape-adaptive dct for high-quality denoising and deblocking of grayscale and color images,” _IEEE transactions on image processing_ , vol. 16, no. 5, pp. 1395–1411, 2007\.
  * [63] H. Sheikh, “Live image quality assessment database release 2,” _http://live.ece.utexas.edu/research/quality_ , 2005.

##  Appendix A Analysis

In this section, we give a detailed explanation of our analysis in the main
paper.

###### Condition 1

The operator $\mathcal{F}$ is contractive, that is,
$\|\mathcal{F}(z^{\prime})-\mathcal{F}(z)\|\leq\rho\|z^{\prime}-z\|$ for some
$0<\rho<1$.

###### Condition 2

$(\theta_{t})_{t\geq 0}$ can approximate $\mathcal{F}$ well, that is,
$\|\mathcal{F}_{\theta_{t}}(z)-\mathcal{F}(z)\|\leq\delta$ for some small
$\delta>0$ and all $z$.

Condition 1 is the contractive property, commonly used in the optimization
area. It is essential for the convergence analysis of Algorithm 1. While
Condition 2 is to characterize that $(\theta_{t})_{t\geq 0}$ are trained by a
acceptable model. With these two conditions, we can derive the following
result.

###### Theorem 2

[Restatement of Theorem 1] Assume Conditions 1 and 2 hold, and $(z_{t})_{1\leq
t\leq T}$ is generated by (5). Let $z^{*}$ be the fixed point of
$\mathcal{F}$, we have

$\|z_{T}-z^{*}\|\leq\rho^{T}\|z_{0}-z^{*}\|+\frac{\delta}{1-\rho}.$ (9)

Proof of Theorem 2: Noticing that $z_{t+1}=\mathcal{F}_{\theta^{t}}(z_{t})$,
we have

$\displaystyle\|z_{t+1}-z^{*}\|$
$\displaystyle=\|\mathcal{F}_{\theta_{t}}(z_{t})-z^{*}\|$
$\displaystyle=\|\mathcal{F}_{\theta_{t}}(z_{t})-\mathcal{F}(z_{t})+\mathcal{F}(z_{t})-z^{*}\|$
$\displaystyle\leq\|\mathcal{F}_{\theta_{t}}(z_{t})-\mathcal{F}(z_{t})\|+\|\mathcal{F}(z_{t})-z^{*}\|$
$\displaystyle\overset{\textrm{Condition}~{}\ref{cond2}}{\leq}\delta+\|\mathcal{F}(z_{t})-z^{*}\|$
$\displaystyle=\delta+\|\mathcal{F}(z_{t})-\mathcal{F}(z^{*})\|$
$\displaystyle\overset{\textrm{Condition}~{}\ref{cond1}}{\leq}\delta+\rho\|z_{t}-z^{*}\|.$

Thus, as $T\geq 1$, we are led to

$\displaystyle\|z_{T}-z^{*}\|$
$\displaystyle\leq\rho^{t}\|z_{0}-z^{*}\|+\delta\sum_{j=0}^{T-1}\rho^{j}$
$\displaystyle\leq\rho^{T}\|z_{0}-z^{*}\|+\frac{\delta}{1-\rho}.$

From Theorem 2, as $T$ is large enough, it holds
$\|z_{T}-z^{*}\|\approx\frac{\delta}{1-\rho}$. Noticing that $\delta$ is to
reflect the level of how the trained model fit $\mathcal{F}$. Hence, the
result indicates that when the $T$ is large enough and the model is trained
well ($\delta>0$ is small), $z_{T}$ is sufficiently close to $z^{*}$.

Based on Theorem 2, we can get two claims as described in the main paper. In
summary, 1) as $T$ increases, the bound of $\|z_{T}-z^{*}\|$ gets small. That
indicates when we use a larger $T$, the unrolling yields better results. 2)
When $T=\frac{\ln\frac{1-\rho}{\delta}}{\ln\frac{1}{\rho}}$, it holds
$\|z_{T}-z^{*}\|=\mathcal{O}(\frac{\delta}{1-\rho})$. That means as $T$ is
fixed as some integer, the performance is only then determined by how the
model fits. Following these two claims, we conclude that the performance of
FPRformer may lag behind that of FPformer because of its parameter-sharing
setting. Following the analysis, we design a module analog to the Anderson
acceleration to explicitly enlarge the iteration times and get $\delta$
smaller.

##  Appendix B Related works

###  B-A Fixed-Point Finding Algorithm

The fixed-point finding in Algorithm 1, generates a series of
$\\{{z_{t}}\\}_{t=1}^{T}$ by successively applying the contraction mapping
$\mathcal{F}(.)$, given a initial point $z_{0}$.

0: $z_{0}$: initial point, $\epsilon$: error bound, $T$, maximum iteration
times.

0: $z^{*}$: end point satisfying either $\epsilon$ error bound or $T$ maximum
iteration

1: $t\leftarrow 0$

2: while $\|z_{t}-\mathcal{F}(z_{t})\|\geq\epsilon$ and $t\leq T$ do

3: $z_{t+1}\leftarrow\mathcal{F}(z_{t})$

4: $t\leftarrow t+1$

5: end while

6: return $z^{*}$

Algorithm 1 Algorithm for Fixed-Point Finding

In DNN-FP methods, the contraction mapping $\mathcal{F}(.)$ is parameterized
by $\theta$ or a couple of $\theta_{t}$ as $\mathcal{F}_{\theta}$ and
$\mathcal{F}_{\theta_{t}}$, respectively. In both forms, the iteration process
in Algorithm 1 requires quite a lot of iterations, i.e., a large $T$, to reach
a feasible equilibrium point $z^{*}$. One should also note that the choice of
hyper-parameters of $\epsilon$ and $T$ is vitally important to achieving a
good performance using $\mathcal{F}_{\theta}$, whose contraction property is
not well-guaranteed. When applying DNN-FP methods, repeating the modern DNN a
couple of times is computation-consuming and time-consuming.

###  B-B Anderson Acceleration Algorithm for Fixed-Point Finding

Anderson acceleration ([31, 32], AA) is proposed to accelerate the fixed-point
finding utilizing the previous $m$ states $\\{z_{t-m+i}\\}_{i=1}^{m}$ to
estimate the next state $z_{t+1}$, as shown in Algorithm 2. As line 6 in
Algorithm 2, the weights are obtained by solving a least-squares method. Then
these normalized weights (whose summation is equal to 1) are used to linearly
combine the previous $m-1$ states and the latest estimate from
$\mathcal{F}(z_{t})$ to generate the next state $z_{t+1}$.

0: $z_{0}$: initial point, $\epsilon$: error bound, $T$, maximum iteration
times, $m$, depth of memory, $m\geq 1$

0: $z^{*}$: end point satisfying either $\epsilon$ error bound or $T$ maximum
iteration

1: $t\leftarrow 1$

2: $z_{1}\leftarrow\mathcal{F}(z_{0})$

3: while $\|z_{t}-\mathcal{F}(z_{t})\|\geq\epsilon$ and $t\leq T$ do

4: $m_{t}=\min\\{m,t\\}$

5: Given $\\{\mathcal{G}_{t-m_{t}+1},\cdots,\mathcal{G}_{t}\\}$, where
$\mathcal{G}_{i}=\mathcal{F}(z_{i})-z_{i}$

6:
$\left(\alpha_{1},\cdots,\alpha_{m_{k}}\right)=\arg\min\sum_{i=1}^{m_{t}}\|\alpha_{i}\mathcal{G}_{t-m_{t}+i}\|$
s.t. $\sum_{i=1}^{m_{t}}\alpha_{i}=1$

7: $z_{t+1}\leftarrow\sum_{i=1}^{m_{t}}\alpha_{i}\mathcal{F}(z_{t-m_{t}+i})$

8: $t\leftarrow t+1$

9: end while

10: return $z^{*}$

Algorithm 2 Anderson Acceleration for Fixed-Point Finding

##  Appendix C Experiments

###  C-A Ablation Studies

Removing residual connection in RSTB and changing the head number of MHSA. For
a given patch size 48, $T=10$, and $R=2$, we conducted experiments for
removing residual connections in RSTB. We also investigate the influence of
different head numbers (4, 8, and 12). The results are shown in Table V. The
experimental results show that removing residual connection in RSTB and the
head number of MHSA has little impact on the performance. This is different
from [27], in which residual connection and convolution in residual connection
have a significant impact on performance. We tend to believe that SwinIR
trained for a specific task, the increase of parameter number will benefit the
performance. However, in our FPformer, FPRformer, and FPAformer trained for a
couple of image restoration tasks, the benefit is marginal.

TABLE V: The influence of Removing residual connection in RSTB and the head number of MHSA. The evaluation metric is PSNR. no resi- |  FPRformer ($T=10$)
---|---
dual | $\\#head=4$ | $\\#head=8$ | $\\#head=12$
32.42 | 32.43 | 32.46 | 32.46

Combination of $N_{j}$s. For a given patch size 48, $T=10$ and $R=2$, we
conducted experiments for different combination of $N_{1}$ and $N_{2}$. The
results are summarized in Table VI. The corresponding FPRformers achieve
almost the same performance. For $R=3$, we have observed a phenomenon similar
to that in the $R=2$ scenario.

TABLE VI: For a given patch size 48, $T=10$ and $R=2$, the influence of
different combination of $N_{1}$ and $N_{2}$. The evaluation metric is PSNR.
FPRformer
---
$N_{1}=1$ | $N_{1}=2$ | $N_{1}=3$ | $N_{1}=4$ | $N_{1}=5$
$N_{2}=9$ | $N_{2}=8$ | $N_{2}=7$ | $N_{2}=6$ | $N_{2}=5$
32.46 | 32.45 | 32.46 | 32.46 | 32.43
- | $N_{1}=6$ | $N_{1}=7$ | $N_{1}=8$ | $N_{1}=9$
- | $N_{2}=4$ | $N_{2}=3$ | $N_{2}=2$ | $N_{2}=1$
- | 32.42 | 32.43 | 32.44 | 32.43

Contraction. The $l_{2}$ distances and the cosine similarity of input $z_{t}$
and output $z_{t+1}$ in color Gaussian denoising and image JPEG deblocking,
are illustrated in Figure 6 and 7. The same behavior occurs in color Gaussian
denoising and image JPEG deblocking as in SISR for FPformer. In FPRformer and
FPAformer, similar trends are also presented. For a given $T$, as $t$
increases, the $l_{2}$ distance is getting smaller. For $T=14$, one can also
note that the l2 distance of input and output embeddings increases in the last
three iterations. And this trend is quite obvious in JPEG deblocking with
$q=10$ for $T=6,8,10,14$. The cosine similarity shows that these last
iterations are still close to each other.

Figure 6: The $l_{2}$ distances and cosine similarity of input $z_{t}$ and
output $z_{t+1}$ for each layer, for color Gaussian denoising with
$\sigma=50$. From left to right, it is for FPformer, FPRformer and
FPAformer.The first row is the $l_{2}$ distances of input $z_{t}$ and output
$z_{t+1}$ for each layer in color Gaussian denoising with $\sigma=50$. The
second row is the cosine similarity of input $z_{t}$ and output $z_{t+1}$ for
each layer in color Gaussian denoising with $\sigma=50$. Figure 7: The $l_{2}$
distances and cosine similarity of input $z_{t}$ and output $z_{t+1}$ for each
layer, for JPEG deblocking with $q=10$. From left to right, it is for
FPformer, FPRformer and FPAformer. The first row is the $l_{2}$ distances of
input $z_{t}$ and output $z_{t+1}$ for each layer in JPEG deblocking with
$q=10$. The second row is the cosine similarity of input $z_{t}$ and output
$z_{t+1}$ for each layer in JPEG deblocking with $q=10$.

###  C-B Comparison with SOTA methods

SISR. We test comparison methods on Set5 [56], Set14 [58] and Manga109 [59]
for SISR with scale 2, 3 and 4. Following [27], we report PSNR and SSIM on the
Y channel of the YCbCr space, as summarized in Table VII. FPformer†
outperforms IPT (115.5M parameters) and is on-par with SwinIR. FPAformer† uses
fewer model parameters and is on-par with SwinIR on Set5, outperforming SwinIR
about 0.02dB on Set14 and 0.06dB on Manga109.

TABLE VII: Comparison with state-of-the-art methods for single image super-resolution on benchmark datasets in terms of average PSNR/SSIM. The best and second-best results are highlighted and underlined, respectively. $\dagger$ means Fine-tuned for a specific task. $*$ means Pre-trained. Dataset | scale | Methods
---|---|---
NLSA | IPT | SwinIR | FPformer$\dagger$ | FPAformer$\dagger$ | FPformer* | FPRformer* | FPAformer*
[57] | [28] | [27] | (ours) | (ours) | (ours) | (ours) | (ours)
Set5 | 2 | 38.34/0.9618 | 38.37/- | 38.42/0.9623 |  38.42/ 0.9620 |  38.39/0.9618 | 38.30/0.9616 | 38.26/0.9614 | 38.28/0.9614
[56] | 3 | 34.85/0.9306 | 34.81/- |  34.97/0.9318 | 34.97/0.9317 |  34.98/0.9314 | 34.92/0.9311 | 34.85/0.9306 | 34.89/0.9308
| 4 | 32.59/0.9000 | 32.64/- | 32.92/0.9044 |  32.92/0.9033 |  32.92/0.9031 |  32.78/0.9024 | 32.78/0.9017 |  32.82/0.9017
Set14 | 2 | 34.08/0.9231 | 34.43/- | 34.46/0.9250 | 34.43/0.9241 | 34.49/0.9251 | 34.08/0.9224 | 34.22/0.9238 | 34.08/0.9224
[58] | 3 | 30.70/0.8485 | 30.85/- | 30.93/0.8534 | 30.94/0.8523 |  30.99/0.8520 | 30.85/0.8511 | 30.74/0.8498 | 30.81/0.8503
| 4 | 28.87/0.7891 | 29.01/- |  29.09/0.7950 |  29.10/0.7939 | 29.07/0.7928 | 29.03/0.7922 | 29.01/0.7918 | 29.00/0.7909
Manga109 | 2 | 39.59/0.9789 | -/- | 39.92/0.9797 | 39.93/0.9795 |  39.95/0.9795 | 39.70/0.9791 | 39.61/0.9787 | 39.66/0.9789
[59] | 3 | 34.57/0.9508 | -/- | 35.12/0.9537 | 35.15/0.9532 |  35.22/0.9532 | 34.90/0.9522 | 34.84/0.9517 | 34.96/0.9520
| 4 | 31.27/0.9184 | -/- | 32.03/0.9260 | 32.05/0.9251 |  32.09/0.9247 | 31.80/0.9227 | 31.70/0.9221 | 31.87/0.9229

The visual comparison for SRx4 of the comparison methods are shown in Figures
8 and 9.

Figure 8: Visual comparison of SR×4 methods on image baby from Set5[56]. From
top to bottom and left to right, the images are from original, SwinIR,
FPformer∗, FPformer†, FPAformer∗, and FPAformer†. Best viewed by zooming.
Figure 9: Visual comparison of SR×4 methods on image butterfly from Set5[56].
From top to bottom and left to right, the images are from original, SwinIR,
FPformer†, FPformer∗, FPAformer†, and FPAformer∗. Best viewed by zooming.

Image denoising. We test comparison methods on BSD68 [60] for grayscale
denoising with noise levels 15, 25, and 50. We compare color denoising with
noise levels 15, 25, and 50 on CBSD68 [60] and Kodak24 [61]. Following [27],
we report the PSNR on the RGB channel and Y channel for color and grayscale
denoising, respectively. The experimental results are summarized in Table
VIII. In task-specific methods, the first 7 methods in Table VIII, FPformer†
and FPAformer† outperform IPT and are on-par with SwinIR. In task-agnostic
methods, the last 5 methods in Table VIII, FPformer∗, FPRformer∗, and
FPAformer∗ use fewer model parameters and provide competitive performance with
DRUNet (32.7M parameters) and Restormer (25.3M parameters). The number of
model parameters of FPAformer is only $7.3\%$ of those in IPT, $25.7\%$ of
DRUNet, $33.2\%$ of Restormer.

TABLE VIII: Comparison with state-of-the-art methods for grayscale and color image denoising on benchmark datasets in terms of average PSNR. The best and second-best results are highlighted and underlined, respectively. From left to right, the first 7 methods are task-specific trained models. The last 5 methods are task-agnostic models. $\dagger$ means Fine-tuned for a specific task. $*$ means Pre-trained. Dataset | $\sigma$ | Methods
---|---|---
BM3D | TNRD | DnCNN | IPT | SwinIR | FPformer$\dagger$ | FPAformer$\dagger$ | DRUNet | ResTormer | FPformer* | FPRformer* | FPAformer*
[38] | [5] | [6] | [28] | [27] | (ours) | (ours) | [41] | [30] | (ours) | (ours) | (ours)
BSD68 | 15 | 31.08 | 31.43 | 31.73 | - | 31.97 | 31.95 | 31.93 | 31.91 | 31.95 | 31.91 | 31.89 | 31.89
[60] | 25 | 28.57 | 28.95 | 29.23 | - | 29.50 | 29.48 | 29.47 | 29.48 | 29.51 | 29.45 | 29.43 | 29.43
| 50 | 25.60 | 26.01 | 26.23 | - | 26.58 | 26.56 | 26.55 | 26.59 | 26.62 | 26.52 | 26.52 | 26.52
CBSD68 | 15 | 33.52 | - | 33.90 | - | 34.42 | 34.40 | 34.37 | 34.30 | 34.39 | 34.34 | 34.28 | 34.31
[60] | 25 | 30.71 | - | 31.24 | - | 31.78 | 31.76 | 31.74 | 31.69 | 31.78 | 31.72 | 31.66 | 31.69
| 50 | 27.38 | - | 27.95 | 28.39 | 28.56 | 28.54 | 28.53 | 28.51 | 28.59 | 28.51 | 28.47 | 28.49
Kodak24 | 15 | 34.28 | - | 34.60 | - | 35.34 | 35.34 | 35.30 | 35.31 | 35.44 | 35.27 | 35.21 | 35.22
[61] | 25 | 32.15 | - | 32.14 | - | 32.89 | 32.88 | 32.86 | 32.89 | 33.02 | 32.83 | 32.78 | 32.79
| 50 | 28.46 | - | 28.95 | 29.64 | 29.79 | 29.79 | 29.77 | 29.86 | 30.00 | 29.75 | 29.70 | 29.72

The visual comparison for grayscale and color Gaussian denoising with
$\sigma=50$ of the comparison methods are shown in Figures 10 and 11,
respectively.

Figure 10: Visual comparison of grayscale Gaussian denoising with $\sigma=50$
methods on image test021 from BSD68[60]. From top to bottom and left to right,
the images are from original, SwinIR, Restormer with blind setting, Restormer
with nonblind setting, FPformer∗, FPformer†, FPAformer∗, and FPAformer†. Best
viewed by zooming. Figure 11: Visual comparison of color Gaussian denoising
with $\sigma=50$ methods on image 12084 from CBSD68[60]. From top to bottom
and left to right, the images are from original, SwinIR, FPformer†, FPformer∗,
FPAformer†, and FPAformer∗. Best viewed by zooming.

Image JPEG deblocking. We test comparison methods on Classic5 [62] and LIVE1
[63] for image JPEG deblocking with quality factor $q=10,20,30,40$. Following
[27], we report PSNR and SSIM on the Y channel, as shown in Table IX. The
FPAformer† use fewer parameters, and provides very competitive performance
compared with SwinIR. The FPformer∗ outperforms DRUNet. Both FPRformer∗ and
FPAformer∗ outperform DRUNet on Classic5, and are on-par with DRUNet on LIVE1.

TABLE IX: Comparison with state-of-the-art methods for image JPEG deblocking on benchmark datasets in terms of average PSNR/SSIM. The best and second best results are highlighted and underlined, respectively. $\dagger$ means Fine-tuned for a specific task. $*$ means Pre-trained. Dataset | $q$ | Methods
---|---|---
RDN | DRUNet | SwinIR | FPAformer$\dagger$ | FPformer* | FPRformer* | FPAformer*
[44] | [41] | [27] | (ours) | (ours) | (ours) | (ours)
Classic5[62] | 10 | 30.00/0.8188 | 30.16/0.8234 | 30.27/0.8249 | 30.23/0.8239 | 30.21/0.8235 | 30.16/0.8223 | 30.17/0.8225
20 | 32.15/0.8699 | 32.39/0.8734 | 32.52/0.8748 | 32.48/0.8742 | 32.43/0.8735 | 32.39/0.8729 | 32.40/0.8731
30 | 33.43/0.8930 | 33.59/0.8949 | 33.73/0.8961 | 33.71/0.8958 | 33.67/0.8952 | 33.61/0.8945 | 33.63/0.8948
40 | 34.27/0.9061 | 34.41/0.9075 | 34.52/0.9082 | 34.52/0.9081 | 34.49/0.9078 | 34.43/0.9073 | 34.45/0.9075
LIVE1[63] | 10 | 29.67/0.8247 | 29.79/0.8278 | 29.86/0.8287 | 29.83/0.8279 | 29.80/0.8276 | 29.77/0.8268 | 29.78/0.8268
20 | 32.07/0.8882 | 32.17/0.8899 | 32.25/0.8909 | 32.22/0.8901 | 32.20/0.8900 | 32.15/0.8895 | 32.17/0.8895
30 | 33.51/0.9153 | 33.59/0.9166 | 33.69/0.9174 | 33.65/0.9168 | 33.62/0.9166 | 33.57/0.9162 | 33.59/0.9162
40 | 34.51/0.9302 | 34.58/0.9312 | 34.67/0.9317 | 34.64/0.9313 | 34.61/0.9312 | 34.56/0.9308 | 34.57/0.9308

The visual comparison for image JPEG deblocking with quality factor $q=10$ of
the comparison methods are shown in Figures 12 and 13.

Figure 12: Visual comparison of image JPEG deblocking with quality factor
$q=10$ methods on image barbara from Classic5[62]. From top to bottom and left
to right, the images are from original, SwinIR, FPAformer∗, and FPAformer†.
Best viewed by zooming. Figure 13: Visual comparison of image JPEG deblocking
with quality factor $q=10$ methods on image buildings from LIVE1[63]. From top
to bottom and left to right, the images are from original, SwinIR, FPAformer†,
and FPAformer∗. Best viewed by zooming.