Dinh Phu<EMAIL_ADDRESS>Dao Duy<EMAIL_ADDRESS>Daeyoung
<EMAIL_ADDRESS>Korea Advanced Institute of
Science and Technology
Daejeon, Korea Channel-Partitioned Windowed Attention & Frequency Learning for
SISR

# Channel-Partitioned Windowed Attention And Frequency Learning for Single
Image Super-Resolution

###### Abstract

Recently, window-based attention methods have shown great potential for
computer vision tasks, particularly in Single Image Super-Resolution (SISR).
However, it may fall short in capturing long-range dependencies and
relationships between distant tokens. Additionally, we find that learning on
spatial domain does not convey the frequency content of the image, which is a
crucial aspect in SISR. To tackle these issues, we propose a new Channel-
Partitioned Attention Transformer (CPAT) to better capture long-range
dependencies by sequentially expanding windows along the height and width of
feature maps. In addition, we propose a novel Spatial-Frequency Interaction
Module (SFIM), which incorporates information from spatial and frequency
domains to provide a more comprehensive information from feature maps. This
includes information about the frequency content and enhances the receptive
field across the entire image. Experimental findings demonstrate the
effectiveness of our proposed modules and architecture. In particular, CPAT
surpasses current state-of-the-art methods by up to 0.31dB.

##  1 Introduction

Single Image Super-Resolution (SISR) is a low-level vision task that aims to
enhance a low-resolution (LR) image into a high-resolution (HR) image.

Initially, convolutional neural networks [Dong et al.(2015)Dong, Loy, He, and
Tang, Kim et al.(2016a)Kim, Lee, and Lee, Kim et al.(2016b)Kim, Lee, and Lee,
Zhang et al.(2018a)Zhang, Li, Li, Wang, Zhong, and Fu, Li et al.(2018)Li,
Fang, Mei, and Zhang] achieved outstanding results in SISR a few years ago.
However, it does not possess the capability to gather global contextual
information, as they mainly focus on nearby areas and might miss out on
important connections that are far apart. Recently, Transformers, which
utilize self-attention mechanisms, excel at modeling long-range dependencies
not only with high-level vision tasks such as Image Captioning [Wang et
al.(2022)Wang, Xu, and Sun, He et al.(2020)He, Liao, Tavakoli, Yang,
Rosenhahn, and Pugeault], 3D-aware Image Synthesis [Sargent et
al.(2023)Sargent, Koh, Zhang, Chang, Herrmann, Srinivasan, Wu, and Sun],
$etc.$, but also with low-level vision tasks such as localization [Chen et
al.(2022a)Chen, Du, Yang, Beyer, Zhai, Lin, Chen, Li, Song, Wang, and Zhou],
segmentation [Huang et al.(2023)Huang, Wang, Wei, Huang, Shi, Liu, and Huang,
Chen et al.(2021b)Chen, Lu, Yu, Luo, Adeli, Wang, Lu, Yuille, and Zhou, Tran
et al.(2022)Tran, Nguyen, Pham, and Tran], $etc.$, including SISR [Chen et
al.(2021a)Chen, Wang, Guo, Xu, Deng, Liu, Ma, Xu, Xu, and Gao, Liang et
al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and Timofte, Chen et
al.(2022b)Chen, Zhang, Gu, Kong, Yuan, et al., Chen et al.(2023a)Chen, Wang,
Zhou, Qiao, and Dong].

Although Transformer has shown great performance in SISR compared to CNN-based
methods, they still have limitations that need to be dealt with. The use of
dense attention in IPT [Chen et al.(2021a)Chen, Wang, Guo, Xu, Deng, Liu, Ma,
Xu, Xu, and Gao] focuses on short token sequences that come from a dense area
of an image. As a result, the receptive field is restricted due to this
approach. SwinIR [Liang et al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and
Timofte] employs a Swin Transformer as the main backbone, which has a main
drawback is that it limits the receptive fields for extracting information
from global information. Although HAT [Chen et al.(2023a)Chen, Wang, Zhou,
Qiao, and Dong] is state-of-the-art in SISR, it still uses Swin Transformer,
thus limiting the extraction of global information. Therefore, the current
methods still can not fully exploit the potential of Transformer for SISR. On
the other hand, current methods in SISR mainly extract features from the
spatial domain without leveraging features extracted from the frequency
domain, which include valuable information and are beneficial for HR image
reconstruction.

In order to tackle the mentioned drawbacks and unlock more potential of
Transformer for SISR, we propose a novel architecture named Channel-
Partitioned Attention Transformer (CPAT), depicted in Fig. 1. A key component
in our CPAT is the new self-attention mechanism called Channel-Partitioned
Windowed Self-Attention (CPWin-SA) to better capture long-range information
and relationships between distant tokens. In addition, we also design a
Spatial-Frequency Interaction Module (SFIM) to integrate the spatial and
frequency domains to fully exploit the information from feature maps, thereby
boosting the quality of output images. Based on these designs, our method can
extract robust features to aid in reconstruction and achieve significant
improvements compared to the current methods.

Contributions: 1) We propose a novel Channel-Partitioned Windowed Self-
Attention (CPWin-SA), a robust feature extraction for better reconstruction of
images. 2). We design a new Spatial-Frequency Interaction Module (SFIM) to
leverage all features from both spatial and frequency domains that improve the
model’s performance. 3) Our network outperforms the current state-of-the-art
methods for SISR.

##  2 Related Work

Deep Neural Networks for SISR. Dong _et al_ [Dong et al.(2015)Dong, Loy, He,
and Tang] conducted the first study utilizing deep learning in SISR, called
SRCNN, a simple yet effective three-layer CNN for SISR. Following SRCNN, CNNs
have been employed in subsequent studies to enhance SISR performance [Lim et
al.(2017)Lim, Son, Kim, Nah, and Mu Lee, Ledig et al.(2017)Ledig, Theis,
Huszár, Caballero, Cunningham, Acosta, Aitken, Tejani, Totz, Wang, et al.,
Zhang et al.(2018b)Zhang, Tian, Kong, Zhong, and Fu, Kim et al.(2016b)Kim,
Lee, and Lee]. Recently, Transformer has been used in various computer vision
applications, from localization [Rambhatla et al.(2023)Rambhatla, Misra,
Chellappa, and Shrivastava, Zhao et al.(2018)Zhao, Li, Zhao, and Feng] to
deeply understanding images [Jaderberg et al.(2015)Jaderberg, Simonyan,
Zisserman, et al., Sun et al.(2022)Sun, Zhou, Black, and Chandrasekaran, Zhu
et al.(2022)Zhu, Shah, and Chen, Shi et al.(2022)Shi, Jiang, Dai, and Schiele]
to sequence-based networks [Bluche(2016), Miech et al.(2017)Miech, Laptev, and
Sivic, Grechishnikova(2019)]. ViT [Dosovitskiy et al.(2021)Dosovitskiy, Beyer,
Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold,
Gelly, Uszkoreit, and Houlsby] was proposed by Dosovitskiy _et al_ that
processes input images by segmenting them into patches and then projecting
these patches into sequential tokens as input of transformer module and
achieved remarkable results with high-level vision tasks. To reduce the high
computational cost of ViT, Liu _et al_ [Liu et al.(2021)Liu, Lin, Cao, Hu,
Wei, Zhang, Lin, and Guo] proposed a hierarchical transformer called Swin
Transformer, using self-attention over local windows instead of the entire
image like ViT. In the field of low-level vision tasks, such as SISR,
Transformer can also be used as a powerful backbone. IPT [Chen et
al.(2021a)Chen, Wang, Guo, Xu, Deng, Liu, Ma, Xu, Xu, and Gao] leveraged the
pre-trained transformer to improve performance for the image super-resolution
task. SwinIR [Liang et al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and Timofte]
utilized Swin Transformer as a deep feature extractor and achieved impressive
results for image restoration. HAT [Chen et al.(2023a)Chen, Wang, Zhou, Qiao,
and Dong] proposed Hybrid Attention Transformer and Overlapping-cross
Attention, achieving state-of-the-art SISR performance. However, the common
limitation of these works is that they are limited in capturing long-range
dependencies and may miss the connection with the distant tokens. Our proposed
Transformer can handle this problem by enhancing window size in the window-
based attention mechanism while still being efficient for high-resolution
images.

Frequency domain in Computer Vision. The frequency domain is widely used in
digital signal processing [Trider(1978), Saxena and Singh(2005), Kim et
al.(2001)Kim, Kim, Lee, and Lim] and benefits the computer vision domain [He
et al.(2016)He, Chen, and Liu, Gohshi(2015), Cai et al.(2021)Cai, Ding, and
Lu, Wang et al.(2023)Wang, Jiang, Zhong, and Liu], such as image super-
resolution. Cai _et al_ proposed FreqNet [Cai et al.(2021)Cai, Ding, and Lu]
consisting of two main branches: spatial and frequency branches. FreqNet
transforms LR and HR images to the frequency domain using discrete cosine
transform (DCT) [Ahmed et al.(1974)Ahmed, Natarajan, and Rao]. DCT features
combine with spatial features, and then, the inverse DCT (iDCT) [Ahmed et
al.(1974)Ahmed, Natarajan, and Rao] is used to convert the feature maps back
to the spatial domain. FreqNet uses a dual branch and DCT transform from the
beginning, which increases the computational complexity. Wang _et al_ proposed
SFMNet [Wang et al.(2023)Wang, Jiang, Zhong, and Liu] for face super-
resolution (FSR). SFMNet uses Fourier transform [Brigham(1988)] in the
frequency domain to convert spatial features to the frequency domain to
capture the global facial structure and inverse Fourier transform
[Brigham(1988)] to convert frequency features to the spatial domain. Similar
to FreqNet, the frequency branch in SFMNet is too complex and the
computational complexity is high due to it computes spatial-frequency cross-
attention module multiple times to combine spatial and frequency domains. Our
Spatial-Frequency Integrated Module (SFIM) is designed to deal with this
problem. It is simple yet effectively leverages features (textures, edges,
$etc.$) from the frequency domain that may miss extraction in the spatial
domain while the computational complexity is not significantly increasing.

##  3 Methodology

###  3.1 The Overall Architecture

Figure 1: Architecture details. (a) The overall architecture of CPAT. (b)
Structure of Channel-Partitioned Windowed Self-Attention. (c) Structure of
Overlapping Cross-Attention Module. (d) Spatial-Frequency Integrated Module.

The overall architecture of Channel-Partitioned Attention Transformer (CPAT)
is shown in Fig. 1(a), which consists of three parts: Dimensionality Expansion
(DE), Complex Feature Learning (CFL), and Image Reconstruction (IR). We set
the input LR and output SR images as $I_{LR}\in\mathbb{R}^{H\times W\times
C_{in}}$ and $O_{SR}$ $\in\mathbb{R}^{H\times W\times C_{out}}$, where
$C_{in}$ and $C_{out}$ are the channel numbers of the LR and SR images,
respectively. First, Dimensionality Expansion transforms $I_{LR}$ from a low-
dimensional space to a high-dimensional space $I_{DE1},\;I_{DE2}$ as follows:

$I_{DE1}=F_{c3}(I_{LR}),\;I_{DE2}=F_{c2}(I_{LR}),$ (1)

where $I_{DE1}$ and $I_{DE2}$ are the output of DE; $F_{c3}$ is a 3x3 conv,
and $F_{c2}$ is a convolution stage (Convolution - Activation - Convolution).
Next, $I_{DE1}$ goes through Complex Feature Learning, which involves a series
of RWAGs to learn the complex and deep features. Each RWAG consists of several
CPWin-SA modules, an overlapping cross-attention module (OCAM), and a SFIM
module. The output of CFL $O_{CFL}$ is

$O_{CFL}=F_{SFIM}(F_{RWAG}^{L_{1}}(F_{RWAG}^{L_{1}-1}(...(F_{RWAG}^{1}(I_{DE1})+I_{DE2}))+I_{DE2})+I_{DE2}),$
(2)

where $F_{RWAG}^{i},\;F_{SFIM}$ represent the functions of i-th RWAG, SFIM
modules, $i=1,2,...,L_{1}$, and $L_{1}$ is the number of RWAG in CFL. Finally,
$O_{CFL}$ is passed to the Image Reconstruction (IR) to obtain the output
$O_{SR}$. IR includes convolution layers and PixelShuffle [Shi et
al.(2016)Shi, Caballero, Huszar, Totz, Aitken, Bishop, Rueckert, and Wang]. We
set the function of IR as $F_{IR}$, and we then have

$O_{SR}=F_{IR}(O_{CFL}+I_{DE1}),$ (3)

###  3.2 Channel-Partitioned Windowed Self-Attention (CPWin-SA)

Figure 2: Enhanced window strategy and One-Direction Shift Operation in V-EWin
and H-EWin

Channel-Partitioned Windowed Self-Attention (CPWin-SA) is a key component of
our method, described in Fig. 1(b). It consists of three different attention
mechanism types: Vertically Enhanced Window Attention (H-EWin), Horizontally
Enhanced Window Attention (H-EWin), and standard Windowed Multi-head Self-
Attention (W-MSA). We split the input feature maps along the channel dimension
into three equal parts, corresponding to the three attentions that are above
mentioned.

Enhanced Window Self-Attention. This refers to V-EWin and H-EWin. We extend
the squared windows along the vertical and horizontal directions of the input
feature maps, shown in Fig. 2. Specifically, the window size is extended from
$ws\times ws$ (squared window) to $H\times ws$ (V-EWin) or $ws\times W$
(H-EWin), where $ws$ is the window size ($ws<H$, $ws<W$). V-EWin and H-Ewin
enhance attention areas by extending the window size, thereby increasing the
ability to extract global contextual information and relationships between
distant tokens. For simplicity, the mathematical descriptions below are used
for V-EWin, and similar formulations can be applied to H-EWin. With the input
feature $X\in\mathbb{R}^{H\times W\times\frac{C}{3}}$ (after splitting), we
compute attention $N$ times in parallel, where $N$ is the head number. We
partition X into non-overlapping windows of size $H\times ws$ for each
attention head, then calculate the self-attention of the $i$-th window feature
as $X_{i}\in\mathbb{R}^{H\times ws\times\frac{C}{3}}$, $i$=1,…,$\frac{H\times
W}{H\times ws}$ for the $n$-th head,

$\begin{array}[]{l}Y_{i}^{n}=Attention(Q_{i}^{n},\;K_{i}^{n},\;V_{i}^{n})=Attention(X_{i}W_{n}^{Q},\;X_{i}W_{n}^{K},\;X_{i}W_{n}^{V}),\end{array}$
(4)

where $Y_{i}^{n}\in\mathbb{R}^{H\times ws\times D}$ denotes the attention
output of $X_{i}$ in the $n$-th head, $D$=$\frac{C}{3N}$ represents the
channel dimension in each head.
$Q_{i}^{n},\;K_{i}^{n},\;V_{i}^{n}\in\mathbb{R}^{\frac{C}{3}\times D}$ are the
projection matrices of query, key, and value, respectively for $n$-th head. We
use conditional position embedding (CPE) [Chu et al.(2023)Chu, Tian, Zhang,
Wang, and Shen] to add the spatial relationships into the network. The
attention feature $Y^{n}\in\mathbb{R}^{H\times W\times D}$ of $X$ is obtained
by calculating the attention operation on all $X_{i}$, then performing
reshaping and merging them in the order of division. We then concatenate the
output of all heads and combine them with a final weight matrix to achieve the
attention map output of V-EWin.

$V{\operatorname{-}}EWin(x)=Concat(Y^{1},...,Y^{N-1},Y^{N})W^{p},$ (5)

where the $W^{p}\in\mathbb{R}^{\frac{C}{3}\times\frac{C}{3}}$ denotes the
projection matrix for feature fusion.

One-Direction Shift Operation. The window-based self-attention module causes a
lack of information linkage between windows, thereby reducing the modeling
power with distant tokens. We enhance the windows along the height and width
of the feature map, we propose One-Directional Shift Operation instead of two
directions as in Swin Transformer while ensuring the transformer’s modeling
power, detailed in Fig. 2. For V-EWin, we move the windows to the left by a
distance of $\frac{ws}{2}$ pixels, while H-EWin moves downward by also
$\frac{ws}{2}$ pixels. Then, we use a cyclic shift to complete the shift
operation. After computing the attention operation on the shifted feature map,
we then revert this feature map to obtain the feature map, which is in the
original order.

Squared Window Self-Attention. For Squared Window Self-Attention, we utilize
Swin Transformer (W-MSA) [Liu et al.(2021)Liu, Lin, Cao, Hu, Wei, Zhang, Lin,
and Guo]. W-MSA is computed entirely similarly to the Enhanced Window Self-
Attention we presented above. The only difference lies in the window size
being $ws\times ws$ instead of $H\times ws$ or $ws\times W$. Using squared
windows helps to focus on local features. Combining standard (squared) and
enhanced windows and computing self-attention in various window shapes
benefits datasets that contain many texture features in various directions.

Channel-Partitioned Windowed Self-Attention. CPWin-SA consists of three
attention modules (V-EWin, H-EWin, and W-MSA) and an MLP, which includes a
GELU [Hendrycks and Gimpel(2016)] activation function between 2 linear
projection layers. Because Transformer has no inductive bias, we simply use a
depthwise convolution to add inductive bias that aims to improve the
performance of Transformer. A Layer Norm layer is used before the attention
modules and MLP. The entire process of CPWin-SA is as follows:

$\displaystyle\begin{split}X=LayerNorm(X_{in}),\;c=C/3,\end{split}$ (6a)
$\displaystyle\begin{split}X_{1}=V{\operatorname{-}}EWin(X[:,:,:c]),\;X_{2}=H{\operatorname{-}}EWin(X[:,:,c:2c]),\;X_{3}=W{\operatorname{-}}MSA(X[:,:,2c:]),\end{split}$
(6b)
$\displaystyle\begin{split}\hat{X}=Concat(X_{1},\;X_{2},\;X_{3})+X_{in}+DWConv(V),\end{split}$
(6c)
$\displaystyle\begin{split}X_{out}=MLP(LayerNorm(\hat{X}))+\hat{X},\end{split}$
(6d)

where $X_{in}$, $X_{out}$ and $C$ are CPWin-SA’s input, output features, and
the channel numbers; DWConv and V are depthwise convolution and $value$
matrix. Shift Operation uses two consecutive Transformer modules to increase
the interaction among non-overlapping windows. The computational complexity of
global MSA (self-attention is computed on the full feature map) and V-EWin are

$\displaystyle\begin{split}\mathcal{O}(Global-
MSA)=4HW(C/3)^{2}+2(HW)^{2}(C/3)),\end{split}$ (7a)
$\displaystyle\begin{split}\mathcal{O}(V{\operatorname{-}}EWin)=4HW(C/3)^{2}+2H^{2}Wws(C/3)),\end{split}$
(7b)

Assuming H=W (squared image) and $ws\ll H,\;C\ll H$, the computational
complexity of $V{\operatorname{-}}EWin$ is $\mathcal{O}(H^{2}Wws(C/3))$ =
$\mathcal{O}(H^{3})$ whereas $Global-MSA$ is $\mathcal{O}((HW)^{2}(C/3))$ =
$\mathcal{O}(H^{4})$. Therefore, our proposed Transformer can be applied to
high-resolution input images.

###  3.3 Overlapping Cross-Attention Module (OCAM)

OCAM [Chen et al.(2023a)Chen, Wang, Zhou, Qiao, and Dong] enhances the
connections between windows by partitioning feature maps into overlapping
windows and calculating self-attention on each window to improve the
performance of the network. The structure of OCAM is depicted in Fig. 1(c).
Specifically, for $X_{Q}$, $X_{K}$, $X_{v}$ from the input feature $X$,
$X_{Q}$ is divided into non-overlapping windows of size $M\times M$, with
$\frac{HW}{M^{2}}$ being the total number of windows. $X_{K}$ and $X_{V}$ are
unfolded to $\frac{HW}{M^{2}}$ overlapping windows of size $M_{o}\times M_{o}$
($M_{o}>M$),

$M_{o}=(1+\alpha)\times M,$ (8)

where $\alpha$ is the overlapping ratio is 0.5. Then, self-attention is
computed within each windows as in Eq. 4.

###  3.4 Spatial-Frequency Interaction Module (SFIM)

Spatial features may lack frequency information and fine-grained details that
are important for HR image reconstruction. To address these issues, we
carefully design a Spatial-Frequency Interaction Module (SFIM) shown in Fig.
1(d) to leverage the spatial and frequency domain features. SFIM consists of
two branches: spatial and frequency branches. Spatial branch is helpful in
extracting local spatial features. We denote the input, output of SFIM, and
output of spatial as $I_{SFIM}$, $O_{SFIM}$, $O_{SB}$, respectively, then
spatial branch is represented as follows:

$\displaystyle\begin{split}O_{SB1}=F_{A}(F_{c1}(I_{SFIM})),\end{split}$ (9a)
$\displaystyle\begin{split}O_{SB}=Concat(F_{c3}(O_{SB1}[:,:C/2,:,:]),\;F_{c3}(O_{SB1}[:,C/2:,:,:]))+O_{SB1},\end{split}$
(9b)

where $F_{c1}$, $F_{c3}$, $F_{A}$ and $C$ are 1x1 conv, 3x3 conv, LeakReLU
[Maas et al.(2013)Maas, Hannun, Ng, et al.], and number channel, respectively.
Frequency branch is used for capturing global structure and frequency
information. To convert spatial features into the frequency domain, we utilize
the Fast Fourier Transform (FFT) [Cooley and Tukey(1965)], and use inverse FFT
(iFFT) [Cooley and Tukey(1965)] to convert frequency features back into the
spatial domain. The FFT can capture global patterns, structures, and frequency
information in an image that might be less apparent in the spatial domain and
are essential for SR reconstruction. We denote the output of frequency branch
as $O_{FB}$, then frequency branch can be represented as follows:

$\displaystyle\begin{split}O_{FB1}=F_{A}(F_{c3}(I_{SFIM})),\end{split}$ (10a)
$\displaystyle\begin{split}O_{FB}=F_{c1}(F_{FD}(F_{c3}(O_{FB1}))+O_{FB1}),\end{split}$
(10b)

where $F_{FD}$ is $Freq\;Domain$. At the end, we combine the outputs of
spatial and frequency domains to obtain the output feature of SFIM is
$O_{SFIM}$

$O_{SFIM}=F_{c1}(Concat([O_{SB},O_{FB}])),\vspace{-6mm}$ (11)

##  4 Experiments

###  4.1 Experimental Settings

We use DF2K (DIV2K [Agustsson and Timofte(2017)]+Flicker2K [Lim et
al.(2017)Lim, Son, Kim, Nah, and Mu Lee]) dataset as the training set for a
fair comparison with other methods. For evaluating our model, we use five
benchmark datasets: Set5 [Bevilacqua et al.(2012)Bevilacqua, Roumy, Guillemot,
and Alberi-Morel], Set14 [Zeyde et al.(2010)Zeyde, Elad, and Protter], BSD100
[Martin et al.(2001)Martin, Fowlkes, Tal, and Malik], Urban100 [Huang et
al.(2015)Huang, Singh, and Ahuja], and Manga109 [Matsui et al.(2016)Matsui,
Ito, Aramaki, Fujimoto, Ogawa, Yamasaki, and Aizawa]. Furthermore, we use peak
signal-to-noise ratio (PSNR) [Wang et al.(2004)Wang, Bovik, Sheikh, and
Simoncelli] and the structural similarity index measure (SSIM) [Wang et
al.(2004)Wang, Bovik, Sheikh, and Simoncelli] as quantitative metrics, which
are calculated on the Y channel in YCbCr Space. For the structure of network,
we set the number of RWAG and SPWin-SA to 6, the channel number to 180, and
the window size is set to 16. The overlapping ratio in OCAM remains at 0.5 as
in [Chen et al.(2023a)Chen, Wang, Zhou, Qiao, and Dong]. We apply the self-
ensemble strategy similarly to [Timofte et al.(2016)Timofte, Rothe, and Van
Gool] in testing that we call CPAT† as in [Zhang et al.(2018a)Zhang, Li, Li,
Wang, Zhong, and Fu, Liang et al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and
Timofte, Chen et al.(2023b)Chen, Zhang, Gu, Kong, Yang, and Yu]. We use a
patch size of $64\times 64$ and a batch size of 32 during the training. We
simply use L1 loss, and Adam optimizer [Kingma and Ba(2014)] to optimize
models with 500K iterations.

###  4.2 Ablation Study

Following [Chen et al.(2022b)Chen, Zhang, Gu, Kong, Yuan, et al., Chen et
al.(2023b)Chen, Zhang, Gu, Kong, Yang, and Yu], we train x2 SR model on DF2K
(DIV2K [Agustsson and Timofte(2017)]+Flicker2K [Lim et al.(2017)Lim, Son, Kim,
Nah, and Mu Lee]), and test on Urban100 [Huang et al.(2015)Huang, Singh, and
Ahuja] for all experiments in this section. FLOPs are calculated on a 256x256
HR image. Results are reported in Tab. 3, 3, 3, 4, and the better results are
shown in bold.

Structure | PSNR | SSIM | FLOPs
---|---|---|---
Squared wins | 34.03 | 0.9438 | 324.8G
Enhanced wins | 34.26 | 0.9448 | 329.0G

Table 1: Effect of the enhanced windows.

Structure | PSNR | SSIM | FLOPs
---|---|---|---
w/o shift | 34.13 | 0.9440 | 329.0G
w/ shift | 34.26 | 0.9448 | 329.0G

Table 2: Effect of One-Direction Shift Operation.

Structure | PSNR | SSIM | FLOPs
---|---|---|---
w/o SFIM | 34.14 | 0.9442 | 256.2G
w/ SFIM | 34.26 | 0.9448 | 329.0G

Table 3: Effect of SFIM module.

Effect of the enhanced window. Tab. 3 shows the effectiveness of the enhanced
window in V-EWin and H-EWin instead of the squared window in Swin Transformer.
PSNR, when using the enhanced window is 34.26dB compared to 34.03dB with the
squared window. These results show that the standard window-based transformer
has yet to fully utilize its potential for SISR, whereas our method shows
significant improvement and the computation complexity does not increase much
(FLOPs is 329.0G compared to 324.8G). These results also indicate that
transformer-based methods remain a promising research direction in SISR.

Effect of shift operation. Tab. 3 shows the effect of One-Direction Shifted
Operation. When One-Direction Shift Operation is applied, PSNR value is
34.26dB, which is higher compared to 34.13dB without this operation.
Additionally, SSIM also increases from 0.9440 to 0.9448. Our shift operation
is helpful in getting correlation attention between different window
partitions, thereby enhancing the performance of CPWin-SA and our network in
general.

Structure | PSNR | SSIM | FLOPs
---|---|---|---
w/o Freq Domain | 34.03 | 0.9434 | 321.4G
w/ Freq Domain | 34.26 | 0.9448 | 329.0G

Table 4: Effect of Freq Domain module in SFIM

Effect of SFIM. Tab. 3 shows the effectiveness of SFIM. We replace SFIM module
with a 3x3 conv (we call it "w/o SFIM") and see how it affects PSNR/SSIM. PSNR
when having SFIM is 34.26dB, while PSNR when not having SFIM is 34.14dB. By
leveraging information from the frequency domain, SFIM boosts the network’s
performance compared to using convolution only, which provides spatial
features.

Effect of Freq Domain module on SFIM’s effectiveness. To show the
effectiveness of frequency features, we conduct an experiment by removing the
Freq Domain module from the SFIM module. This means that the SFIM module will
solely work with the spatial domain. The results are reported in Tab. 4, which
is evident that SFIM operates significantly less effectively when the Freq
Domain module is removed. By combining both spatial and frequency domains in
the SFIM module, comprehensive features are extracted, thereby enhancing the
performance of image reconstruction. Specifically, SFIM with Freq Domain
achieves 34.26dB, whereas, without Freq Domain, it achieves 34.03dB. This also
demonstrates the potential of combining spatial and frequency domains for
other vision tasks.

Method | Scale |  Set5 [Bevilacqua et al.(2012)Bevilacqua, Roumy, Guillemot, and Alberi-Morel] |  Set14 [Zeyde et al.(2010)Zeyde, Elad, and Protter] |  BSD100 [Martin et al.(2001)Martin, Fowlkes, Tal, and Malik] |  Urban100 [Huang et al.(2015)Huang, Singh, and Ahuja] |  Manga109 [Matsui et al.(2016)Matsui, Ito, Aramaki, Fujimoto, Ogawa, Yamasaki, and Aizawa]
---|---|---|---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
EDSR [Lim et al.(2017)Lim, Son, Kim, Nah, and Mu Lee] | x2 | 38.11 | 0.9601 | 33.92 | 0.9195 | 32.32 | 0.9013 | 32.93 | 0.9351 | 39.10 | 0.9773
RCAN [Zhang et al.(2018a)Zhang, Li, Li, Wang, Zhong, and Fu] | 38.27 | 0.9614 | 34.12 | 0.9216 | 32.41 | 0.9027 | 33.34 | 0.9384 | 39.44 | 0.9786
NLSA [Mei et al.(2021)Mei, Fan, and Zhou] | 38.34 | 0.9618 | 34.08 | 0.9231 | 32.43 | 0.9027 | 33.42 | 0.9394 | 39.59 | 0.9789
ELAN [Zhang et al.(2022)Zhang, Zeng, Guo, and Zhang] | 38.36 | 0.9620 | 34.20 | 0.9228 | 32.45 | 0.9030 | 33.44 | 0.9391 | 39.62 | 0.9793
IPT∗ [Chen et al.(2021a)Chen, Wang, Guo, Xu, Deng, Liu, Ma, Xu, Xu, and Gao] | 38.37 | - | 34.43 | - | 32.48 | - | 33.76 | - | - | -
RCAN-it [Lin et al.(2022)Lin, Garg, Banerjee, Magid, Sun, Zhang, Van Gool, Wei, and Pfister] | 38.37 | 0.9620 | 34.49 | 0.9250 | 32.48 | 0.9034 | 33.62 | 0.9410 | 39.88 | 0.9799
SwinIR [Liang et al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and Timofte] | 38.42 | 0.9623 | 34.46 | 0.9250 | 32.53 | 0.9041 | 33.81 | 0.9427 | 39.92 | 0.9797
CAT-A [Chen et al.(2022b)Chen, Zhang, Gu, Kong, Yuan, et al.] | 38.51 | 0.9626 | 34.78 | 0.9265 | 32.59 | 0.9047 | 34.26 | 0.9440 | 40.10 | 0.9805
DAT [Chen et al.(2023b)Chen, Zhang, Gu, Kong, Yang, and Yu] | 38.58 | 0.9629 | 34.81 | 0.9272 | 32.61 | 0.9051 | 34.37 | 0.9458 | 40.33 | 0.9807
HAT [Chen et al.(2023a)Chen, Wang, Zhou, Qiao, and Dong] | 38.63 | 0.9630 | 34.86 | 0.9274 | 32.62 | 0.9053 | 34.45 | 0.9466 | 40.26 | 0.9809
CPAT (Ours) | 38.68 | 0.9633 | 34.91 | 0.9277 | 32.64 | 0.9056 | 34.76 | 0.9481 | 40.48 | 0.9814
CPAT† (Ours) | 38.72 | 0.9635 | 34.97 | 0.9280 | 32.66 | 0.9058 | 34.89 | 0.9487 | 40.59 | 0.9816
EDSR [Lim et al.(2017)Lim, Son, Kim, Nah, and Mu Lee] | x3 | 34.65 | 0.9280 | 30.52 | 0.8462 | 29.25 | 0.8093 | 28.80 | 0.8653 | 34.17 | 0.9476
RCAN [Zhang et al.(2018a)Zhang, Li, Li, Wang, Zhong, and Fu] | 34.74 | 0.9299 | 30.65 | 0.8482 | 29.32 | 0.8111 | 29.09 | 0.8702 | 34.44 | 0.9499
NLSA [Mei et al.(2021)Mei, Fan, and Zhou] | 34.85 | 0.9306 | 30.70 | 0.8485 | 29.34 | 0.8117 | 29.25 | 0.8726 | 34.57 | 0.9508
ELAN [Zhang et al.(2022)Zhang, Zeng, Guo, and Zhang] | 34.90 | 0.9313 | 30.80 | 0.8504 | 29.38 | 0.8124 | 29.32 | 0.8745 | 34.73 | 0.9517
IPT∗ [Chen et al.(2021a)Chen, Wang, Guo, Xu, Deng, Liu, Ma, Xu, Xu, and Gao] | 34.81 | - | 30.85 | - | 29.38 | - | 29.49 | - | - | -
RCAN-it [Lin et al.(2022)Lin, Garg, Banerjee, Magid, Sun, Zhang, Van Gool, Wei, and Pfister] | 34.86 | 0.9308 | 30.76 | 0.8505 | 29.39 | 0.8125 | 29.38 | 0.8755 | 34.92 | 0.9520
SwinIR [Liang et al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and Timofte] | 34.97 | 0.9318 | 30.93 | 0.8534 | 29.46 | 0.8145 | 29.75 | 0.8826 | 35.12 | 0.9537
CAT-A [Chen et al.(2022b)Chen, Zhang, Gu, Kong, Yuan, et al.] | 35.06 | 0.9326 | 31.04 | 0.8538 | 29.52 | 0.8160 | 30.12 | 0.8862 | 35.38 | 0.9546
DAT [Chen et al.(2023b)Chen, Zhang, Gu, Kong, Yang, and Yu] | 35.16 | 0.9331 | 31.11 | 0.8550 | 29.55 | 0.8169 | 30.18 | 0.8886 | 35.59 | 0.9554
HAT [Chen et al.(2023a)Chen, Wang, Zhou, Qiao, and Dong] | 35.07 | 0.9329 | 31.08 | 0.8555 | 29.54 | 0.8167 | 30.23 | 0.8896 | 35.53 | 0.9552
CPAT (Ours) | 35.16 | 0.9334 | 31.15 | 0.8557 | 29.56 | 0.8174 | 30.52 | 0.8923 | 35.66 | 0.9559
CPAT† (Ours) | 35.19 | 0.9335 | 31.19 | 0.8559 | 29.59 | 0.8177 | 30.63 | 0.8934 | 35.77 | 0.9563
EDSR [Lim et al.(2017)Lim, Son, Kim, Nah, and Mu Lee] | x4 | 32.46 | 0.8968 | 28.80 | 0.7876 | 27.71 | 0.7420 | 26.64 | 0.8033 | 31.02 | 0.9148
RCAN [Zhang et al.(2018a)Zhang, Li, Li, Wang, Zhong, and Fu] | 32.63 | 0.9002 | 28.87 | 0.7889 | 27.77 | 0.7436 | 26.82 | 0.8087 | 31.22 | 0.9173
NLSA [Mei et al.(2021)Mei, Fan, and Zhou] | 32.59 | 0.9000 | 28.87 | 0.7891 | 27.78 | 0.7444 | 26.96 | 0.8109 | 31.27 | 0.9184
ELAN [Zhang et al.(2022)Zhang, Zeng, Guo, and Zhang] | 32.75 | 0.9022 | 28.96 | 0.7914 | 27.83 | 0.7459 | 27.13 | 0.8167 | 31.68 | 0.9226
IPT∗ [Chen et al.(2021a)Chen, Wang, Guo, Xu, Deng, Liu, Ma, Xu, Xu, and Gao] | 32.64 | - | 29.01 | - | 27.82 | - | 27.26 | - | - | -
RCAN-it [Lin et al.(2022)Lin, Garg, Banerjee, Magid, Sun, Zhang, Van Gool, Wei, and Pfister] | 32.69 | 0.9007 | 28.99 | 0.7922 | 27.87 | 0.7459 | 27.16 | 0.8168 | 31.78 | 0.9217
SwinIR [Liang et al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and Timofte] | 32.92 | 0.9044 | 29.09 | 0.7950 | 27.92 | 0.7489 | 27.45 | 0.8254 | 32.03 | 0.9260
CAT-A [Chen et al.(2022b)Chen, Zhang, Gu, Kong, Yuan, et al.] | 33.08 | 0.9052 | 29.18 | 0.7960 | 27.99 | 0.7510 | 27.89 | 0.8339 | 32.39 | 0.9285
DAT [Chen et al.(2023b)Chen, Zhang, Gu, Kong, Yang, and Yu] | 33.08 | 0.9055 | 29.23 | 0.7973 | 28.00 | 0.7515 | 27.87 | 0.8343 | 32.51 | 0.9291
HAT [Chen et al.(2023a)Chen, Wang, Zhou, Qiao, and Dong] | 33.04 | 0.9056 | 29.23 | 0.7973 | 28.00 | 0.7517 | 27.97 | 0.8368 | 32.48 | 0.9292
CPAT (Ours) | 33.19 | 0.9069 | 29.34 | 0.7991 | 28.04 | 0.7527 | 28.22 | 0.8408 | 32.69 | 0.9309
CPAT† (Ours) | 33.24 | 0.9071 | 29.36 | 0.7996 | 28.06 | 0.7532 | 28.33 | 0.8425 | 32.85 | 0.9318

Table 5: Quantitative comparison with state-of-the-art methods. The best,
second-best, and third-best results are marked in red, blue, and green colors,
respectively. “$\dagger$” indicates that self-ensemble is used. IPT∗ [Chen et
al.(2021a)Chen, Wang, Guo, Xu, Deng, Liu, Ma, Xu, Xu, and Gao] is trained on
ImageNet.

Figure 3: LAM [Gu and Dong(2021)] and DI [Gu and Dong(2021)] comparison
results.

###  4.3 Comparison with State-of-the-Art Methods

Quantitative results. Tab. 5 reports the quantitative comparison of CPAT with
different state-of-the-art methods including EDSR [Lim et al.(2017)Lim, Son,
Kim, Nah, and Mu Lee], RCAN [Zhang et al.(2018a)Zhang, Li, Li, Wang, Zhong,
and Fu], NLSA [Mei et al.(2021)Mei, Fan, and Zhou], ELAN [Zhang et
al.(2022)Zhang, Zeng, Guo, and Zhang], RCAN-it [Lin et al.(2022)Lin, Garg,
Banerjee, Magid, Sun, Zhang, Van Gool, Wei, and Pfister], SwinIR [Liang et
al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and Timofte], CAT-A [Chen et
al.(2022b)Chen, Zhang, Gu, Kong, Yuan, et al.], DAT [Chen et al.(2023b)Chen,
Zhang, Gu, Kong, Yang, and Yu], and HAT [Chen et al.(2023a)Chen, Wang, Zhou,
Qiao, and Dong]. Our method surpasses the current methods on all benchmark
datasets with all scales. The highest increase achieved is 0.31dB on Urban100
on x2 SR when compared with HAT. CPAT improves by more than 0.7dB compared to
SwinIR, which uses Swin Transformer as its central backbone for all scales.
With the self-ensemble strategy in testing, CPAT† performs better than CPAT,
but the inference time is much longer and not helpful for high-resolution
input images. All quantitative results demonstrate that enhancing the windows
along the height and width of feature maps instead of using the squared
windows when computing attention in CPWin-SA and leveraging frequency features
in SFIM are very effective for improving the quality of the SR image.

Figure 4: Qualitative comparison (x4 SR). The patch images being compared are
the green boxes in the HR images. PSNR/SSIM is also computed correspondingly
on these patches to demonstrate the improvement of our method.

Qualitative results. Local Attribution Map (LAM) [Gu and Dong(2021)] and
Diffusion Index (DI) [Gu and Dong(2021)] comparisons are shown in Fig. 3. LAM
emphasizes the significance of pixels in the LR image during upscaling of the
patches marked with green boxes. DI is indicative of the wider range of pixels
utilized. A higher DI indicates a wider range of pixels in upscaling images.
LAM and DI results show the superiority of our method over other methods. The
visual comparison is shown in Fig. 4 with $"img\\_16"$, $"img\\_49"$ from
Urban100, and $"UchuKigekiM774"$ from Manga109. Our method can enhance the
details of the LR image more clearly, with less blur, and higher PSNR/SSIM
compared to other methods. All qualitative results show the effectiveness of
our method for SISR. More details on qualitative results, self-ensemble
strategy, and lightweight version of CPAT can be found in the $supp.$ file.

##  5 Conclusions

In this study, we propose a novel Channel-Partitioned Attention Transformer
(CPAT) for SISR. This Transformer consists of V-EWin, H-EWin, and W-MSA
attentions. EWin and H-EWin enhance windows along the height and width of
input features to better capture long-range dependencies and relationships
between distant tokens. We also use squared window-based attention in CPAT,
which focuses on local features. We calculate self-attention in various window
shapes to apply to datasets that contain texture features in various
directions (_e.g_., Urban100 [Huang et al.(2015)Huang, Singh, and Ahuja]).
Additionally, we propose Spatial-Frequency Interaction Module (SFIM), which is
simple yet effectively leverages features (patterns, textures, edges, $etc.$)
from the frequency domain that might be less apparent in the spatial domain.
Integrating frequency spatial features helps to achieve comprehensive
information from feature maps that is important for HR image reconstruction.
Based on the proposals above, our method outperforms the current methods in
both quantitative and qualitative results.

## References

  * [Agustsson and Timofte(2017)] E. Agustsson and R. Timofte.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 1122–1131, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society.  10.1109/CVPRW.2017.150.  URL https://doi.ieeecomputersociety.org/10.1109/CVPRW.2017.150.
  * [Ahmed et al.(1974)Ahmed, Natarajan, and Rao] N. Ahmed, T. Natarajan, and K.R. Rao.  Discrete cosine transform.  _IEEE Transactions on Computers_ , C-23(1):90–93, 1974.  10.1109/T-C.1974.223784.
  * [Bevilacqua et al.(2012)Bevilacqua, Roumy, Guillemot, and Alberi-Morel] Marco Bevilacqua, Aline Roumy, Christine M. Guillemot, and Marie-Line Alberi-Morel.  Low-complexity single-image super-resolution based on nonnegative neighbor embedding.  In _British Machine Vision Conference_ , 2012.  URL https://api.semanticscholar.org/CorpusID:5250573.
  * [Bluche(2016)] Théodore Bluche.  Joint line segmentation and transcription for end-to-end handwritten paragraph recognition.  _Advances in neural information processing systems_ , 29, 2016.
  * [Brigham(1988)] E Oran Brigham.  _The fast Fourier transform and its applications_.  Prentice-Hall, Inc., 1988.
  * [Cai et al.(2021)Cai, Ding, and Lu] Runyuan Cai, Yue Ding, and Hongtao Lu.  Freqnet: A frequency-domain image super-resolution network with dicrete cosine transform.  _ArXiv_ , abs/2111.10800, 2021.  URL https://api.semanticscholar.org/CorpusID:244478402.
  * [Chen et al.(2021a)Chen, Wang, Guo, Xu, Deng, Liu, Ma, Xu, Xu, and Gao] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao.  Pre-trained image processing transformer.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12299–12310, 2021a.
  * [Chen et al.(2021b)Chen, Lu, Yu, Luo, Adeli, Wang, Lu, Yuille, and Zhou] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation.  _arXiv preprint arXiv:2102.04306_ , 2021b.
  * [Chen et al.(2022a)Chen, Du, Yang, Beyer, Zhai, Lin, Chen, Li, Song, Wang, and Zhou] Wuyang Chen, Xianzhi Du, Fan Yang, Lucas Beyer, Xiaohua Zhai, Tsung-Yi Lin, Huizhong Chen, Jing Li, Xiaodan Song, Zhangyang Wang, and Denny Zhou.  A simple single-scale vision transformer for object detection and instance segmentation.  In _Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part X_ , page 711–727, Berlin, Heidelberg, 2022a. Springer-Verlag.  ISBN 978-3-031-20079-3.  10.1007/978-3-031-20080-9_41.  URL https://doi.org/10.1007/978-3-031-20080-9_41.
  * [Chen et al.(2023a)Chen, Wang, Zhou, Qiao, and Dong] Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong.  Activating more pixels in image super-resolution transformer.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 22367–22377, 2023a.
  * [Chen et al.(2022b)Chen, Zhang, Gu, Kong, Yuan, et al.] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xin Yuan, et al.  Cross aggregation transformer for image restoration.  _Advances in Neural Information Processing Systems_ , 35:25478–25490, 2022b.
  * [Chen et al.(2023b)Chen, Zhang, Gu, Kong, Yang, and Yu] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu.  Dual aggregation transformer for image super-resolution.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 12312–12321, 2023b.
  * [Chu et al.(2023)Chu, Tian, Zhang, Wang, and Shen] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen.  Conditional positional encodings for vision transformers.  In _ICLR 2023_ , 2023.  URL https://openreview.net/forum?id=3KWnuT-R1bh.
  * [Cooley and Tukey(1965)] James W Cooley and John W Tukey.  An algorithm for the machine calculation of complex fourier series.  _Mathematics of computation_ , 19(90):297–301, 1965.
  * [Dong et al.(2015)Dong, Loy, He, and Tang] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(2):295–307, 2015.
  * [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_ , 2021.
  * [Gohshi(2015)] Seiichi Gohshi.  Frequency domain analysis for super resolution image reconstruction and its limitations.  In _2015 10th Asia-Pacific Symposium on Information and Telecommunication Technologies (APSITT)_ , pages 1–3, 2015.  10.1109/APSITT.2015.7217088.
  * [Grechishnikova(2019)] Daria Grechishnikova.  Transformer neural network for protein specific de novo drug generation as machine translation problem.  _bioRxiv_ , 2019.  10.1101/863415.  URL https://www.biorxiv.org/content/early/2019/12/03/863415.
  * [Gu and Dong(2021)] Jinjin Gu and Chao Dong.  Interpreting super-resolution networks with local attribution maps.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9199–9208, 2021.
  * [He et al.(2016)He, Chen, and Liu] Chao He, Zhenxue Chen, and Chengyun Liu.  Salient object detection via images frequency domain analyzing.  _Signal, Image and Video Processing_ , 10:1295–1302, 2016.
  * [He et al.(2020)He, Liao, Tavakoli, Yang, Rosenhahn, and Pugeault] Sen He, Wentong Liao, Hamed R Tavakoli, Michael Yang, Bodo Rosenhahn, and Nicolas Pugeault.  Image captioning through image transformer.  In _Proceedings of the Asian conference on computer vision_ , 2020.
  * [Hendrycks and Gimpel(2016)] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * [Huang et al.(2015)Huang, Singh, and Ahuja] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja.  Single image super-resolution from transformed self-exemplars.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5197–5206, 2015.  10.1109/CVPR.2015.7299156.
  * [Huang et al.(2023)Huang, Wang, Wei, Huang, Shi, Liu, and Huang] Z. Huang, X. Wang, Y. Wei, L. Huang, H. Shi, W. Liu, and T. S. Huang.  Ccnet: Criss-cross attention for semantic segmentation.  _IEEE Transactions on Pattern Analysis & Machine Intelligence_, 45(06):6896–6908, jun 2023.  ISSN 1939-3539.  10.1109/TPAMI.2020.3007032.
  * [Jaderberg et al.(2015)Jaderberg, Simonyan, Zisserman, et al.] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  _Advances in neural information processing systems_ , 28, 2015.
  * [Kim et al.(2016a)Kim, Lee, and Lee] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1646–1654, 2016a.
  * [Kim et al.(2016b)Kim, Lee, and Lee] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Deeply-recursive convolutional network for image super-resolution.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1637–1645, 2016b.
  * [Kim et al.(2001)Kim, Kim, Lee, and Lim] JM Kim, SH Kim, DJ Lee, and HS Lim.  Signal processing using fourier & wavelet transform for pulse oximetry.  In _Technical Digest. CLEO/Pacific Rim 2001. 4th Pacific Rim Conference on Lasers and Electro-Optics (Cat. No. 01TH8557)_ , volume 2, pages II–II. IEEE, 2001.
  * [Kingma and Ba(2014)] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2014.  URL https://api.semanticscholar.org/CorpusID:6628106.
  * [Ledig et al.(2017)Ledig, Theis, Huszár, Caballero, Cunningham, Acosta, Aitken, Tejani, Totz, Wang, et al.] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4681–4690, 2017.
  * [Li et al.(2018)Li, Fang, Mei, and Zhang] Juncheng Li, Faming Fang, Kangfu Mei, and Guixu Zhang.  Multi-scale residual network for image super-resolution.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [Liang et al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and Timofte] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1833–1844, 2021.
  * [Lim et al.(2017)Lim, Son, Kim, Nah, and Mu Lee] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 136–144, 2017.
  * [Lin et al.(2022)Lin, Garg, Banerjee, Magid, Sun, Zhang, Van Gool, Wei, and Pfister] Zudi Lin, Prateek Garg, Atmadeep Banerjee, Salma Abdel Magid, Deqing Sun, Yulun Zhang, Luc Van Gool, Donglai Wei, and Hanspeter Pfister.  Revisiting rcan: Improved training for image super-resolution.  _arXiv preprint arXiv:2201.11279_ , 2022.
  * [Liu et al.(2021)Liu, Lin, Cao, Hu, Wei, Zhang, Lin, and Guo] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * [Maas et al.(2013)Maas, Hannun, Ng, et al.] Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al.  Rectifier nonlinearities improve neural network acoustic models.  In _Proc. icml_ , volume 30, page 3. Atlanta, GA, 2013.
  * [Martin et al.(2001)Martin, Fowlkes, Tal, and Malik] D. Martin, C. Fowlkes, D. Tal, and J. Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , volume 2, pages 416–423 vol.2, 2001.  10.1109/ICCV.2001.937655.
  * [Matsui et al.(2016)Matsui, Ito, Aramaki, Fujimoto, Ogawa, Yamasaki, and Aizawa] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Sketch-based manga retrieval using manga109 dataset.  _Multimedia Tools and Applications_ , 76(20):21811–21838, November 2016.  ISSN 1573-7721.  10.1007/s11042-016-4020-z.  URL http://dx.doi.org/10.1007/s11042-016-4020-z.
  * [Mei et al.(2021)Mei, Fan, and Zhou] Yiqun Mei, Yuchen Fan, and Yuqian Zhou.  Image super-resolution with non-local sparse attention.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3516–3525, 2021.  10.1109/CVPR46437.2021.00352.
  * [Miech et al.(2017)Miech, Laptev, and Sivic] Antoine Miech, Ivan Laptev, and Josef Sivic.  Learnable pooling with context gating for video classification.  _arXiv:1706.06905_ , 2017.
  * [Rambhatla et al.(2023)Rambhatla, Misra, Chellappa, and Shrivastava] Sai Saketh Rambhatla, Ishan Misra, Rama Chellappa, and Abhinav Shrivastava.  Most: Multiple object localization with self-supervised transformers for object discovery.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15823–15834, 2023.
  * [Sargent et al.(2023)Sargent, Koh, Zhang, Chang, Herrmann, Srinivasan, Wu, and Sun] Kyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang, Charles Herrmann, Pratul Srinivasan, Jiajun Wu, and Deqing Sun.  Vq3d: Learning a 3d-aware generative model on imagenet.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4240–4250, 2023.
  * [Saxena and Singh(2005)] Rajiv Saxena and Kulbir Singh.  Fractional fourier transform: A novel tool for signal processing.  _Journal of the Indian Institute of Science_ , 85(1):11, 2005.
  * [Shi et al.(2022)Shi, Jiang, Dai, and Schiele] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele.  Motion transformer with global intention localization and local movement refinement.  _Advances in Neural Information Processing Systems_ , 35:6531–6543, 2022.
  * [Shi et al.(2016)Shi, Caballero, Huszar, Totz, Aitken, Bishop, Rueckert, and Wang] W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1874–1883, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society.  10.1109/CVPR.2016.207.  URL https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.207.
  * [Sun et al.(2022)Sun, Zhou, Black, and Chandrasekaran] Jiankai Sun, Bolei Zhou, Michael J Black, and Arjun Chandrasekaran.  Locate: End-to-end localization of actions in 3d with transformers.  _arXiv preprint arXiv:2203.10719_ , 2022.
  * [Timofte et al.(2016)Timofte, Rothe, and Van Gool] Radu Timofte, Rasmus Rothe, and Luc Van Gool.  Seven ways to improve example-based single image super resolution.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1865–1873, 2016.
  * [Tran et al.(2022)Tran, Nguyen, Pham, and Tran] Dinh-Phu Tran, Quoc-Anh Nguyen, Van-Truong Pham, and Thi-Thao Tran.  Trans2unet: Neural fusion for nuclei semantic segmentation.  In _11th International Conference on Control, Automation and Information Sciences, ICCAIS 2022, Hanoi, Vietnam, November 21-24, 2022_ , pages 583–588. IEEE, 2022.  10.1109/ICCAIS56082.2022.9990159.  URL https://doi.org/10.1109/ICCAIS56082.2022.9990159.
  * [Trider(1978)] R Trider.  A fast fourier transform (fft) based sonar signal processor.  _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , 26(1):15–20, 1978.
  * [Wang et al.(2023)Wang, Jiang, Zhong, and Liu] Chenyang Wang, Junjun Jiang, Zhiwei Zhong, and Xianming Liu.  Spatial-frequency mutual learning for face super-resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 22356–22366, June 2023.
  * [Wang et al.(2022)Wang, Xu, and Sun] Yiyu Wang, Jungang Xu, and Yingfei Sun.  End-to-end transformer based model for image captioning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 2585–2594, 2022.
  * [Wang et al.(2004)Wang, Bovik, Sheikh, and Simoncelli] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * [Zeyde et al.(2010)Zeyde, Elad, and Protter] Roman Zeyde, Michael Elad, and Matan Protter.  On single image scale-up using sparse-representations.  volume 6920, pages 711–730, 06 2010.  ISBN 978-3-642-27412-1.  10.1007/978-3-642-27413-8_47.
  * [Zhang et al.(2022)Zhang, Zeng, Guo, and Zhang] Xindong Zhang, Hui Zeng, Shi Guo, and Lei Zhang.  Efficient long-range attention network for image super-resolution.  In _European conference on computer vision_ , pages 649–667. Springer, 2022.
  * [Zhang et al.(2018a)Zhang, Li, Li, Wang, Zhong, and Fu] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 286–301, 2018a.
  * [Zhang et al.(2018b)Zhang, Tian, Kong, Zhong, and Fu] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual dense network for image super-resolution.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2472–2481, 2018b.
  * [Zhao et al.(2018)Zhao, Li, Zhao, and Feng] Fang Zhao, Jianshu Li, Jian Zhao, and Jiashi Feng.  Weakly supervised phrase localization with multi-scale anchored transformer network.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5696–5705, 2018.
  * [Zhu et al.(2022)Zhu, Shah, and Chen] Sijie Zhu, Mubarak Shah, and Chen Chen.  Transgeo: Transformer is all you need for cross-view image geo-localization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1162–1171, 2022.