# Generating fMRI volumes from T1-weighted volumes using 3D CycleGAN

###### Abstract

Registration between an fMRI volume and a T1-weighted volume is challenging,
since fMRI volumes contain geometric distortions. Here we present preliminary
results showing that 3D CycleGAN can be used to synthesize fMRI volumes from
T1-weighted volumes, and vice versa, which can facilitate registration.

##  1 Introduction

Preprocessing of fMRI data is a difficult task, due to head motion and
geometric distortions. For example, to perform registration between anatomical
(T1) space and fMRI space is difficult due to the contrast difference. The
fact that fMRI volumes contain geometric distortions further complicates the
registration, since these distortions are normally not present in the
anatomical volumes. It is also difficult to evaluate how good a T1-fMRI
registration is, mainly due to the distortions, and thereby it is also
difficult to compare different fMRI softwares. If it would be possible to
synthesize an fMRI volume from a T1-weighted volume, the two volumes would be
perfectly registered and can thereby be used as ground truth for evaluating
registration algorithms. If the synthetic fMRI image does not contain
distortions, it can be used for distortion correction through non-linear
registration [1].

Generative adversarial networks (GANs) [2] can today produce very realistic
synthetic images [3]. GANs use adversarial training, where a generator creates
an image from noise and a discriminator classifies each image as synthetic or
real. During the training, the generator becomes better at generating
realistic images, and the discriminator becomes better at discriminating
images as synthetic or real. GANs can broadly be divided into noise-to-image
GANs [2, 3], which produce an image from a noise vector, and image-to-image
GANs, which produce an image from another image (image to image translation)
[4, 5]. In medical imaging, GANs have already been used for a number of
applications, such as segmentation, synthesis, registration, reconstruction
and detection [6]. For image to image translation, CycleGAN [5] is one of the
most popular choices. It has previously been used to generate T2-weighted
images from T1-weighted images [7], and to generate diffusion MRI scalar maps
from T1-weighted images [1], but we are not aware of any application of
CycleGAN for T1 - fMRI translation.

In this work, we evaluate if CycleGAN can be used to synthesize realistic fMRI
volumes from T1-weighted volumes. Compared to our previous work [7, 1, 8]
which used a 2D CycleGAN, we here use a 3D CycleGAN to take advantage of the
3D correlations present in the data.

##  2 Theory

###  2.1 CycleGAN

CycleGAN [5] can be trained using two unpaired groups of images or volumes, to
translate data between domain A and domain B (e.g. T1 and fMRI). CycleGAN
consists of four main components, two generators ($G_{A2B}$, $G_{B2A}$) and
two discriminators ($D_{A}$ and $D_{B}$). The two generators generate domain
A/B images based on domain B/A images. The two discriminators discriminate
each image as synthetic or real. The unsupervised training is regularized by
the cycle consistency

$\displaystyle G_{B2A}(G_{A2B}(I_{A}))$ $\displaystyle\approx I_{A},$ (1)
$\displaystyle G_{A2B}(G_{B2A}(I_{B}))$ $\displaystyle\approx I_{B},$ (2)

where $I_{A}$ and $I_{B}$ are two images from domain A and B.

###  2.2 2D vs 3D

CycleGAN was initially proposed for 2D images, but conditional GANs have also
been applied in 3D. Nie et al. [9] used a 3D conditional GAN to synthesize CT
from MR, and worked on subvolumes of 32 x 32 x 32 voxels. Näppi et al. [10]
used 3D CycleGAN for virtual bowel cleansing in CT colonography, and worked on
volumes of 96 x 96 x 96 voxels. Yu et al. [11] used a 3D conditional GAN to
synthesize FLAIR images from T1-weighted images, to improve brain tumour
segmentation. The conditional GAN processed volumes of size 240 x 240 x 155
voxels using subvolumes of 128 x 128 x 128 voxels.

For 3D volumes there are several different options. The most straight forward
approach is to train a 2D CycleGAN and then apply it to every slice in a
volume. Such an approach will not take advantage of the available 3D
information, and can lead to discontinuities between the slices. Training a 3D
CycleGAN can be more computationally demanding, since a (non-separable) 3D
convolution involves more calculations compared to a 2D convolution. Another
challenge of a 3D CycleGAN is the memory consumption, since training a
CycleGAN involves training four convolutional neural networks (CNNs) at the
same time (two generators and two discriminators), and many filter responses
need to be stored for every layer of each CNN.

Training in 3D can be performed with full size volumes (e.g. 128 x 128 x 128
or 256 x 256 x 256 voxels), or with subvolumes (e.g. 32 x 32 x 32 voxels).
Using full size volumes means that only a single training example is available
for each subject, and to fit several hundred filter responses in GPU memory
can be difficult for large volumes. Using subvolumes can lead to a large
number of training examples for each subject (it is possible to extract a 32 x
32 x 32 subvolume from a 128 x 128 x 128 volume in many different ways), and
the memory consumption is also substantially reduced. If the network is fully
convolutional, the trained network can then be applied to a volume of any
size. The drawback of using subvolumes is that the performance may be reduced,
since the network does not see the full volume. Since it in our case is
possible to use a minibatch of 1 for volumes of 152 x 180 x 120 voxels we
focused on the full size approach.

##  3 Data

We used fMRI and T1-weighted volumes from the 1000 functional connectomes
project [12], and focused on the Beijing and Cambridge datasets since they
consist of data from 198 subjects each. For Beijing, the T1 volumes have a
varying size and voxel size of 1.3 x 1.0 x 1.0 mm3, while the fMRI data have a
size of 64 x 64 x 33 voxels and a voxel size of 3.125 x 3.125 x 3.6 mm3. For
Cambridge, the T1 volumes have a varying size and a voxel size of 1.2 x 1.2 x
1.2 mm3, while the fMRI data have a size of 72 x 72 x 47 voxels and a voxel
size of 3 x 3 x 3 mm3. We trained the network with 160 subjects, and used the
remaining 38 subjects for testing.

##  4 Methods

###  4.1 Preprocessing

CycleGAN can be trained using unpaired (and unregistered) images, but we
registered the fMRI and T1 data to the MNI 152 brain template to facilitate
training and evaluation. We first used the epi_reg script in FSL, which uses
boundary based registration [13] for fMRI to T1 registration. We then used
flirt in FSL [14] to linearly register all T1 volumes to the MNI 152 template,
and finally combined the transformations to transform the fMRI data to MNI
space. To reduce training time and memory consumption, we cropped the MNI 152
1 mm brain template from 182 x 218 x 182 to 152 x 180 x 120 voxels. Processing
scripts are available on Github111https://github.com/wanderine/fMRI_GAN [15].

###  4.2 Data augmentation

The number of available training volumes is only 160. To increase the number
of training volumes we performed data augmentation by applying 10 random 3D
rotations to each of the 160 volumes, to achieve a total of 1760 training
volumes. The random rotations were generated from a normal distribution with a
mean of 0 and a standard deviation of 10 degrees.

###  4.3 CycleGAN

We based our 3D CycleGAN on the 2D Keras implementation provided in [7]. We
replaced all 2D convolutions with 3D convolutions, and reduced the number of
residual layers from 9 to 6, as recommended for smaller images [5]. The
architecture for the generator is given in Table 1, and the architecture for
the discriminator is given in Table 2. The volume size for the patchGAN, which
classifies each patch (subvolume) as real or synthetic, was reduced from 70 x
70 x 70 to 51 x 51 x 51, since our volumes are rather small compared to the
images in the original CycleGAN paper [5]. Training the 3D CycleGAN for 200
epochs, using a minibatch size of 1, took 24 days on an Nvidia Tesla V100
graphics card with 32 GB of memory.

##  5 Results

Figures 1 \- 4 show results for Beijing data, and Figures 5 \- 8 show results
for Cambridge data. For slices in the upper part of the brain the fMRI-T1
transformation is not so difficult (mainly a change in contrast), while a
rather non-liner transformation must be learned for slices in the lower part
of the brain (due to severe distortions).

Fig. 1: fMRI to T1 image translation results for 6 slices of a test subject,
for Beijing data. First row: real fMRI images, second row: real T1 images,
third row: synthetic T1 images.

Fig. 2: T1 to fMRI image translation results for 6 slices of a test subject,
for Beijing data. First row: real T1 images, second row: real fMRI images,
third row: synthetic fMRI images.

Fig. 3: fMRI to T1 image translation results for 6 slices of a test subject,
for Beijing data. First row: real fMRI images, second row: real T1 images,
third row: synthetic T1 images.

Fig. 4: T1 to fMRI image translation results for 6 slices of a test subject,
for Beijing data. First row: real T1 images, second row: real fMRI images,
third row: synthetic fMRI images.

Fig. 5: fMRI to T1 image translation results for 6 slices of a test subject,
for Cambridge data. First row: real fMRI images, second row: real T1 images,
third row: synthetic T1 images.

Fig. 6: T1 to fMRI image translation results for 6 slices of a test subject,
for Cambridge data. First row: real T1 images, second row: real fMRI images,
third row: synthetic fMRI images.

Fig. 7: fMRI to T1 image translation results for 6 slices of a test subject,
for Cambridge data. First row: real fMRI images, second row: real T1 images,
third row: synthetic T1 images.

Fig. 8: T1 to fMRI image translation results for 6 slices of a test subject,
for Cambridge data. First row: real T1 images, second row: real fMRI images,
third row: synthetic fMRI images.

##  6 Conclusion

We have demonstrated that 3D CycleGAN can be used to synthesize fMRI images
from T1-weighted images, and vice versa. We have here focused on transforming
a full volume, rather than working on subvolumes, but in future work we will
make a comparison between the two solutions.

In future work we will also investigate the performance of T1-fMRI
registration for different fMRI softwares, by generating a synthetic fMRI
volume from the T1-weighted volume, and then apply different transformations
that should be found by the registration algorithms.

## Acknowledgements

This study was supported by Swedish research council grant 2017-04889. Funding
was also provided by the Center for Industrial Information Technology (CENIIT)
at Linköping University, Analytic Imaging Diagnostics Arena (AIDA) and the
ITEA3 / VINNOVA funded project ”Intelligence based iMprovement of Personalized
treatment And Clinical workflow supporT” (IMPACT).

Table 1: Architecture used for the generator in our 3D CycleGAN. Layer | Layer type | Number of filters | Filter size | Stride | Activation function
---|---|---|---|---|---
1 | 3D Convolution | 32 | 7 | 1 | ReLU
2 | Downsampling (3D Convolution) | 64 | 3 | 2 | ReLU
3 | Downsampling (3D Convolution) | 128 | 3 | 2 | ReLU
4 - 9 | Residual (3D Convolution) | 128 | 3 | 1 | None
10 | Upsampling (3D Transpose Convolution) | 64 | 3 | 2 | ReLU
11 | Upsampling (3D Transpose Convolution) | 32 | 3 | 2 | ReLU
12 | 3D Convolution | 1 | 7 | 1 | tanh
Table 2: Architecture used for the discriminator in our 3D CycleGAN. Layer | Layer type | Number of filters | Filter size | Stride | Activation function
---|---|---|---|---|---
1 | 3D Convolution | 64 | 4 | 2 | LeakyReLU (0.2)
2 | 3D Convolution | 128 | 4 | 2 | LeakyReLU (0.2)
3 | 3D Convolution | 256 | 4 | 1 | LeakyReLU (0.2)
4 | 3D Convolution | 512 | 4 | 1 | LeakyReLU (0.2)
5 | 3D Convolution (PatchGAN) | 1 | 4 | 1 | Sigmoid

## References

  * [1] Xuan Gu, Hans Knutsson, Markus Nilsson, and Anders Eklund,  “Generating diffusion MRI scalar maps from T1 weighted images using generative adversarial networks,”  in Scandinavian Conference on Image Analysis, 2019, pp. 489–498.
  * [2] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial nets,”  in Advances in neural information processing systems, 2014, pp. 2672–2680.
  * [3] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen,  “Progressive growing of GANs for improved quality, stability, and variation,”  arXiv preprint arXiv:1710.10196, 2017.
  * [4] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros,  “Image-to-image translation with conditional adversarial networks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125–1134.
  * [5] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros,  “Unpaired image-to-image translation using cycle-consistent adversarial networks,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223–2232.
  * [6] Xin Yi, Ekta Walia, and Paul Babyn,  “Generative adversarial network in medical imaging: A review,”  Medical image analysis, p. 101552, 2019.
  * [7] Per Welander, Simon Karlsson, and Anders Eklund,  “Generative adversarial networks for image-to-image translation on multi-contrast MR images-a comparison of CycleGAN and UNIT,”  arXiv preprint arXiv:1806.07777, 2018.
  * [8] David Abramian and Anders Eklund,  “Refacing: reconstructing anonymized facial features using GANs,”  in IEEE International Symposium on Biomedical Imaging (ISBI), 2019, pp. 1104–1108.
  * [9] Dong Nie, Roger Trullo, Jun Lian, Caroline Petitjean, Su Ruan, Qian Wang, and Dinggang Shen,  “Medical image synthesis with context-aware generative adversarial networks,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention, 2017, pp. 417–425.
  * [10] Janne J Näppi and Hiroyuki Yoshida,  “Cycle-consistent 3D-generative adversarial network for virtual bowel cleansing in CT colonography,”  in Medical Imaging 2019: Image Processing, 2019, vol. 10949, p. 109492Z.
  * [11] Biting Yu, Luping Zhou, Lei Wang, Jurgen Fripp, and Pierrick Bourgeat,  “3D cGAN based cross-modality MR image synthesis for brain tumor segmentation,”  in IEEE International Symposium on Biomedical Imaging (ISBI), 2018, pp. 626–630.
  * [12] Bharat B Biswal, Maarten Mennes, Xi-Nian Zuo, Suril Gohel, Clare Kelly, Steve M Smith, Christian F Beckmann, Jonathan S Adelstein, Randy L Buckner, Stan Colcombe, et al.,  “Toward discovery science of human brain function,”  Proceedings of the National Academy of Sciences, vol. 107, no. 10, pp. 4734–4739, 2010.
  * [13] Douglas N Greve and Bruce Fischl,  “Accurate and robust brain image alignment using boundary-based registration,”  Neuroimage, vol. 48, no. 1, pp. 63–72, 2009.
  * [14] Mark Jenkinson, Peter Bannister, Michael Brady, and Stephen Smith,  “Improved optimization for the robust and accurate linear registration and motion correction of brain images,”  Neuroimage, vol. 17, no. 2, pp. 825–841, 2002.
  * [15] Anders Eklund, Thomas E Nichols, and Hans Knutsson,  “Reply to Brown and Behrmann, Cox, et al., and Kessler et al.: Data and code sharing is the way forward for fMRI,”  Proceedings of the National Academy of Sciences, vol. 114, no. 17, pp. E3374–E3375, 2017.


# Attention-based Image Upsampling

Souvik Kundu*, Hesham Mostafa*, Sharath Nittur Sridhar*, Sairam Sundaresan* S.
Kundu is a Ph.D. student at the University of Southern California, USA. The
work was done during his internship at Intel Labs. H. Mostafa, S. N. Sridhar,
and S. Sundaresan are with Intel Labs, San Diego, USA. E-mail:
<EMAIL_ADDRESS>{hesham.mostafa, sharath.nittur.sridhar,
<EMAIL_ADDRESS>* Equal contributions. Authors listed in
alphabetical order.

###### Abstract

Convolutional layers are an integral part of many deep neural network
solutions in computer vision. Recent work shows that replacing the standard
convolution operation with mechanisms based on self-attention leads to
improved performance on image classification and object detection tasks. In
this work, we show how attention mechanisms can be used to replace another
canonical operation: strided transposed convolution. We term our novel
attention-based operation attention-based upsampling since it
increases/upsamples the spatial dimensions of the feature maps. Through
experiments on single image super-resolution and joint-image upsampling tasks,
we show that attention-based upsampling consistently outperforms traditional
upsampling methods based on strided transposed convolution or based on
adaptive filters while using fewer parameters. We show that the inherent
flexibility of the attention mechanism, which allows it to use separate
sources for calculating the attention coefficients and the attention targets,
makes attention-based upsampling a natural choice when fusing information from
multiple image modalities.

###### Index Terms:

Convolution, Self-attention, Image upsampling, Image super-resolution

##  I Introduction

Convolutional layers are an integral part of most deep neural networks used in
vision applications such as image recognition [1, 2, 3], image segmentation
[4, 5], joint image filtering [6, 7], image super-resolution [8, 9, 10, 11,
12] to name a few. The convolution operation is translationally equivariant,
which when coupled with pooling, introduces useful translation invariance
properties in image classification tasks. Computationally, local receptive
fields and weight sharing are highly beneficial in reducing the number of
floating point operations (FLOPs) and the parameter count, respectively,
compared to fully connected layers. However, using the same convolutional
weights at all pixel positions might not necessarily be the optimal choice.
Motivated by the success of attention mechanisms [13, 14] in natural language
processing tasks, several recent methods replace the convolution operation
with a self-attention mechanism [15, 16]. While the receptive field size does
not necessarily change, the use of self-attention allows the pixel aggregation
weights (attention coefficients) to be content-dependent. This, in effect,
results in an adaptive kernel whose weights vary across the spatial locations
based on the receptive field contents.

In this paper, we focus on another fundamental operation in vision
applications which has received relatively little attention: strided
transposed convolution or deconvolution. Unlike the standard strided
convolution operation which reduces the input’s spatial dimension by the
stride factor, strided transposed convolution increases the input’s spatial
dimension by the stride factor. Strided transposed convolution is thus a
natural operation to use in image upsampling tasks. We replace the fixed
kernel used in the strided transposed convolution by a self-attention
mechanism that calculates the kernel weights at each pixel position based on
the contents of the receptive field.

We test our new attention-based upsampling mechanism on two popular tasks:
single image super-resolution and joint-image upsampling. We show on various
benchmarks in the single-image super-resolution task that our attention-based
upsampling mechanism performs consistently better than using standard strided
transposed convolution operations. Our attention-based mechanism also uses
considerably fewer parameters. Thus, the proposed attention-based upsampling
layer can be used as as a drop-in replacement of the strided transposed
convolution layer, which improves model performance due to the content-
adaptive nature of the attention-generated kernels.

In joint-image upsampling tasks, the goal is to upsample an image from one
modality using a high-resolution image from another modality. Unlike the
standard strided transposed convolution operation, our attention-based
formulation produces three intermediate tensors, namely the query, key, and
value tensors. This allows us more flexibility in defining the attention-based
upsampling mechanism as we do not necessarily have to obtain these tensors
from the same set of feature maps. Indeed, we show that in joint-image
upsampling tasks that involve multiple input modalities (for example a depth
map and an RGB image), obtaining the query and key tensors from one modality
and the value tensor from another modality naturally allows our attention-
based upsampling mechanism to combine information from multiple modalities.
The conventional strided transposed convolution operation does not have this
ability. We thus compare against joint image upsampling models of similar
complexity and show that our attention-based joint upsampling model is more
accurate and parameter-efficient.

The reminder of this paper is arranged as follows. Section II describes
related work. Details of our proposed method and experimental evaluations are
presented in Section III and IV, respectively. We provide our conclusions in
Section V.

##  II Related Work

###  II-A Self-Attention

Originally introduced in [17], attention mechanisms have displayed impressive
performance in sequence to sequence modeling tasks, where they significantly
outperform traditional methods based on recurrent neural networks [18]. An
attention mechanism aggregates features from multiple inputs where the
aggregation weights are dynamically calculated based on the input features.
Attention mechanisms differ in the way they calculate the aggregation weights
(attention coefficients) [18, 19, 20]. In this paper, we primarily use the
scaled dot product attention mechanism of [18], which is an extension of the
unscaled dot product method of [19].

Beyond their use in sequence to sequence tasks, attention mechanisms have more
recently been used to replace or augment convolutional neural networks (CNNs)
[21, 22, 23]. Instead of using a fixed kernel to aggregate features from all
pixels within a receptive field, attention-based convolution uses an attention
mechanism to dynamically calculate these aggregation weights based on the
contents of the receptive field. This offers more flexibility as the kernel is
now content-dependent and not fixed. This leads to improved accuracy on
several classification and object detection tasks [21, 22, 23]. It is
improtant to note that these convolutional attention models are fundamentally
different from previous attention based vision models that use attention to
preferentially aggregate information from parts of the input image [24, 25],
but do not modify the convolution operation itself.

###  II-B Joint Image Upsampling

In joint image upsampling tasks, in addition to the low-resolution image, the
network also receives a corresponding high-resolution image, but from a
different modality. Information from the high-resolution modality (the guide
modality) can be used to guide the upsampling of the low-resolution modality
(the target modality). In particular, image filtering with the help of a
guidance image has been used on a variety of tasks which include depth map
enhancement [26, 27, 28], cross-modality image restoration [29], texture
removal [30], dense correspondence [31], and semantic segmentation [32]. The
basic idea of joint image filtering is to transfer the structural details from
a high-resolution guidance image to a low-resolution target through estimation
of spatially-varying kernels from the guidance. Classical approaches of joint
image filtering focus on hand-crafted kernels [33]. For example, both the
bilateral filter [34] and the guided filter [35] leverage the local structure
of the guidance image using spatially-variant Gaussian kernels and matting
Laplacian kernels [36], respectively, without handling inconsistent structures
in the guidance and target images. The SD filter [28] constructs spatially-
variant kernels from both guidance and target images and exploits common
structures by formulating joint image filtering as an optimization problem.
Recent advances in learnable upsampling mechanisms using convolutional layers
achieve state-of-the-art performance at various upsampling factors [7, 37]. In
particular, [37] proposes a deep joint filtering (DJF) CNN based model to
learn the spatially-variant kernels by non-linearly mixing activation values
of spatially-invariant kernels. On the the other hand, the fast deformable
kernel network (FDKN) [7] exploits the advantage of spatially-variant kernels
by learning to generate kernels at each spatial positions, together with
learning the receptive field structure at each position, outperforming
previous approaches. The closest technique to our method uses the feature maps
from the guide modality to adaptively modify the convolutional kernels applied
to the target modality image [38]. Similar to our approach, these kernel
adaptations are different at each position.

###  II-C Single Image Super Resolution

Increasing the resolution of images, or image upscaling is a common task with
many variations in computer vision. Initial approaches were based on non-
learnable interpolation methods such as bilinear or bicubic interpolation
[39]. Non-learnable interpolation methods, however, significantly under-
perform learning based methods that utilize a deep network trained in a
supervised setting to produce the high-resolution image from the low-
resolution input [40]. These two are often combined where a non-learnable
interpolation mechanism is used to upsample the image, followed by a learnable
CNN to produce the final high-resolution output [41], or combined in the
reverse order where a CNN is first applied to the low-resolution image
followed by non-learnable interpolation to upsample the final image [42]. The
upsampling mechanism itself can be learned; this is the case when using the
strided transposed convolution, or the deconvolution [43] operation to
upsample the image. An alternative learned upsampling mechanism is sub-pixel
convolution [44], where $F*S^{2}$ feature maps of dimensions $H\times W$ are
reshaped into $F$ feature maps of dimensions $H*S\times W*S$, where $S$ is the
upsampling factor.

##  III Methods

Figure 1: Illustration of the strided transposed convolution with stride 2.

###  III-A Background

####  III-A1 Scaled dot product attention

The attention mechanism we use is a scaled dot product attention mechanism
similar to that proposed in ref. [18], which we summarize here. The inputs to
the attention mechanism are query, key, and value tensors: ${\bf Q}\in
R^{N\times F_{K}}$, ${\bf K}\in\mathbb{R}^{N\times F_{K}}$, and ${\bf
V}\in\mathbb{R}^{N\times F_{V}}$, respectively. $N$ is the number of input and
output tokens. $F_{K}$ is the dimension of the key and query vectors for each
token, and $F_{V}$ is the dimension of the value vector at each token. The
output of the attention mechanism, ${\bf Y}\in\mathbb{R}^{N\times F_{V}}$ is
given by:

$\displaystyle{\bf A}$ $\displaystyle=\frac{{\bf Q}{\bf
K}^{T}}{\sqrt{F_{K}}},$ (1) $\displaystyle{\bf Y}$ $\displaystyle=softmax({\bf
A}){\bf V},$ (2)

where ${\bf A}\in\mathbb{R}^{N\times N}$ is the matrix of attention
coefficients and the softmax operation is applied row-wise.

Equation 1 is invariant to the ordering of the input tokens, which is
undesirable as the order contains important information. A common approach to
address this issue is to add positional encoding vectors to the query and key
vectors to inject positional information. These positional encodings can
either be absolute [18], i.e, they encode the absolute position of the token’s
query and key vectors, or they can be relative, i.e, they encode the relative
position between the query/key vectors at one position and the query/key
vectors at a second position. We primarily focus on relative positional
encoding methods inspired by ref. [45]. With relative positional encoding,
instead of Eq. 1, the entries of the attention matrix ${\bf A}$ are given by:

$a_{ij}=\frac{{\bf q}_{i}^{T}({\bf k_{j}}+{\bf p}_{i-j})}{\sqrt{F_{K}}},$ (3)

where ${\bf p}_{i-j}$ is the relative positional encoding term, ${\bf q}_{i}$
is the query vector at output position $i$, and ${\bf k}_{j}$ is the key
vector at input position $j$.

####  III-A2 Strided transposed convolution

Convolution is one of the prototypical operations in computer vision. In this
paper, we are primarily interested in a variant of the convolution operation
which is the strided transposed convolution. This operation is also called
fractionally strided convolution or deconvolution [43]. We use these terms
interchangeably throughout the paper. The architectural parameters of a
strided transposed convolution is the kernel size($K$), the stride($S$), the
number of input feature maps($C_{in}$), and number of output feature
maps($C_{out}$). We do not consider the padding parameters, because we always
choose them so that the operation exactly scales all spatial dimensions by a
factor of $S$. Given an input set of feature maps ${\bf
X}\in\mathbb{R}^{C_{in}\times H\times W}$ where $H$ and $W$ are are the height
and width of the input feature maps, the strided transposed convolution
operation first zero-upsamples ${\bf X}$ by a factor $S$ to yield ${\bf
X}_{up}=zero\\_upsample({\bf X};S)\in\mathbb{R}^{C_{in}\times S*H\times S*W}$,
where $S-1$ zeros are inserted between adjacent entries, and after the final
row and column:

$X_{up}[c,i,j]=\begin{cases}X[c,i//S,j//S],&\parbox{85.35826pt}{if ($i$ mod
$S=0$) and ($j$ mod $S=0$)}\\\ 0,&\text{otherwise},\end{cases}$ (4)

Figure 2: The attention-based upsampling operation for $S=2$ and $K=3$. For a
particular output position, position $(1,2)$, we show its dependence on
entries in ${\bf Q}_{up}$, ${\bf K}_{up}$, and ${\bf V}_{up}$. To produce the
output at position $(1,2)$, we use a masked attention mechanism in a $K\times
K$ window around (1,2). There are only two input positions allowed by the mask
(position $(0,2)$ and position $(2,2)$). We use the query vector at position
(1,2) to obtain the normalized attention coefficients through a dot product
with the key vectors at these two positions followed by a softmax. The
resulting two normalized attention coefficients are then used to sum the value
vectors at the two allowed positions to yield the output vector.

where $//$ denotes integer division and indexing is 0-based. ${\bf X}_{up}$ is
then fed to a standard convolution operation with stride one, kernel size $K$,
and padding $(K-1)//2$ (we always assume $K$ is odd) to yield ${\bf
Z}\in\mathbb{R}^{C_{out}\times S*H\times S*W}$:

${\bf Z}=conv2d({\bf X}_{up},{\bf W},padding=(K-1)//2),$ (5)

where ${\bf W}\in\mathbb{R}^{C_{out}\times C_{in}\times K\times K}$ is the
learnable kernel tensor. The strided transposed convolution operation is
illustrated in Fig. 1.

####  III-A3 Attention-based convolution

Multiple prior work has looked into replacing the standard convolution
operation by an attention mechanism [21, 22, 23]. In each receptive field
window, the attention mechanism is responsible for generating a set of
aggregation weights (attention coefficients) that are used by the center pixel
to aggregate the features of the pixels in the receptive field window.
Attention can thus be seen as a mechanism to generate a convolutional kernel
at each spatial position. Unlike the convolutional kernel in standard CNNs
which is shared across all spatial positions, the attention-generated kernels
depend on the contents of the receptive field and are thus different at each
spatial position.

Figure 3: Network architecture of the self-attention based deep joint
upsampling model for an upsampling factor of 4. The proposed model consists of
four major components. Similar to [37] the sub-networks $CNN_{T}$ and
$CNN_{G}$ aim to extract informative feature responses from the target and
guidance images, respectively. The responses are then fed to a self-attention
(SA) block to learn pixel specific contextual information, before being fed to
the final convolutional block $CNN_{F}$. The desired reconstructed image is
available at the output of $CNN_{F}$. $CNN_{M}$ takes the concatenated low-
resolution and high-resolution features to produce the query and key tensors
at each upsampling stage. All convolutional layers (except the final layer)
and self-attention layers are followed by a ReLU non-linearity (not shown for
simplicity).

We are primarily interested in a dot product convolutional attention mechanism
similar to that introduced in ref. [21]. Given an input set of feature maps
${\bf X}\in\mathbb{R}^{C_{in}\times H\times W}$, three standard $1\times 1$
convolution operations are used to obtain the query, key, and value tensors:

$\displaystyle\begin{split}{\bf Q}&=conv1d({\bf X},{\bf W}^{Q}),\\\ {\bf
K}&=conv1d({\bf X},{\bf W}^{K}),\\\ {\bf V}&=conv1d({\bf X},{\bf
W}^{V}),\end{split}$ (6)

where ${\bf Q},{\bf K},{\bf V}\in\mathbb{R}^{C_{out}\times H\times W}$. ${\bf
W}^{Q},{\bf W}^{K},{\bf W}^{V}\in\mathbb{R}^{C_{out}\times C_{in}\times
1\times 1}$ are the weight tensors of the three $1\times 1$ convolution
operations. The output feature tensor ${\bf Z}\in\mathbb{R}^{C_{out}\times
H\times W}$ is given by:

${\bf z}_{ij}=\sum\limits_{a,b\in\mathcal{N}_{K}(i,j)}softmax_{ab}\left({\bf
q}_{ij}^{T}({\bf k}_{ab}+{\bf p}_{a-i,b-j})\right){\bf v}_{ab},$ (7)

where $\mathcal{N}_{K}(i,j)$ is the $K\times K$ spatial neighborhood around
position $(i,j)$. ${\bf z}_{ij},{\bf q}_{ij},{\bf k}_{ij},{\bf
v}_{ij}\in\mathbb{R}^{C_{out}}$ are the output, query, key, and value feature
vectors at position $(i,j)$, respectively. The softmax operator ensures that
the attention coefficients within each $K\times K$ receptive field windows sum
to one. The relative positional encoding vector ${\bf
p}_{a-i,b-j}\in\mathbb{R}^{C_{out}}$ is decomposed into the concatenation of
two relative positional vectors:

${\bf p}_{a-i,b-j}\equiv({\bf p}^{x}_{a-i}||{\bf p}^{y}_{b-j}),$

where $||$ is the concatenation operator and ${\bf p}^{x}_{a-i},{\bf
p}^{y}_{b-j}\in\mathbb{R}^{C_{out}/2}$ are the relative positional encoding
vectors in the X and Y dimensions, respectively, and they are learned together
with the model parameters using standard stochastic gradient descent.

###  III-B Attention-based upsampling

We propose a flexible attention-based upsampling layer that can be used as a
drop-in replacement for the strided transposed convolution layer. Given ${\bf
X}\in\mathbb{R}^{C_{in}\times H\times W}$, the output of the attention-based
upsampling layer is a set of feature maps: ${\bf
Z}\in\mathbb{R}^{C_{out}\times H*S\times W*S}$, where $S$ is the upsampling
factor. As in attention-based convolution, we apply three $1\times 1$
convolutional layers on ${\bf X}$ to get the query, key, and value tensors:
${\bf Q},{\bf K},{\bf V}\in\mathbb{R}^{C_{out}\times H\times W}$ (see Eq. 6).
We zero-upsample ${\bf K}$ and ${\bf V}$ by a factor $S$ (see Eq. 4) and use
bilinear interpolation to upscale ${\bf Q}$ by a factor $S$:

$\displaystyle{\bf K}_{up}$ $\displaystyle=zero\\_upsample({\bf K};S),$ (8)
$\displaystyle{\bf V}_{up}$ $\displaystyle=zero\\_upsample({\bf V};S),$ (9)
$\displaystyle{\bf Q}_{up}$ $\displaystyle=bilinear\\_interpolate({\bf Q};S),$
(10)

to obtain ${\bf Q}_{up},{\bf K}_{up},{\bf V}_{up}\in\mathbb{R}^{C_{out}\times
H*S\times W*S}$. We define an attention mask ${\bf M}\in\mathbb{R}^{H*S\times
W*S}$:

$M[i,j]=\begin{cases}0&\parbox{85.35826pt}{if ($i$ mod $S=0$) and ($j$ mod
$S=0$)}\\\ -\inf,&\text{otherwise},\end{cases}$ (11)

The mask ${\bf M}$ thus has the value $-inf$ at all the zero positions of the
zero-upsampled tensors. We then use a masked attention mechanism to obtain the
output feature maps:

$\displaystyle{\bf
z}_{ij}=\sum\limits_{a,b\in\mathcal{N}_{K}(i,j)}softmax_{ab}({\bf
q}_{ij}^{T}({\bf k}_{ab}$ $\displaystyle+{\bf p}_{a-i,b-j}$
$\displaystyle+M[a,b]){\bf v}_{ab},$ (12)

where we use the same notation and positional encoding scheme as Eq. 7.
Wherever the mask value is $-inf$, it effectively sets the attention
coefficients to zero at these positions due to the action of the softmax. In
effect, this causes these positions to be ignored by the attention mechanism.
The attention-based upsampling mechanism is illustrated in Fig. 2. By using
bilinear interpolation, we generate query vectors at the new spatial
positions. The attention mechanism then uses the high-resolution query map to
aggregate information only from the valid (non-zero) positions in the zero-
upsampled values map using the key vectors at these valid (non-zero)
positions.

Our proposed upsampling layer can also go beyond the strided transposed
convolution layer by allowing the attention-based fusion of features from
different modalities. In joint image upsampling tasks, for example, the goal
is to upsample a low-resolution image of one modality (the target modality)
using a high-resolution image from a different modality (the guide modality).
In this setting, we thus have two sets of input features maps: the guide
feature maps ${\bf X}^{HR}\in\mathbb{R}^{C_{hr}\times(S*H)\times(S*W)}$, and
the target feature maps ${\bf X}^{LR}\in\mathbb{R}^{C_{lr}\times H\times W}$
that we want to upsample by a factor $S$. In this setting, we generate the
query and key tensors from the high resolution guide maps, and the value
tensor from the low resolution target maps:

$\displaystyle{\bf Q}_{up}$ $\displaystyle=conv1d({\bf X}^{HR},{\bf W}^{Q}),$
(13) $\displaystyle{\bf K}_{up}$ $\displaystyle=conv1d({\bf X}^{HR},{\bf
W}^{K}),$ (14) $\displaystyle{\bf V}_{up}$
$\displaystyle=zero\\_upsample(conv1d({\bf X}^{LR},{\bf W}^{V});S)$ (15)

We then use masked attention (Eqs. 11 and 12) to obtain the output feature
maps from the query, key, and value tensors.

Figure 4: Performance of our attention based joint upsampling method as
compared to Deep Joint Filtering [37] on two target-guide image pairs. For
this visual comparison task we use an upsampling factor of $8\times$. Our
attention-based method typically produces sharper edges that are in-line with
the edges in the guide image while DJF has trouble transferring the high-
resolution edge information from the guide image to the upsampled target
image.

The number of parameters in a traditional strided transposed convolution
operation is $C_{in}C_{out}K^{2}$ (ignoring biases), whereas the number of
parameters in our attention-based upsampling operation is
$3C_{in}C_{out}+KC_{out}$; these are the parameters of the 3 $1\times 1$
convolution layers used to generate the key, query, and value tensors, and the
positional encoding parameters. Our attention-based upsampling operation has
significantly fewer parameters: approximately 3X less parameters when $K=3$
and the parameter advantage grows as the kernel size $K$ increases. We will
show in the single-image super-resolution tasks that using our attention-based
upsampling mechanism as a drop-in replacement of strided transposed
convolution not only lowers the parameter count significantly, but also
improves accuracy.

(a)

(b)

(c)

Figure 5: Parameters vs RMSE plot for the convolution based DJF, FDKN and our
proposed self-attention based models for upsampling factor
LABEL:sub@subfig:plot_1 $4\times$, LABEL:sub@subfig:plot_2 $8\times$, and
LABEL:sub@subfig:plot_3 $16\times$. It is clear that, for all three upsampling
factors, our proposed attention-based upsampling models result in better
parameter-RMSE trade-offs.

##  IV Experimental Analysis

###  IV-A Joint Image Upsampling

####  IV-A1 Architectural Details

The model architecture is shown in Fig. 3 where we use upsampling factor of 4
as an example. The general approach to design the architecture is described in
algorithm 1. The low-resolution target image and the high-resolution guide
image are each first passed through a 3-layer CNN to produce a low resolution
set and a high resolution set of feature maps: ${\bf Y}^{LR}$ and ${\bf
Y}^{HR}$, respectively. We then successively upsample by a factor of 2 across
multiple attention-based upsampling layers (we thus require the upsampling
factor to be a power of 2). In each upsampling stage, the value tensor is the
output of the previous stage, or the zero-upsampled low resolution feature
maps for the first stage.The query and key tensors are obtained by applying a
CNN ($CNN_{M}$ in Fig. 3) to a concatenation of the value tensor and an
appropriately downsampled version of the high resolution feature maps ${\bf
Y}^{HR}$. The output feature maps of $CNN_{M}$ are split into two parts to
obtain the query and key tensors. For an upsampling factor $2^{k}$, there are
$k$ upsampling stages with each stage upsampling by a factor of 2X.

In the proposed architecture, the high-resolution guide is used to obtain the
key and query tensors for the attention-based upsampling layer. These two
tensors define how similar two positions are in the guide modality. They
provide a high-resolution source of similarity information to the attention-
based upsampling layer. Each position in the upsampled output of the target
modality would thus be produced by aggregating information from input
positions that are most similar to it in the guide modality.

Algorithm 1 Attention-based joint upsampling

: ${\bf X}^{LR}\in\mathbb{R}^{1\times(H/M)\times(W/M)}$, ${\bf
X}^{HR}\in\mathbb{R}^{3\times H\times W}$

: Upsampling factor $M$

${\bf Y}^{LR}\leftarrow CNN_{T}({\bf X}^{LR})$

${\bf Y}^{HR}\leftarrow CNN_{G}({\bf X}^{HR})$

${\bf V}_{up}^{1}\leftarrow zero\\_upsample({\bf Y}^{LR};2)$

$n_{steps}\leftarrow log_{2}(M)$

for $i\leftarrow 1$ to $n_{steps}$ do

${\bf Y}^{HR}_{DS}\leftarrow downsample({\bf Y}^{HR};M//2^{i})$

${\bf Q}_{up}^{i},{\bf K}_{up}^{i}\leftarrow CNN_{M}({\bf V}_{up}^{i}||{\bf
Y}^{HR}_{DS})$

${\bf V}_{up}^{i+1}\leftarrow self\\_attention({\bf Q}_{up}^{i},{\bf
K}_{up}^{i},{\bf V}_{up}^{i})$

end for

${\bf out}\leftarrow CNN_{F}({\bf V}_{up}^{n_{steps}+1})$

####  IV-A2 Experimental Setup

We use the NYU depth dataset v2 [46] which has 1449 pairs of aligned RGB and
depth images. We follow [37], where we use the first 1000 image-pairs for
training and the rest for testing. We use the high-resolution RGB images to
guide and upsample the low-resolution depth map and experiment with upsampling
factors of $4\times$, $8\times$, and $16\times$. For each upsampling factor,
we obtain the low-resolution depth map by sampling the depth map at grid
points that are 4, 8, and 16 pixels apart, respectively. For all these
upsampling factors, our goal is to minimize the Root Mean Square Error (RMSE)
between the upsampled depth map and the ground truth high resolution depth
map.

We train our network for 2000 epochs with the Adam optimizer and a starting
learning rate of 0.001 with a batch-size of 10. We use $256\times 256$ image
crops 111We reduced the learning rate proportionally while using smaller
training batches due to memory constraints when using larger models.. The
learning rate is decayed by a factor of 10 after 1200 and 1600 epochs. With
this setting, the Deep Joint Filtering(DJF) model [37] gives better RMSE than
in the original paper.

####  IV-A3 Results

We use multiple variants of the attention based upsampling model along with
increased parameter variants of the original DJF model for upsampling, as
described in Table I. In particular, we use different convolutional layers in
the mixing block and adjust the channel size to control the parameter count of
the proposed model. For example, the ’SA_M2_F32’ means the self-attention
based upsampling model will have 2 convolutional layers in the $CNN_{M}$ with
$2F=64$ channels (refer to Fig. 3). For the Fast deformable Kernel Network
(FDKN) [47] model we use the results reported in the original paper.

Model Name | Details of CONV layer channel widths
---|---
| $CNN_{T/G}$ | $CNN_{F}$ | $CNN_{M}$
DJFScaled+ | [384, 192, 1] | [288,144] | –
DJF+ | [512, 256, 1] | [384,192] | –
SA_M1_F8 | [96, 48, 8] | [16,16] | [16]
SA_M1_F16 | [96, 48, 16] | [32,32] | [32]
SA_M1_F32 | [96, 48, 32] | [64,64] | [64]
SA_M2_F32 | [96, 48, 32] | [64,64] | [64, 64]
TABLE I:  Descriptions of the various models used for joint image upsampling.
For the different component CNNs used in the models, we list the number of
channels in the CNN convolutional layers.

Table II compares the performance (RMSE) of our proposed model against prior
models. As we can see in this table, our attention-based joint-upsampling
model outperforms other approaches for all three upscale factors.

Fig. 4 shows a visual comparison of the upsampled depth maps ($8\times$). Our
model’s superior ability to extract common structures from the color and depth
images is clearly visible. Compared to DJF, our results show much sharper
depth transition without the texture-copying artifacts and thus have closer
resemblance to the ground truth.

Fig. 5 shows how RMSE varies with the number of parameters for various
learnable filter based techniques including our approach. Interestingly, the
original DJF model saturates in terms of providing better performance with
more parameters and thus cannot provide better RMSE with the DJF+ and
DJFScaled+ models (the increased parameter variants of DJF). However, this is
not true for our proposed attention-based joint upsampling model. As we can
see in Fig. 5, with the increased parameter count our model performance
consistently improves. Also, the FDKN model, having around $1.9\times$ more
parameters under-performs compared to our ’SA_M2_F32’ model for all the three
upsampling factors. In this paper, we focused on improving the performance of
our attention-based joint upsampling model by simply increasing the model’s
number of channels and the number of convolution layers in $CNN_{M}$. We
believe, however, that better and more parameter efficient models can be
obtained by replacing the standard convolutions by attention-based
convolutions. This is a topic for future work, however.

Method | Scale factor
---|---
| $4\times$ | $8\times$ | $16\times$
Bicubic | 8.16 | 14.22 | 22.32
MRF | 7.84 | 13.98 | 22.20
GF [35] | 7.32 | 13.62 | 22.03
JBU [48] | 4.07 | 8.29 | 13.35
Ham et al. [49] | 5.27 | 12.31 | 19.24
DMSG [50] | 3.78 | 6.37 | 11.16
FBS [32] | 4.29 | 8.94 | 14.59
DJF [37] | 3.54 | 6.20 | 10.21
FDKN [47] | 2.65 | 5.02 | 8.69
Ours (Self-Attention) | 2.64 | 5.01 | 8.63
TABLE II: Joint image upsampling. Results in RMSE (the lower, the better) on
NYU v2 dataset show that our upsampling network consistently outperforms
traditional techniques including conventional convolution based approach [37,
6, 47] for different upsampling factors. Figure 6: (a) Network architecture
used for single image super resolution with an upsampling factor of $2^{k}$.
The proposed model uses an attention-based upsampling block for upscaling the
image instead of strided transposed convolution. We use $k$ upsampling blocks
(where each upsampling block upsamples the image by a factor of 2X) for an
upsampling factor of $2^{k}$. (b) Internal block diagram of an attention
upsampling unit as described in more details in section III-B and Fig. 2. For
the baseline model, instead of the attention-based upsampling layer, we use a
standard strided transposed convolution layer with stride 2 in each upsampling
block.

###  IV-B Single Image Super Resolution

####  IV-B1 Architectural Details

The low-resolution input image is first passed through a CNN layer with PReLU
activation [51]. This is followed by an upsampling block which mainly has two
components. The first is a residual block having two CNN layers with ReLU
activation in between. The second component is the proposed attention-based
upsampling layer. For the baseline model, we use a simple strided transposed
convolution instead of the attention-based upsampling layer. Each upsampling
block upsamples by a factor of 2X. The model architecture used for single
image super-resolution with an upsampling factor of $2^{k}$, having $k$
upsampling blocks is shown in Figure 6. We use upsampling factors of $2$,$4$
and $8$ for our experiments.

####  IV-B2 Experimental Setup

We mainly train on two different training sets for our experiments. The $91$
image dataset (T-$91$) is popularly used as the training set for super-
resolution [8]. Additionally [10] introduced the General-100 dataset that
contains $100$ bmp-format images with no compression. The size of these $100$
images ranges from $710$ × $704$ to $131$ × $112$. We combine both T-91 and
General-100 to create one of our training sets. Our second training set is the
BSDS-200 [9] dataset. Additionally, we also perform data augmentation on the
training set similar to [10]. For augmentation, we downscale images with
factors of $0.9$, $0.8$, $0.7$ and $0.6$. Images are additionally rotated at
$90$, $180$ and $270$ degrees. To prepare the training samples after
augmentation, we initially downsample the original training images by the
desired scaling factor to generate the low-resolution (LR) images. The LR
images are then cropped into a set of $m\times m$ patches with a fixed stride.
The corresponding high-resolution (HR) patches are also cropped from the
ground truth images.These LR/HR image patches form the basis of our training
set similar to [10]. For evaluation, we use Set5 [11], Set14 [12] and BSDS100
[9] datasets.

We train all networks with a learning rate of $0.001$ , batch size of $20$
with the Adam optimizer. Additionally we use a learning rate scheduler that
decays the learning rate by a factor of $0.8$, each time there is a plateau in
the evaluation loss. Over the course of training, we saved the model weights
which provided the best PSNR on the evaluation dataset for each run.

####  IV-B3 Results

Tables IV and V show the PSNR values(measured in dB) for the deconvolution and
attention-based upsampling variants across different evaluation datasets. For
statistical significance, the PSNR values shown in Tables IV and V are an
average across three runs with different random seeds. The results for
BSDS100, Set5 and Set14 indicate that our proposed architecture performs
better than the equivalent deconvolution variant in almost all cases despite
having roughly 2x fewer parameters.We hypothesize that this is due to the
attention mechanism’s context aware feature extraction. Table III shows the
number of parameters across different scales for both models. This indicates
that self-attention can act as a drop in replacement for strided transposed
convolution with much fewer parameters.

Figure 7 shows the qualitative results for super-resolution across different
scales. We used models trained on the combination of T91 [8] and General-100
[10] to generate images shown in Figure 7. We can see that as the upsampling
factor increases, our attention-based upsampling model produces crisper images
compared to the strided transposed convolution model.

Scale | Attention-based | Deconvolution | Parameter
---|---|---|---
| upsampling |  | Improvement
2 | 0.35 M | 0.71 M | $2.02\times$
4 | 0.69 M | 1.41 M | $2.03\times$
8 | 1.04 M | 2.12 M | $2.03\times$
TABLE III: Parameter Count For Single Image Super-resolution. The total parameter count (in millions) for the proposed attention based upsampling architecture and the deconvolution baseline models. Note that parameters from the CNN layers in the residual blocks and the first CNN layer are also included in the total count. Our model has roughly 2x fewer parameters compared to the baseline. Test dataset | Scale | Attention-based | Deconvolution
---|---|---|---
|  | upsampling |
BSDS100 | 2 | 33.22 | 33.19
4 | 27.84 | 27.76
8 | 24.72 | 24.60
Set5 | 2 | 36.88 | 36.86
4 | 30.66 | 30.44
8 | 25.56 | 25.42
Set14 | 2 | 32.52 | 32.52
4 | 27.48 | 27.37
8 | 23.87 | 23.81
TABLE IV: Single Image Super-resolution. Results in PSNR measured in $dB$ (the higher, the better) on the BSDS100, Set5 and Set14 datasets show that our attention-based upsampling network consistently outperforms the deconvolution based counterpart for different upsampling factors. The results here showcase the performance of the models trained on a combination of the T-91 [8] and General-100 [10] datasets. Test Dataset | Scale | Attention-based | Deconvolution
---|---|---|---
|  | upsampling |
BSDS100 | 2 | 33.28 | 33.25
4 | 27.82 | 27.78
8 | 24.80 | 24.74
Set5 | 2 | 36.84 | 36.86
4 | 30.51 | 30.41
8 | 25.43 | 25.34
Set14 | 2 | 32.51 | 32.48
4 | 27.45 | 27.37
8 | 23.87 | 23.82
TABLE V: Single Image Super-resolution. Results in PSNR measured in $dB$ (the higher, the better) on the BSDS100, Set5 and Set14 datasets. We observe similar trends as in Table IV where our attention-based model outperforms the model that uses deconvolution operations. The results here showcase the performance of the models trained on only the BSDS200 dataset [9]. |  |
---|---|---
Scale |  2x |  2x
|  |
Scale |  4x |  4x
|  |
Scale |  8x |  8x
|  |
Scale |  2x |  2x
|  |
Scale |  4x |  4x
|  |
Scale |  8x |  8x
(a) Ground Truth HR |  (b) Ours |  (c) Deconvolution
Figure 7:  Single Image Super Resolution. The first column represents the
ground truth image, the second shows the highlighted region in the output of
the attention based upsampling model, and the third shows the output of the
model using strided transposed convolutions. As the scale increases, the
attention based model produces crisper images than the model based on strided
transposed convolutions.

##  V Conclusion

Convolutions are a standard mechanism to learn translation equivariant
features. Translation equivariance is achieved by sweeping a fixed and small
kernel across the input domain. Untying the kernel weights across spatial
positions increases the expressiveness of the network [52]. However, parameter
count increases dramatically and translation equivariance is lost making this
approach suitable only in cases where the objects of interest are aligned in
the different images. Kernel generating networks generate convolutional
kernels dynamically from the output of auxiliary neural networks [53, 7],
thereby keeping parameter count low while allowing the convolutional kernel to
vary across spatial positions. However, due to the use of auxiliary kernel-
generating networks, these methods incur additional parameter and
computational cost compared to vanilla CNNs.

Attention-based convolutions allow the kernels to vary across spatial
positions while simultaneously reducing parameter count compared to a standard
convolution [18, 19, 20] and maintaining translation equivariance. In this
paper, we have shown that attention mechanisms can also be used to replace
traditional strided transposed convolutions. The attention-based upsampling
mechanism we introduce simultaneously improves accuracy while greatly reducing
the parameter count compared to conventional strided transposed convolution in
single image super resolution tasks. Moreover, our attention-based upsampling
mechanism naturally allows us to integrate information from multiple image
modalities. This is done by generating the query and key tensors from one
modality, and the value tensor from a different modality. We have shown that
this allows us to naturally solve joint-image upsampling tasks and achieve
competitive performance.

Even though our attention-based upsampling mechanism has fewer parameters than
standard strided transposed convolution, and only a slightly higher number of
floating point operations (FLOPs), training is considerably slower. This is a
common issue across convolutional attention mechanisms [23, 22, 21]. The root
of the problem is that vanilla convolutions/deconvolutions operations have
been heavily optimized across GPU and CPU platforms with several high
performance compute kernels available from major hardware vendors and deep
learning libraries such as Pytorch and Tensorflow. This is not the case for
attention-based convolutions and for our attention-based upsampling mechanisms
which lack high quality low-level implementations from hardware vendors or
deep learning libraries. Implementing high quality low-level compute kernels
for attention-based convolutions/deconvolutions which are efficient across
different input dimensions, strides, kernel sizes, etc.. and across different
hardware platforms is not an easy task. We thus believe that more widespread
support for optimized attention-based convolutions/deconvolutions operations
in popular deep learning libraries is the logical next step to translate the
theoretical advantages attention-based convolutions/deconvolutions have in
terms of reduced parameter count into practical advantages in terms of reduced
hardware power consumption and wall-clock run times.

## References

  * [1] A. Krizhevsky _et al._ , “ImageNet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems_ , 2012, pp. 1097–1105.
  * [2] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [3] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 1–9.
  * [4] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “Overfeat: Integrated recognition, localization and detection using convolutional networks,” _arXiv preprint arXiv:1312.6229_ , 2013.
  * [5] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271.
  * [6] Y. Li, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Joint image filtering with deep convolutional networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 8, pp. 1909–1923, 2019.
  * [7] B. Kim, J. Ponce, and B. Ham, “Deformable kernel networks for joint image filtering,” _International Journal of Computer Vision_ , pp. 1–22, 2020\.
  * [8] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” _IEEE Transactions on Image Processing_ , vol. 19, no. 11, pp. 2861–2873, 2010.
  * [9] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , vol. 2, 2001, pp. 416–423 vol.2.
  * [10] C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” 2016.
  * [11] M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” 2012\.
  * [12] R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in _International conference on curves and surfaces_. Springer, 2010, pp. 711–730.
  * [13] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” in _Advances in neural information processing systems_ , 2015, pp. 577–585.
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [15] P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens, “Stand-alone self-attention in vision models,” _arXiv preprint arXiv:1906.05909_ , 2019.
  * [16] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, “Attention augmented convolutional networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3286–3295.
  * [17] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [19] M.-T. Luong, H. Pham, and C. Manning, “Effective approaches to attention-based neural machine translation,” _arXiv preprint arXiv:1508.04025_ , 2015.
  * [20] A. Graves, G. Wayne, and I. Danihelka, “Neural turing machines,” _arXiv preprint arXiv:1410.5401_ , 2014.
  * [21] P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens, “Stand-alone self-attention in vision models,” _arXiv preprint arXiv:1906.05909_ , 2019.
  * [22] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. Le, “Attention augmented convolutional networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3286–3295.
  * [23] H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for image recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 076–10 085.
  * [24] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in _International conference on machine learning_ , 2015, pp. 2048–2057.
  * [25] V. Mnih, N. Heess, A. Graves _et al._ , “Recurrent models of visual attention,” in _Advances in neural information processing systems_ , 2014, pp. 2204–2212.
  * [26] Q. Yang, R. Yang, J. Davis, and D. Nistér, “Spatial-depth super resolution for range images,” in _2007 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2007, pp. 1–8.
  * [27] D. Ferstl, C. Reinbacher, R. Ranftl, M. Rüther, and H. Bischof, “Image guided depth upsampling using anisotropic total generalized variation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2013, pp. 993–1000.
  * [28] B. Ham, M. Cho, and J. Ponce, “Robust guided image filtering using nonconvex potentials,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 1, pp. 192–207, 2017.
  * [29] X. Shen, C. Zhou, L. Xu, and J. Jia, “Mutual-structure for joint filtering,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 3406–3414.
  * [30] Q. Zhang, X. Shen, L. Xu, and J. Jia, “Rolling guidance filter,” in _European conference on computer vision_. Springer, 2014, pp. 815–830.
  * [31] A. Hosni, C. Rhemann, M. Bleyer, C. Rother, and M. Gelautz, “Fast cost-volume filtering for visual correspondence and beyond,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 2, pp. 504–511, 2012\.
  * [32] J. T. Barron and B. Poole, “The fast bilateral solver,” in _European Conference on Computer Vision_. Springer, 2016, pp. 617–632.
  * [33] K. He, J. Sun, and X. Tang, “Guided image filtering,” in _European conference on computer vision_. Springer, 2010, pp. 1–14.
  * [34] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in _Sixth international conference on computer vision (IEEE Cat. No. 98CH36271)_. IEEE, 1998, pp. 839–846.
  * [35] K. He, J. Sun, and X. Tang, “Guided image filtering,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 35, no. 6, pp. 1397–1409, 2012.
  * [36] A. Levin, D. Lischinski, and Y. Weiss, “A closed-form solution to natural image matting,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 30, no. 2, pp. 228–242, 2007.
  * [37] Y. Li, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep joint image filtering,” in _European Conference on Computer Vision_. Springer, 2016, pp. 154–169.
  * [38] H. Su, V. Jampani, D. Sun, O. Gallo, E. Learned-Miller, and J. Kautz, “Pixel-adaptive convolutional neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 166–11 175.
  * [39] R. Keys, “Cubic convolution interpolation for digital image processing,” _IEEE transactions on acoustics, speech, and signal processing_ , vol. 29, no. 6, pp. 1153–1160, 1981.
  * [40] Z. Wang, J. Chen, and S. Hoi, “Deep learning for image super-resolution: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [41] C. Dong, C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 38, no. 2, pp. 295–307, 2015.
  * [42] C. Dong, C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in _European conference on computer vision_. Springer, 2016, pp. 391–407.
  * [43] V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” _arXiv preprint arXiv:1603.07285_ , 2016.
  * [44] W. Shi, J. Caballero, F. Huszár, J. Totz, A. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1874–1883.
  * [45] P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” _arXiv preprint arXiv:1803.02155_ , 2018.
  * [46] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in _European conference on computer vision_. Springer, 2012, pp. 746–760.
  * [47] B. Kim, J. Ponce, and B. Ham, “Deformable kernel networks for joint image filtering,” _International Journal of Computer Vision_ , pp. 1–22, 2020\.
  * [48] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,” _ACM Transactions on Graphics (ToG)_ , vol. 26, no. 3, pp. 96–es, 2007.
  * [49] B. Ham, M. Cho, and J. Ponce, “Robust image filtering using joint static and dynamic guidance,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4823–4831.
  * [50] T.-W. Hui, C. C. Loy, and X. Tang, “Depth map super-resolution by deep multi-scale guidance,” in _European conference on computer vision_. Springer, 2016, pp. 353–369.
  * [51] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” 2015.
  * [52] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 1701–1708.
  * [53] X. Jia, B. D. Brabandere, T. Tuytelaars, and L. Gool, “Dynamic filter networks,” in _Advances in neural information processing systems_ , 2016, pp. 667–675.