# Residual Dense Swin Transformer for Continuous Depth-Independent Ultrasound
Imaging

###### Abstract

Ultrasound imaging is crucial for evaluating organ morphology and function,
yet depth adjustment can degrade image quality and field-of-view, presenting a
depth-dependent dilemma. Traditional interpolation-based zoom-in techniques
often sacrifice detail and introduce artifacts. Motivated by the potential of
arbitrary-scale super-resolution to naturally address these inherent
challenges, we present the Residual Dense Swin Transformer Network (RDSTN),
designed to capture the non-local characteristics and long-range dependencies
intrinsic to ultrasound images. It comprises a linear embedding module for
feature enhancement, an encoder with shifted-window attention for modeling
non-locality, and an MLP decoder for continuous detail reconstruction. This
strategy streamlines balancing image quality and field-of-view, which offers
superior textures over traditional methods. Experimentally, RDSTN outperforms
existing approaches while requiring fewer parameters. In conclusion, RDSTN
shows promising potential for ultrasound image enhancement by overcoming the
limitations of conventional interpolation-based methods and achieving depth-
independent imaging.

Index Terms—  Ultrasound imaging, Arbitrary-scale image super-resolution,
Depth-independent imaging, Non-local implicit representation

##  1 Introduction

Ultrasound imaging serves as a pivotal tool in medical diagnostics for its
non-invasive nature and real-time imaging capabilities, allowing visualization
of superficial and deep structures [1]. However, adjusting the imaging depth
presents challenges that impact image quality and field-of-view [2].

Fig. 1: An example of magnified images of abitrary scales generated by RDSTN.

Modifying the imaging depth in ultrasound requires altering the echo reception
time. Longer reception times, necessary for deeper imaging, tend to lower the
frame rate, subsequently reducing temporal resolution [3]. A shallow imaging
depth, however, may lead to interference from adjacent echo signals,
compromising image quality. Therefore, selecting the appropriate depth
threshold is crucial.

Traditionally, zoom-in operations utilizing interpolation have been employed
to counterbalance unsatisfactory image quality during depth adjustments [4].
This often results in the loss of intricate details and the emergence of
aliasing artifacts.

Addressing this challenge, our study presents the arbitrary-scale super-
resolution (ASSR) as a cutting-edge approach that offers an effective solution
within the desired depth threshold. Existing ASSR models often overlook the
unique attributes of datasets, a particularly critical aspect given the
notable long-range similarities in ultrasound images.

To address the shortcoming, we introduce the Residual Dense Swin Transformer
Network (RDSTN), which integrates a linear embedding layer, a Residual Dense
Shifted-window Transformer (RDST) encoder, and a Multilayer Perceptron (MLP)
decoder. The linear embedding layer projects the input into higher-dimensional
space for enhanced feature extraction. The RDST encoder leverages non-locality
to promote essential feature reuse. Finally, the MLP decoder maps each target
pixel coordinate to its corresponding latent code, achieving refined
reconstruction of detail. Figure 1 shows an example of magnified images
generated by RDSTN.

Fig. 2: The main pipeline of our RDSTN. RDSTN introduces non-locality and
allows for essential feature reuse, improving representation and performance.

In conclusion, our contributions are:

  * •

The first introduction of implicit neural representation-based arbitrary-scale
super-resolution to multiple organ ultrasound datasets, ensuring a versatile
and depth-independent display.

  * •

Proposing an efficient non-local encoder to model the spatial dependencies
between pixels, reducing parameters by 45 $\%$ with respect to the state-of-
the-art method.

  * •

Conducting extensive experiments to prove the capability of RDSTN in
addressing the balance issue.

##  2 Related Methods

###  2.1 Single Image Super-Resolution

Numerous single image super-resolution (SISR) methods have been proposed over
time. Among the earlier techniques, interpolation stands out, which works by
filling values between pixels to enhance image resolution. However, its major
downside is the inability to restore intricate details effectively. Recently,
deep learning-based methods have taken the spotlight in the SISR domain.
Notably, Convolutional Neural Networks (CNNs) are now widely recognized due to
their prowess in autonomously learning hierarchical features and establishing
a relationship between low-resolution (LR) and high-resolution (HR) images.
The SRCNN [5], a notable CNN-based model, leverages a three-layer CNN
structure to learn this LR to HR mapping. Various other models, such as VDSR
[6], EDSR [7], ESPCN [8], LapSRN [9], MemNet [10], RDN [11], and DBPN [12],
have carved their niche by adopting unique strategies. These include deep
architectures, sub-pixel convolutional layers, the Laplacian pyramid
framework, memory blocks, cascaded ResNet-like blocks, densely connected
residual blocks, and deep back-projection units. However, a common limitation
across these models is their lack of flexibility due to predefined upsampling
mechanisms, thereby confining their application to fixed integer-scale super-
resolution.

###  2.2 Arbitrary-Scale Image Super-Resolution

Arbitrary-scale super-resolution (ASSR) techniques are gaining traction due to
their adaptability in handling super-resolution tasks across various scaling
factors [13, 14, 15, 16, 17, 18, 19]. MetaSR [15] led the charge in CNN-based
methods for ASSR, while LIIF [16] showcased an innovative framework by
leveraging implicit neural representations to treat images as continuous
functions. Specifically, LIIF uses a Multi-Layer Perceptron (MLP) to determine
RGB values from feature maps, coordinates, and scaling factors. Building on
the foundation set by LIIF, other ASSR studies have emerged. For instance, LTE
[17] inputs coordinates into a high-dimensional Fourier feature space to
tackle spectral bias. SADN [18] augments the latent code drawn by the encoder
with multi-scale features. A-LIIF [19] addresses artifacts by capturing pixel
differences using multiple MLPs. Nevertheless, a limitation of these methods
is their dependence on convolution or fully connected layers, which curtails
the encoder’s capacity to glean global information and might bypass crucial
spatial similarities among pixels. Consequently, integrating non-locality into
the encoder is imperative.

Table 1: Quantitative comparison in terms of PSNR(dB). The evaluation is performed on the BUSI testing set. The models are trained with continuous scale sampled from $U(1,4)$. Best result of each scale is in bold. Methods | Num. of | In-distribution | Out-distribution
---|---|---|---
Parameters | ×1.6 | ×1.7 | ×1.8 | ×1.9 | ×2 | ×3 | ×4 | ×6 | ×8 | ×10
Bicubic | – | 40.21 | 39.36 | 38.88 | 38.21 | 38.68 | 33.17 | 30.40 | 26.88 | 24.86 | 23.64
EDSR-LIIF [16] | 496.4K | 43.92 | 43.06 | 42.26 | 41.50 | 40.80 | 35.80 | 32.87 | 29.42 | 27.34 | 26.04
RDN-LIIF [16] | 5.8M | 44.71 | 43.81 | 43.03 | 42.28 | 41.57 | 36.36 | 33.22 | 29.58 | 27.46 | 26.12
Unet [20] | 31.4M | 42.39 | 41.71 | 41.05 | 40.42 | 39.83 | 35.24 | 32.55 | 29.20 | 27.16 | 25.91
Resnet50 [21] | 4.1M | 42.86 | 42.07 | 41.35 | 40.62 | 39.95 | 35.17 | 32.46 | 29.12 | 27.11 | 25.87
RDSTN (ours) | 3.2M | 44.78 | 43.89 | 43.10 | 42.35 | 41.62 | 36.34 | 33.20 | 29.64 | 27.54 | 26.18
Table 2: Ablation study of RDSTN on Local Feature Fusion (LFF) and Global Feature Fusion (GFF). The evaluation is performed on the BUSI testing set, with a focus on measuring the peak signal-to-noise ratio (PSNR(dB)) to assess the performance of these strategies. The best result of each scale is in bold. Model Settings | Module | In-distribution | Out-distribution
---|---|---|---
LFF | GFF | ×1.2 | ×1.4 | ×1.6 | ×1.8 | ×2 | ×3 | ×4 | ×6 | ×8 | ×10
$S_{1}$ | ✘ | ✘ | 48.65 | 46.17 | 44.36 | 42.69 | 41.21 | 36.04 | 33.01 | 29.51 | 27.42 | 26.07
$S_{2}$ | ✘ | ✔ | 48.71 | 46.23 | 44.42 | 42.73 | 41.27 | 36.07 | 33.03 | 29.54 | 27.46 | 26.11
$S_{3}$ | ✔ | ✘ | 48.89 | 46.40 | 44.61 | 42.94 | 41.46 | 36.23 | 33.13 | 29.59 | 27.50 | 26.14
$S_{4}$ | ✔ | ✔ | 49.27 | 46.62 | 44.78 | 43.10 | 41.62 | 36.34 | 33.20 | 29.64 | 27.54 | 26.18

##  3 Model Architecture

This section introduces the proposed Residual dense swin transformer network
(RDSTN). The main pipeline of our RDSTN is shown in Figure 2.

###  3.1 Residual Dense Swin Transformer Encoder

Swin Transformer block [22] has emerged as a highly promising architecture due
to its unique capabilities in handling computation cost and non-local
characteristics. Our proposed Residual Dense Swin Transformer (RDST) encoder
is designed on the basis of swin transformer block, composed of four
hierarchical stages, each with six consecutive swin transformer blocks. The
output of each stage maintains a fixed resolution and number of channels, thus
the latent codes extracted by RDST have the same spatial resolution as the LR
counterparts. Therefore, every pixel in low-resolution images can match a
latent code in the same position. RDST assists the model to extract non-
locality in broader receptive fields at the beginning stages, while retaining
fine-grained detail at the higher stages.

A distinctive innovation in RDST is the fusion of both local and global
features, enabling the model to retain vital contextual information throughout
its processing stages. This is achieved by adeptly concatenating outputs from
each stage, thereby harnessing the rich feature spectrum across varying layers
and blocks. To foster rapid model convergence, we incorporate residual
connections with the originating image. Additionally, RDST adopts a cyclic
interplay among successive layers, a strategic maneuver ensuring that the all-
encompassing non-local information is captured with refined precision.

Specifically, global feature fusion and local feature fusion can be formulated
as:

$F_{out}=F_{in}+Conv_{1\times 1}([F_{in},F_{1},\cdots,F_{N}])$ (1)
$F_{n}=Conv_{1\times 1}([F_{n-1},STB^{D}(F_{n-1})])$ (2)

where STB denotes the swin transformer blocks. $F_{i}$ denotes the output of
the $i^{th}$ RST block $(i\in{1,\cdots,N})$. $Conv_{1\times 1}$ represents the
convolution operation with 1×1 kernel size, which is used to recover channels.

###  3.2 Local Enhanced Implicit Representation Upscaling

Building on the paradigm of LIIF [16], we introduce the Local Enhanced
Implicit Representation Upscaling (LEIRU) decoder. This refined Multi-Layer
Perceptron (MLP) is tailored to operate on the coordinates of the anticipated
scale super-resolution image, aligning each coordinate with its nearest latent
code. For a singular coordinate, denoted as $x_{q}$, its corresponding nearest
latent code is represented as $C(x_{q})$. The RGB values for every coordinate
are thus defined by:

$RGB(x_{q})=MLP([C(x_{q}),x_{q}-x^{*}])$ (3)

where MLP represents an MLP with learnable parameters, $x^{*}$ denotes the
coordinate of $C(x_{q})$.

In essence, the relative distance gauges the affinity between features and
latent codes. A closer coordinate to the latent code implies higher
similarity, gauged by the inverse of this relative distance. The underlying
premise is that coordinates within the same grid derive from the same latent
code, thereby utilizing local information. The non-local encoder’s design is
pivotal to infuse non-locality into the local decoder, optimizing the model’s
performance.

However, an issue arises when a coordinate traverses the grid boundary. At
this juncture, the MLP output can shift dramatically due to the abrupt latent
code alteration, resulting in chessboard artifacts in the output image. To
mitigate this, our module introduces a local ensemble operation. Drawing from
interpolation principles, the local ensemble harnesses the weights of the
rectangle, formed by the coordinate and its four adjacent latent codes,
executing a nearest neighbor interpolation. Mathematically, this is
articulated as:

$LEIRU(x_{q})=\sum_{x_{i}\in{grid}}w_{x_{i}}RGB(x_{i})$ (4)

where $w_{x_{i}}$ represents the weights for each coordinate $x_{i}$.
$RGB(x_{i})$ represents the corresponding RGB values computed by the MLP for
the coordinate $x_{i}$.

Fig. 3: Visual comparison between methods. RDSTN achieves better continuity in
texture and has the fewest white noise points compared to other methods.

##  4 Experiment Results

We utilize the BUSI dataset [23], which comprises 779 high-resolution breast
ultrasound images, to train our model. We allocate 80$\%$ of these images for
training, with the remainder designated for testing. In line with prior
research, our primary evaluation metric is the peak signal-to-noise ratio
(PSNR).

###  4.1 Comparison with other methods

We benchmark our RDSTN against other implicit representation techniques, such
as EDSR-LIIF, RDN-LIIF [16], and the conventional bicubic interpolation [24].
Table 1 showcases the qualitative results. Across various scales, both within
and beyond distribution, our method surpasses contemporary techniques while
utilizing fewer parameters. When tested on low-resolution images tainted with
Gaussian noise, the visual evidence in Figure 3 reveals our method’s
superiority in minimizing white noise spots and maintaining texture
continuity. Therefore, our approach not only excels in standard metrics but
also demonstrates impressive noise resistance.

###  4.2 Ablation Study

To further validate the efficacy of our model, we perform an ablation study on
global and local feature fusion strategies. Our experiments indicate that
setting $S_{4}$ yields the best results among all the tested scales, followed
by $S_{3}$, $S_{2}$, and $S_{1}$, respectively. This underscores the
effectiveness of our designed local and global feature fusion, with detailed
results provided in Table 2. By integrating both global and local feature
fusion, our network enhances its ability to capture non-local features while
simultaneously reducing computational costs.

Table 3: Generalization test of RDSTN on out-distribution dataset. The evaluation is performed on the 2023 MICCAI ultrasound enhancement challenge dataset, we use PSNR(dB) to assess the performance. The best result is in bold. Method | scale
---|---
×1.6 | ×1.7 | ×1.8 | ×1.9 | ×2
train: BUSI [23], test: MICCAI USenhance breast
Bicubic | 34.28 | 33.52 | 31.55 | 31.23 | 31.66
EDSR-LIIF | 35.30 | 34.55 | 33.63 | 33.07 | 32.42
RDN-LIIF | 35.12 | 34.41 | 33.56 | 33.13 | 32.47
RDSTN (ours) | 35.35 | 34.62 | 33.74 | 33.23 | 32.59
train: BUSI [23], test: MICCAI USenhance thyroid
Bicubic | 38.17 | 37.14 | 34.91 | 34.26 | 34.81
EDSR-LIIF | 39.87 | 38.73 | 37.72 | 36.86 | 36.07
RDN-LIIF | 39.91 | 38.77 | 37.76 | 36.90 | 36.10
RDSTN (ours) | 39.99 | 38.81 | 37.83 | 36.96 | 36.14
train: BUSI [23], test: MICCAI USenhance carotid
Bicubic | 38.11 | 37.14 | 34.95 | 34.37 | 34.84
EDSR-LIIF | 40.05 | 38.96 | 37.90 | 37.07 | 36.21
RDN-LIIF | 40.08 | 38.98 | 37.89 | 37.11 | 36.28
RDSTN (ours) | 40.16 | 39.07 | 37.99 | 37.17 | 36.30

###  4.3 Generalization

To further assess the generalization capability of our model, we train it on
breast ultrasound images and subsequently test on different tissues, including
thyroid and carotid arteries. The results, present in Table 3, show that our
model surpasses other methods, highlighting its strong generalization
potential. Thus, our model can deliver exceptional results on expansive
samples even when trained on a small, focused dataset, underscoring its
adaptability in practical applications.

##  5 Conclusion

Our advanced RDSTN effectively tackles the challenges associated with long-
range modeling and non-local feature extraction in arbitrary-scale super-
resolution. It streamlines the delicate balance between image quality and
field-of-view, showcasing enhanced noise suppression capabilities. Testing
reveals that RDSTN performs competitively in both metrics and visual quality
compared to other methods, yet utilizes fewer parameters. Through RDSTN, we
can adeptly navigate continuous imaging at suitable depth thresholds.

##  6 Acknowledgments

We wish to extend our heartfelt appreciation to the organizers of the 2023
Ultrasound Image Enhancement (USenhance) Challenge for availing the dataset
pivotal for our generalization examination. Dataset can be accessed via the
official 2023 USenhance portal: https://ultrasoundenhance2023.grand-
challenge.org/.

## References

  * [1] Carmel M. Moran and Adrian J. W. Thomson,  “Preclinical ultrasound imaging—a review of techniques and imaging applications,”  Frontiers in Physics, vol. 8, 2020.
  * [2] Giner Maslebu, Kusworo Adi, and Suryono,  “Effect of gain changing to maximum visualization depth on diagnostic ultrasound b-mode image,”  International Journal of Applied Engineering Research, vol. 10, pp. 33449–33452, 08 2015.
  * [3] Gowthaman Gunabushanam and Leslie M. Scoutt,  “Ultrasound image optimization for the interventional radiologist,”  Techniques in Vascular and Interventional Radiology, vol. 24, no. 3, pp. 100766, 2021.
  * [4] Zander D, Hüske S, Hoffmann B, Cui XW, Dong Y, Lim A, Jenssen C, Löwe A, Koch JBH, and Dietrich CF,  “Ultrasound image optimization (”knobology”): B-mode,”  Ultrasound Int Open, vol. 6, no. 1, 08 2020.
  * [5] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang,  “Image super-resolution using deep convolutional networks,” 2015.
  * [6] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee,  “Accurate image super-resolution using very deep convolutional networks,” 2016.
  * [7] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee,  “Enhanced deep residual networks for single image super-resolution,” 2017.
  * [8] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang,  “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” 2016.
  * [9] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang,  “Deep laplacian pyramid networks for fast and accurate super-resolution,” 2017.
  * [10] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu,  “Memnet: A persistent memory network for image restoration,” 2017.
  * [11] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu,  “Residual dense network for image super-resolution,” 2018.
  * [12] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita,  “Deep back-projection networks for super-resolution,” 2018.
  * [13] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy,  “Recovering realistic texture in image super-resolution by deep spatial feature transform,”  in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 606–615.
  * [14] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli,  “Singan: Learning a generative model from a single natural image,”  in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 4569–4579.
  * [15] Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun,  “Meta-sr: A magnification-arbitrary network for super-resolution,” 2019.
  * [16] Yinbo Chen, Sifei Liu, and Xiaolong Wang,  “Learning continuous image representation with local implicit image function,” 2021.
  * [17] Jaewon Lee and Kyong Hwan Jin,  “Local texture estimator for implicit representation function,” 2022.
  * [18] Hanlin Wu, Ning Ni, and Libao Zhang,  “Scale-aware dynamic network for continuous-scale super-resolution,” 2021.
  * [19] Hongwei Li, Tao Dai, Yiming Li, Xueyi Zou, and Shu-Tao Xia,  “Adaptive local implicit image function for arbitrary-scale super-resolution,” 2022.
  * [20] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,” 2015.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,” 2015.
  * [22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo,  “Swin transformer: Hierarchical vision transformer using shifted windows,” 2021.
  * [23] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy,  “Dataset of breast ultrasound images,”  Data in Brief, vol. 28, pp. 104863, 2020.
  * [24] R. Keys,  “Cubic convolution interpolation for digital image processing,”  IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 29, no. 6, pp. 1153–1160, 1981.