# Accelerating Cardiac MRI Reconstruction with CMRatt: An Attention-Driven
Approach

Anam Hashmi  ML-Labs
Dublin City University
Dublin, Ireland
<EMAIL_ADDRESS>Julia Dietlmeier  Insight SFI Research Centre for
Data Analytics
Dublin City University
Dublin, Ireland
<EMAIL_ADDRESS>Kathleen M. Curran  School of Medicine
University College Dublin
Dublin, Ireland
<EMAIL_ADDRESS>Noel E. O’Connor  Insight SFI Research Centre for Data
Analytics
Dublin City University
Dublin, Ireland
<EMAIL_ADDRESS>
###### Abstract

Cine cardiac magnetic resonance (CMR) imaging is recognised as the benchmark
modality for the comprehensive assessment of cardiac function. Nevertheless,
the acquisition process of cine CMR is considered as an impediment due to its
prolonged scanning time. One commonly used strategy to expedite the
acquisition process is through k-space undersampling, though it comes with a
drawback of introducing aliasing effects in the reconstructed image. Lately,
deep learning-based methods have shown remarkable results over traditional
approaches in rapidly achieving precise CMR reconstructed images. This study
aims to explore the untapped potential of attention mechanisms incorporated
with a deep learning model within the context of the CMR reconstruction
problem. We are motivated by the fact that attention has proven beneficial in
downstream tasks such as image classification and segmentation, but has not
been systematically analysed in the context of CMR reconstruction. Our primary
goal is to identify the strengths and potential limitations of attention
algorithms when integrated with a convolutional backbone model such as a
U-Net. To achieve this, we benchmark different state-of-the-art spatial and
channel attention mechanisms on the CMRxRecon dataset and quantitatively
evaluate the quality of reconstruction using objective metrics. Furthermore,
inspired by the best performing attention mechanism, we propose a new, simple
yet effective, attention pipeline specifically optimised for the task of
cardiac image reconstruction that outperforms other state-of-the-art attention
methods. The layer and model code will be made publicly available.

###### Index Terms:

Cardiac MRI reconstruction, undersampled k-space, CMRxRecon dataset, attention
mechanisms, U-Net.

##  I Introduction

Cardiac magnetic resonance (CMR) imaging is used as a fundamental clinical
tool for the comprehensive evaluation of cardiovascular diseases. CMR imaging
is widely acknowledged for its non-invasiveness and its ability to generate
high-resolution images without resorting to ionizing radiation [1]. However,
there is a major drawback associated with the MRI acquisition process which is
their prolonged scanning time. To address this problem i.e. to reduce scanning
times, undersampling in k-space (Fourier domain) is often used, though it
leads to aliasing artifacts in the reconstructed image.

However, reconstructing CMR images from undersampled k-space poses a
significant challenge with traditional approaches like Compressed Sensing (CS)
[2] being computationally time consuming. In contrast, deep learning (DL)
based models have marked a transformative shift in accelerated CMR
reconstruction and have shown superior performance to traditional methods [3].
Therefore, the construction of a robust deep Convolutional Neural Network
(CNN) is critical for achieving enhanced outcomes. To achieve this, numerous
lightweight plug-and-play attention modules have been designed to efficiently
capture global context information and suppress irrelevant features and thus
enhance the representational properties of CNNs. These attention modules are
designed to seamlessly integrate into a CNN architecture, ensuring minimal
computational overhead.

In general, attention mechanisms, often referred to as feature recalibration
modules, inspired by human visual perception, represent a valuable addition to
deep learning models as they excel in amplifying the crucial image features
while attenuating the irrelevant ones [2]. However, navigating the existing
research landscape concerning the unified performance analysis of attention in
medical imaging is a challenging task. The common trend in the majority of the
state-of-the-art attention studies present performance gains in terms of
classification accuracy metrics or using top-1 and top-5 error rates,
underscoring that these modules were specifically designed for classification,
object detection, and segmentation tasks [4, 5]. Moreover, the datasets
commonly used in these studies are typically not from the medical imaging
domain. Instances of such datasets include ImageNet for classification, COCO
for object detection and Kinetics for video classification [6].

Recently, a new large-scale public CMRxRecon dataset [7] including multi-
contrast, multi-view and multi-slice CMR imaging data from 300 subjects has
been introduced to the research community at the CMRxRecon reconstruction
challenge hosted at MICCAI 2023. The top-performing teams have developed
pipelines integrating a combination of physics-based, deep learning-based and
attention-based approaches [8, 3]. Some studies employed modified 2D U-Net
networks [9] with attention mechanisms [8, 10]. However, the addition of
attention mechanisms lacks supporting performance analysis and we argue that
integrating task-specific attention modules could further improve the results.
Thereby, we conclude that there is a timely need for a performance analysis
study on attention methods in the image reconstruction task, specifically
focusing on medical MRI datasets and considering objective image
reconstruction quality metrics such as Peak Signal to Noise Ratio (PSNR), Mean
Square Error (MSE) and Structural Similarity Index Measure (SSIM) [11].

This observation has motivated us to conduct the first systematic study on the
addition of attention mechanisms with a well-known deep learning model for the
CMR reconstruction problem. To the best of our knowledge,this is the first
study comprehensively exploring different state-of-the-art attention modules
in the context of CMR reconstruction. To sum up, the main contributions of our
research are:

  * •

We conduct extensive experiments to benchmark different state-of-the-art
attention blocks by integrating them into a U-Net based CMR reconstruction
model on the CMRxRecon dataset.

  * •

We determine the attention modules that most significantly enhance the
performance of the reconstruction task by ranking them based on the SSIM
metric.

  * •

Inspired by the best-performing algorithm, we propose a new, simple yet
effective, attention pipeline for the CMR reconstruction task. Our proposed
pipeline outperforms the state-of-the-art attention blocks.

##  II Related Research

Recently, several attention modules have been proposed to capture rich global
feature relationships in CNNs. These attention modules can be broadly
categorised into channel attention, spatial attention, temporal attention and
branch attention [12]. In this work, we focus on state-of-the-art (SOTA)
channel, spatial and hybrid attention mechanisms, integrating both channel and
spatial components. One representative module is Squeeze-and-Excitation (SE)
attention [4], designed to capture cross-channel relationships and
recalibrates feature responses on a channel-wise basis. Following SE
attention, CBAM [13] and BAM [14] were proposed to refine feature maps by
integrating both channel and spatial attentions. Another attention network
sharing similar structure to SE called Global-Context Networks (GC-Net) [15]
was proposed. GC-Net aimed at capturing long-range dependencies and global
context features more effectively by employing non-local blocks. Following the
advance of style transfer with CNNs, a Style-based Recalibration Module (SRM)
[16] that leverages style information to reweight the feature maps was
proposed. To reduce the parameters introduced by the attention mechanisms, a
parameter-free Gaussian Context Transformer (GCT) [5] attention module was
introduced, utilizing a Gaussian function to capture global contextual feature
excitation. A recent and computationally inexpensive module, Attention Bias
(AB) [17], was proposed, utilizing the Hadamard product for recalibrating the
feature maps. In contrast to the above mentioned channel and spatial attention
modules, a parameter free block, SimAM [18] was designed to calculate 3-D
attention weights in a feature map. This attention block is based on the
property of spatial suppression and uses an energy function for calculating
feature weights. A trending attention mechanism is Self-attention [19], also
known as intra-attention, which effectively captures long-range dependencies.
The performance of transformers relies significantly on the integration of
stacked self-attention blocks. However, all the above referenced papers
conducted experiments on their attention blocks in the context of
classification, segmentation and object detection tasks. Only a very limited
number of studies have reported incorporation of attention blocks in their
U-Net-based networks for the task of cardiac MRI reconstruction from
undersampled k-space. Some of the studies to do so include [8, 20, 10].

##  III Network Architecture

This paper is an extension of our previous study [10] for CMR reconstruction
with an objective to delve deeper into the integration and performance
analysis of attention mechanisms and further it. Therefore, a vanilla U-Net
architecture [9] serves as the baseline model for our investigation.

In this study, we worked with 2D slices and built a 2D U-Net with four encoder
and four decoder blocks connected through a bridge section. Each encoder
subnetwork of U-Net consisted of two $3\times 3$ convolutions followed by a
batch normalization layer and a ReLU (Rectified Linear Unit) activation
function, referred to as the conv_block. The ReLU output functioned as the
skip connection for the corresponding decoder block. To regularize our model,
we also added dropout layers with dropout probability of 0.25 in each of four
encoder blocks which is followed by a $2\times 2$ max pooling layer. In the
initial downsampling stage, 32 feature maps were employed. The decoder block
mirrors the encoder block in reverse and consisted of the transposed
convolution layer, concatenated with the corresponding skip connection from
the encoder block, and finally followed by the above mentioned conv_block.

TABLE I:  Our experimental results of benchmarking state-of-the-art attention mechanisms on the CMRxRecon dataset. Methods are ranked according to the SSIM metric. Computational overhead is given in terms of model parameters. Method | Parameters | Computational overhead |  PSNR $\uparrow$ |  MSE $\downarrow$ |  SSIM $\uparrow$
---|---|---|---|---|---
Baseline (without attention) | see Section III | 0 | 36.20684 | 0.00028783 | 0.92451
SimAM [18] | lambda=1e-4 | 0 | 37.04925 | 0.00025836 | 0.94432
SE [4] | reduction=16 | 119,936 | 37.28633 | 0.00023553 | 0.94297
CAB [8] | reduction=4, kernel size=3 | 17,750,529 | 36.75172 | 0.00030422 | 0.94206
GCT [5] | c=2 | 0 | 36.58741 | 0.00026953 | 0.94088
CBAM [13] | reduction=16, kernel size=7 | 122,092 | 37.62119 | 0.00023203 | 0.93845
AB [17] | None | 3,424 | 34.36335 | 0.00042996 | 0.93570
GC-Net [15] | reduction=16 | 127,448 | 36.67425 | 0.00026559 | 0.90888
BAM [14] | reduction=16 | 259,632 | 34.52592 | 0.00039782 | 0.90293
SRM [16] | None | 13,696 | 33.27101 | 0.00053154 | 0.90292

###  III-A Comparative Analysis of State-of-the-art Attention Mechanisms: A
Benchmarking Approach

We incorporated various attention modules into our baseline U-Net model and
conducted a comprehensive comparative analysis of their performances. The
findings from our benchmarking study, conducted on nine state-of-the-art
attention modules using the CMRxRecon dataset, are presented in Table 1. The
experimental results highlight that the SimAM attention module [18] performed
the best for the CMR reconstruction task, based on the SSIM metric, which is
considered more robust than PSNR and MSE [21]. The SimAM attention module
integrated with the baseline model demonstrated superior performance to the
baseline alone.

Figure 1: Proposed CMRatt attention framework, comprising SimAM attention
block, L2-normalization layer, and a modified Hadamard attention.

###  III-B Proposed Attention Pipeline

Our experimental results demonstrate the superior performance of the SimAM
[18], making it the ideal choice as a key component in our proposed pipeline
CMRatt illustrated in Fig.1. In a nutshell, our design novelty is to combine
SimAM with a L2 normalization layer and a modified Hadamard block [22]. SimAM
is a very lightweight attention module based on a neuroscience theory of
spatial suppression [23]. The SimAM block effectively generates 3D attention
weights by optimizing an energy function to determine the importance of each
neuron. To estimate the importance of each neuron, linear separability between
one target neuron and other neurons is calculated as:

$e_{t}^{*}=\frac{4(\hat{\sigma}^{2}+\lambda)}{(t-\hat{\mu})^{2}+2\hat{\sigma}^{2}+2\lambda}$
(1)

In Equation (1), $t$ represents the target neuron, $\hat{\sigma}^{2}$
represents the variance of other neurons except $t$, $\hat{\mu}$ represents
the mean of other neurons except $t$, and $\lambda$ is the coefficient.
According to this equation, neurons with lower energy values show greater
separability from other neurons, signifying their higher importance [24].
Sometimes, the values of the energy function can be too large and to restrict
those, a sigmoid function is used. The final feature refinement equation is:

$\tilde{X}=\text{sigmoid}\left(\frac{1}{E}\right)\odot X$ (2)

In Equation (2), $E$ represents the sum of energy functions $e_{t}^{*}$ across
channel and spatial dimensions.

Normalization layers have become a fundamental component of modern deep neural
networks [25]. An important addition to our attention pipeline is the L2
normalization layer that serves the purpose of promoting stability in the
model’s learning process by stabilizing the gradients. Moreover, we integrate
a different form of self-attention to our proposed pipeline by taking
inspirations from Hadamard attention [22] and Criss-Cross attention [26]. The
aim of our proposed pipeline is to enhance the model’s ability to capture
nuanced and subtle patterns inherent in cardiac MRI images, thus contributing
to improved reconstructions. Therefore, incorporating a self-attention block
becomes essential to capture rich contextual relationships. However, self
attention blocks are computationally expensive with both time and space
requirements scaling quadratically with the length of the input [27]. Due to
memory limitations, we could not employ a transfomer self-attention block [19]
into our pipeline. Instead, we opted for a simpler Hadamard attention block
with minute changes. This block consists of two $1\times 1$ convolutional
kernels to enhance the relevant features and also to reduce the number of
channels, thus aiding in reducing the overall computational complexity of the
network. Following this, two Sigmoid activation functions are applied owing to
their better expressive ability [28], succeeded by the Hadamard product
between the two. Subsequently, a $1\times 1$ convolution is applied and the
resultant is multiplied by the input. The final output is obtained by adding
this to the connection from the input.

The Proposed CMRatt is integrated after every convolutional layer and within
the skip connections in the baseline U-Net model.

##  IV Methodology

The released CMRxRecon dataset [7] contained a total of 300 healthy volunteers
from a single center (raw k-space data obtained on three Tesla magnets). The
dataset is divided into 120 volunteers for training , 60 for validation, and
120 for test data. The training set for cine reconstruction included fully
sampled k-space single-coil data and undersampled k-space data with
acceleration factors ×4, ×8 and ×10. We performed the experiments on the
training set of CMRxRecon using only the long-axis fully sampled (Ground
Truth) and undersampled single-coil data with the acceleration factor of ×10
as these contained the strongest aliasing artifacts.

###  IV-A Preprocessing

To preprocess the training set which has been provided in the frequency domain
as k-space data, an Inverse Fast Fourier Transform (IFFT) was applied to
obtain the images in the spatial domain. Subsequently, we further processed
the single-channel 2D grayscale images, and rescaled them by the maximum image
intensity. Additionally, a linear scaling transform was applied to ensure that
the pixel intensities fall in the range [0, 1]. We uniformly resized all IFFT-
reconstructed images to dimensions of 256 × 256 pixels to serve as an input to
the U-Net model.

###  IV-B Implementation Details

The processing pipeline was implemented in Python 3.11.5 and the PyTorch
framework. All experiments were conducted on a GTX4090 GPU with 24 GB of
memory. For model training, we used an AdamW optimizer with a learning rate of
0.001. The training was done with a batch size of 2, and all models were
trained for 300 epochs to ensure convergence. The Mean Square Error (MSE) was
used as the main optimization objective and no data augmentation was used in
the training phase.

###  IV-C Evaluation Metrics

Mean Square Error (MSE) is the most widely used image quality assessment
metric with better values closer to zero. The MSE between two images $\hat{y}$
and $y$ are defined as follows:

$\text{MSE}=\frac{1}{MN}\sum_{m=1}^{M}\sum_{n=1}^{N}[\hat{y}(n,m)-y(n,m)]^{2}$
(3)

PSNR is defined as the ratio between the maximum possible signal power and the
power of the distorting noise. This ratio between two images is computed in
the decibel form as [21]:

$\text{PSNR}=10\cdot\log_{10}\left(\frac{\text{peakval}^{2}}{\text{MSE}}\right)$
(4)

where peakval is the maximum possible intensity value.

The Structural Similarity Index Measure (SSIM) is a perception-based metric
that captures the mutual dependencies among adjacent pixels to assess the
similarity of two images, such as brightness, contrast and structural
properties:

$\text{SSIM}(y,\hat{y})=\frac{(2\mu_{y}\mu_{\hat{y}}+c_{1})(2\sigma_{y\hat{y}}+c_{2})}{(\mu_{y}^{2}+\mu_{\hat{y}}^{2}+c_{1})(\sigma_{y}^{2}+\sigma_{\hat{y}}^{2}+c_{2})}$
(5)

where $\mu_{y}$ and $\mu_{\hat{y}}$ represent the mean values of the model
output $\hat{y}$ and the target output y, $\sigma_{y}$ and $\sigma_{\hat{y}}$
denote the corresponding pixel variance values and $\sigma_{y\hat{y}}$ is the
covariance.

###  IV-D Experimental results

To allow fast experimentation, we do not use all training data provided but
construct the training subset S1 (single coil) as follows: we first process
all timeframes and slices in the supplied cine lax.mat files [10]. We further
randomly subsample the full training set and select 1,000 LAX images for
training and 200 LAX images for testing. Our experimental benchmarking results
were shown in Table 1. For all attention methods benchmarked, we used the
standard value of the parameters from their papers.

Figure 2: The reconstruction results and normalized absolute error maps of
different methods of Patient 002 with acceleration factor $\times 10$. The
last two rows show the zoomed area. Proposed method can recover fine details
on reconstructed images as marked by arrows.

Without bells and whistles, our proposed CMRatt mechanism significantly
outperformed the baseline and the SimAM in all three metrics and we achieved
PSNR = 39.95486, MSE = 0.0001798 and SSIM = 0.96534. Moreover, in order to
obtain further improvements in the reconstruction pipeline, consideration
could be given to increasing the number of encoder-decoder blocks and the
input resolution (e.g. $512\times 512$) in the U-Net model. Our CMRatt, with a
computational overhead of 969,870 model parameters, demonstrated
competitiveness and outperformed the other state-of-the-art attention blocks
including the Channel Attention Block (CAB) which has a substantial
computational overhead of 17,750,529 parameters, used by the winners of
CMRxRecon challenge [8] in all the three quantitative metrics. The qualitative
results are shown in Fig.2. The qualitative comparisons demonstrate that our
approach is capable of capturing fine details in the reconstructed images.

##  V Conclusion

This study systematically explored and benchmarked state-of-the-art attention
mechanisms in the context of CMR reconstruction. Moreover, this study
investigated the computational overheads associated with each attention
module, thus addressing concerns related to memory limitations. Furthermore,
we proposed a new CMRatt attention pipeline which outperformed the baseline,
SimAM and the most recent CAB attention mechanisms. In conclusion, our study
emphasized the potential of attention mechanisms to significantly improve the
quality of CMR reconstruction tasks. These findings provide valuable insights
for advancing DL approaches in the field of CMR reconstruction.

## Acknowledgment

This publication has emanated from research conducted with the financial
support of Science Foundation Ireland under Grant numbers 18/CRT/6183 and
12/RC/2289_P2.

## References

  * [1] E. Larose and et al., “Cardiovascular magnetic resonance for the clinical cardiologist,” _Canadian J. of Cardiology_ , vol. 23, pp. 84B–88B, 2007.
  * [2] P. Guo and et al., “ReconFormer: Accelerated MRI reconstruction using recurrent transformer,” _IEEE TMI_ , vol. 43, no. 1, pp. 582–593, 2024.
  * [3] G. Yiasemis and et al., “Deep cardiac MRI reconstruction with ADMM,” _arXiv preprint arXiv:2310.06628_ , 2023.
  * [4] J. Hu and et al., “Squeeze-and-excitation networks,” in _CVPR_ , 2018, pp. 7132–7141.
  * [5] D. Ruan and et al., “Gaussian context transformer,” in _CVPR_ , 2021, pp. 15 129–15 138.
  * [6] Z. Yang and et al., “Gated channel transformation for visual recognition,” in _CVPR_ , 2020, pp. 11 794–11 803.
  * [7] C. Wang and et al., “CMRxRecon: An open cardiac MRI dataset for the competition of accelerated image reconstruction,” _arXiv preprint arXiv:2309.10836_ , 2023.
  * [8] B. Xin and et al., “Fill the k-space and refine the image: Prompting for dynamic and multi-contrast MRI reconstruction,” _arXiv preprint arXiv:2309.13839_ , 2023.
  * [9] O. Ronneberger and et al., “U-net: Convolutional networks for biomedical image segmentation,” in _MICCAI_. Springer, 2015, pp. 234–241.
  * [10] J. Dietlmeier and et al., “Cardiac MRI reconstruction from undersampled k-space using double-stream IFFT and a denoising GNA-UNET pipeline,” _MICCAI 2023, STACOM CMRxRecon Proceedings_ , 2023.
  * [11] A. Mason and et al., “Comparison of objective image quality metrics to expert radiologists’ scoring of diagnostic quality of mr images,” _IEEE Trans. on Med. Imaging_ , vol. 39, no. 4, pp. 1064–1072, 2019.
  * [12] M.-H. Guo and et al., “Attention mechanisms in computer vision: A survey,” _Computational visual media_ , vol. 8, no. 3, pp. 331–368, 2022.
  * [13] S. Woo and et al., “CBAM: Convolutional block attention module,” in _ECCV_ , 2018, pp. 3–19.
  * [14] J. Park and et al., “BAM: Bottleneck attention module,” _arXiv preprint arXiv:1807.06514_ , 2018.
  * [15] Y. Cao and et al., “GCNET: Non-local networks meet squeeze-excitation networks and beyond,” in _ICCVW_ , 2019, pp. 0–0.
  * [16] H. Lee and et al., “SRM: A style-based recalibration module for convolutional neural networks,” in _ICCV_ , 2019, pp. 1854–1862.
  * [17] S. R. Klomp and et al., “Performance-efficiency comparisons of channel attention modules for resnets,” _Neural Processing Letters_ , pp. 1–17, 2023.
  * [18] L. Yang and et al., “SimAM: A simple, parameter-free attention module for convolutional neural networks,” in _ICML_. PMLR, 2021, pp. 11 863–11 874.
  * [19] A. Vaswani and et al., “Attention is all you need,” _NIPS_ , vol. 30, 2017.
  * [20] Q. Huang and et al., “MRI reconstruction via cascaded channel-wise attention network,” in _ISBI_. IEEE, 2019, pp. 1622–1626.
  * [21] U. Sara and et al., “Image quality assessment through FSIM, SSIM, MSE and PSNR — a comparative study,” _Journal of Computer and Communications_ , vol. 7, no. 3, pp. 8–18, 2019.
  * [22] Y. Jin and et al., “Simplified inception module based Hadamard attention mechanism for medical image classification,” _Journal of Computer and Communications_ , vol. 11, no. 6, pp. 1–18, 2023.
  * [23] B. S. Webb and et al., “Early and late mechanisms of surround suppression in striate cortex of macaque,” _Journal of Neuroscience_ , vol. 25, no. 50, pp. 11 666–11 675, 2005.
  * [24] Z. Cai and et al., “Repvgg-simam: An efficient bad image classification method based on RepVGG with simple parameter-free attention module,” _Applied Sciences_ , vol. 13, no. 21, p. 11925, 2023.
  * [25] J. Shao and et al., “Is normalization indispensable for training deep neural network?” _NIPS_ , vol. 33, pp. 13 434–13 444, 2020.
  * [26] Z. Huang and et al., “CCNET: Criss-cross attention for semantic segmentation,” in _ICCV_ , 2019, pp. 603–612.
  * [27] F. D. Keles and et al., “On the computational complexity of self-attention,” in _International Conference on Algorithmic Learning Theory_. PMLR, 2023, pp. 597–619.
  * [28] Y. Dai and et al., “Attention as activation,” in _ICPR_. IEEE, 2021, pp. 9156–9163.