# Reconstruction-driven motion estimation for motion-compensated MR CINE
imaging

Jiazhen Pan  Wenqi Huang  Daniel Rueckert  Thomas Küstner† and Kerstin
Hammernik† This work was supported in part by the European Research Council
(Grant Agreement no. 884622).$\dagger$ Thomas Küstner and Kerstin Hammernik
contributed equally.Jiazhen Pan, Wenqi Huang, Daniel Rueckert and Kerstin
Hammernik are with Klinikum Rechts der Isar, Technical University of Munich,
Munich, Germany (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>k.hammernik@tum.de).Thomas Küstner is with the
University Hospital of Tuebingen, Department of Diagnostic and Interventional
Radiology, Medical Image and Data Analysis (MIDAS.lab), Tübingen, Germany
(e-mail: thomas.kuestner@med.uni-tuebingen.de).Daniel Rueckert and Kerstin
Hammernik are also with the Department of Computing, Imperial College London,
London, United Kingdom.

###### Abstract

In cardiac CINE, motion-compensated MR reconstruction (MCMR) is an effective
approach to address highly undersampled acquisitions by incorporating motion
information between frames. In this work, we propose a deep learning-based
framework to address the MCMR problem efficiently. Contrary to state-of-the-
art (SOTA) MCMR methods which break the original problem into two sub-
optimization problems, i.e. motion estimation and reconstruction, we formulate
this problem as a single entity with one single optimization. We discard the
canonical motion-warping loss (similarity measurement between motion-warped
images and target images) to estimate the motion, but drive the motion
estimation process directly by the final reconstruction performance. The
higher reconstruction quality is achieved without using any smoothness loss
terms and without iterative processing between motion estimation and
reconstruction. Therefore, we avoid non-trivial loss weighting factors tuning
and time-consuming iterative processing. Experiments on 43 in-house acquired
2D CINE datasets indicate that the proposed MCMR framework can deliver
artifact-free motion estimation and high-quality MR images even for imaging
accelerations up to 20x. The proposed framework is compared to SOTA non-MCMR
and MCMR methods and outperforms these methods qualitatively and
quantitatively in all applied metrics across all experiments with different
acceleration rates.

###### Index Terms:

motion-compensated reconstruction, Cardiac CINE reconstruction, deep learning,
reconstruction-driven registration / motion estimation

##  I Introduction

Figure 1: The difference between the proposed MCMR framework (bottom) and the
conventional MCMR work (top) is shown. The conventional approaches divide the
original MCMR problem into two sub-optimization problems: motion estimation
and reconstruction. Its motion estimation is optimized by minimizing the
intermediate motion-warping loss (brightness similarity measurement between
motion-warped images and target images) and if deep learning is used, the
motion prediction back-propagation is only exerted on the motion estimation
part. In contrast, we develop a deep learning-based framework that predicts
the motion from the perspective of our ultimate goal: reconstruction. We
discard using any intermediate motion-warping loss. The back-propagation is
performed through the whole pipeline and reconstruction-driven motion
estimation is established.

Cine cardiac magnetic resonance imaging (CMR) serves as a versatile tool for
characterizing cardiac morphology and assessing cardiac function. Quantitative
indicators such as volume and ejection fraction can be calculated from CMR and
an evidence-based diagnosis of cardiovascular disease can be accomplished. A
reconstruction with high spatial and temporal resolutions across the whole
cardiac sequence is an indispensable prerequisite for CMR. In this context, a
short scan time, ideally within a single breath-hold, is preferred to
alleviate the patients’ scan discomfort and prevent potential image artifacts
due to patient motion. To this aim, only a limited amount of k-space
(frequency domain) data can be collected for every temporal frame, violating
the Nyquist–Shannon sampling theorem and resulting in aliasing artifacts in
the image domain. In the past decade, Parallel Imaging [1, 2] and Compressed
Sensing [3, 4] were introduced in CMR, enabling shorter scan time and improved
reconstruction performance. However, reconstruction performance can be further
improved if adequate spatial-temporal information is shared along the cardiac
cycle. This information is linked by the cardiac motion, which bridges every
single frame of the whole cardiac sequence and serves as the key to successful
reconstruction. A straightforward way to leverage this motion information in
CMR reconstruction is to use motion-compensated MR reconstruction (MCMR) [5]
in which the cardiac motion has to be estimated. However, precise cardiac
motion estimation remains a challenging problem due to the non-rigid nature of
the cardiac motion, especially in the case of accelerated imaging where motion
has to be estimated from undersampled data.

To circumvent the non-trivial tasks of cardiac motion estimation, different
CMR reconstruction methods sidestep the motion estimation and aim to exploit
spatio-temporal redundancies. The works of [6, 7] suggested disentangling the
original reconstruction problem into a low-rank and a sparse component and
these two sub-optimizations are carried out jointly. However, the preservation
of dynamic information crucially depends on the optimization of the sparse
component and the implementation of soft thresholding can incur information
loss. Moreover, deep learning reconstructions were proposed e.g. [8, 9] that
unroll the dynamic MR optimization process with a spatio-temporal
regularization. In this case, multiple unrolled gradient descent steps have to
be executed, giving rise to the training difficulty of the network and
processing time in both training and testing. Other methods [10, 11, 12]
utilized the $k-t$ domain to leverage the spatio-temporal redundancies to
ameliorate the dynamic reconstruction. Whereas all these methods endeavor to
extract the spatio-temporal correlation implicitly, there is no guarantee that
the correlation of every cardiac frame is fully exploited. On the contrary,
MCMR leverages the estimated cardiac motion to explicitly share cardiac
spatio-temporal information.

A high-quality MCMR can be performed if the cardiac motion can be estimated
precisely over the whole cardiac cycle. Therefore, the selection of a proper
motion estimation/registration approach plays a decisive role in MCMR.
Conventional registration methods based on B-spline [13, 14] or diffusion
method [15] can be employed as motion estimators in MCMR. These methods can
provide meaningful registration results but demand enormous computing time in
the order of hours for a single CMR sequence. Furthermore, hyper-parameter
tuning for these methods [13, 14] is also a non-trivial task, hindering their
implementation in clinical practice. Lately, learning-based
registration/motion estimation approaches have been introduced into medical
imaging [16, 17, 18] and embodied in the application of cardiac motion
estimation [19, 20, 21]. These methods accelerate the registration time from
hours to seconds by leveraging a trained neural network during inference and
mitigating hyper-parameter tuning. However, these cardiac registration methods
are not designed for the MCMR context but are designed to minimize the
brightness inconsistency of estimated motion-warped images and target images
(motion-warping error). Yet in the context of accelerated imaging, the
undersampled input images exhibit artifacts and intensity inconsistencies. The
direct application of these general motion estimation/registration methods to
accelerated imaging data can result in imprecise motion fields and can thus
incur error propagation in MCMR. Qi _et al._ [22] circumvented this problem by
providing reference images in the training loss whilst feeding undersampled
data as network inputs. Concurrently, a registration method designed for the
MCMR context is proposed by Küstner _et al._ [23] in which the registration is
directly estimated from the k-space. All aforementioned methods conduct a
pair-wise motion estimation and they have to be carried out multiple times in
MCMR, in which for every single frame a registration from multiple other
frames is required. To provide a more efficient and time-continuous
registration, groupwise motion estimation has been studied [24, 25]. In group-
wise registration, the spatial-temporal redundancy over multiple frames can be
leveraged to facilitate the registration, especially when through-plane motion
occurs in the context of 2D CMR. Furthermore, the temporal coherence over the
cardiac cycle can be instilled during training by applying a temporal loss
term [24].

After the choice of a proper motion estimation/registration method, there are
multiple MCMR frameworks available to combine the motion estimator and
reconstruction. The seminal work [5] of Batchelor _et al._ pioneered the MCMR
concept in which the motion information is embedded as a general matrix into
the MR forward model. This work formulated the MCMR problem with two
individual stages: motion estimation and reconstruction. The motion estimation
in the first stage and the reconstruction in the second stage are both carried
out separately, while the pre-calculated motion from the first stage is
regarded as a fixed matrix in the second-stage reconstruction [26, 27, 22].
Furthermore, MCMR can also be reformulated as a joint optimization problem in
which an iterative optimization of image reconstruction and motion estimation
are carried out alternatively. A potential synergy can be established: a more
accurate motion estimation can provide a better reconstruction, and based on a
less artifacts-affected image a better motion estimation can be accomplished.
Odille _et al._ proposed a reconstruction method using sensor-based motion
estimation e.g. respiratory belt or ECG signal [28, 29]. The need for external
tracking hardware is relieved by adopting B-spline-based and optical flow-
based motion estimation in this joint optimization context [30, 31]. More
recently, variational methods [32] and dictionary learning [33] are also
employed to solve this joint optimization problem for CMR reconstruction.
However, all these methods demand a relatively long estimation time because of
their iterative optimization nature. Therefore, deep-learning-based methods
were proposed to speed up joint optimization. [25, 34] unrolled MCMR joint
optimization with a group-wise motion estimation network and the mutual
benefit of CMR reconstruction and motion estimation is demonstrated in their
work.

However, the decomposition of the MCMR into two sub-optimization problems
serves as a workaround to solve MCMR has two major drawbacks: First, the
solution space of the full problem is restricted by the solution of the
motion-estimation problem itself whose goal is to minimize the motion-warping
loss between different cardiac frames. This goal is not necessarily aligned
with the final reconstruction objective due to undersampled images’ artifact-
degradation and intensity-inconsistency amongst cardiac frames. Second, extra
efforts have to be built in to cope with motion estimation in the case of
accelerated imaging with undersampled data, e.g. extra pre-processing steps
with intra-bin motion correction [26, 27], loss function tuning [22] or
k-space motion estimation [35]. Although the estimation difficulty of the
motion can be reduced if the alternating joint optimization is used, it
requires multiple iterations of motion estimation and reconstruction to yield
satisfactory reconstruction, prolonging the processing time. On the contrary,
in this work we propose an MCMR framework that optimizes the complete MCMR
framework together without breaking it into two sub-optimization problems.

Moreover, all aforementioned MCMR methods follow the suggestions of [5] which
applied all temporal frames to reconstruct one single frame of the sequence so
that all temporal redundancy can be exploited. We argue in this work that
using a smaller amount of temporal frames to conduct the MCMR can achieve a
better result. This setting reduces the residual motion-warping error from
other temporal frames while still leveraging enough redundant information.

Figure 2: Architecture of the proposed method: Motion-compensated MR
reconstruction (MCMR) framework with a Motion Estimation Block (refer to
III-A) and a complex-valued Motion-Compensated Reconstruction Block (refer to
III-B). The motion estimation learning process is directly driven by the final
reconstruction performance. A pre-processing reconstruction is implemented
(Reconstruction Initialization, refer to II-A) prior to the proposed method to
alleviate the reconstruction difficulty.

In summary, the main contributions of our work are as follows: (a) We propose
a deep learning-based approach, which efficiently solves the motion-
compensated reconstruction and addresses the MCMR problem as a single entity.
Our framework estimates motion from the perspective of CMR reconstruction,
rather than motion estimation alone. We establish an efficient mechanism in
which the motion estimation process is directly driven by the final
reconstruction results (refer to Fig. 1) and without using iterative joint
optimization of motion estimation and reconstruction. (b) We investigate the
optimal number of temporal frames to use during the MCMR. We observe that
using a smaller amount of frames to reconstruct the cardiac frames achieves
better performance than using all frames of a sequence. We find a balance
between the exploitation of sequence redundancy and the suppression of
residual warping error, which can inspire all other MCMR methods. (c) We
demonstrated the reconstruction of images from undersampling rates up to 20x
with the optimization only depending on one final reconstruction loss term.
The canonical motion-warping loss including smoothness terms that serve as an
intermediate loss in MCMR is discarded in this work. Therefore, we avoid the
non-trivial weighting factor tuning. We applied our method on 43 in-house
acquired CMR CINE data and compared it to several canonical and SOTA methods.
Our method outperforms the baselines with superior qualitative and
quantitative results.

##  II Problem Formulation

###  II-A General MR Reconstruction

Let $x^{(t)}\in\mathbb{C}^{N}$ indicate the $t$-th complex-valued temporal
frame of the dynamic CINE sequence $\mathbf{x}=[x^{(1)},\ldots,x^{(T)}]$
stacked as a column vector and $N$ denotes the amount of pixels in the 2D
plane, i.e. $N=N_{X}N_{Y}$ with $X$, $Y$ the height and width of the frame and
$T$ the number of temporal phases. $y^{(t)}\in\mathbb{C}^{SN}$ from
$\mathbf{y}=[y^{(1)},\ldots,y^{(T)}]$ is the corresponding undersampled
k-space data with $S$ being the number of MR receiver coils. Regarding the CMR
reconstruction task of a retrospectively gated CINE, the following inverse
problem has to be solved:

$\displaystyle\min\limits_{x^{(t)}}\left\|\mathbf{A}^{(t)}x^{(t)}-y^{(t)}\right\|^{2}_{2},\
t=1,\dots,T.$ (1)

$\mathbf{A}^{(t)}$ represents the MR forward multi-coil encoding operator with
$\mathbf{A}^{(t)}=\mathbf{D}^{(t)}\mathbf{F}\mathbf{S}$, in which
$\mathbf{S}\in\mathbb{C}^{SN\times N}$ denotes the coil sensitivity maps,
$\mathbf{F}\in\mathbb{C}^{SN\times SN}$ is the forward Fourier encoding
matrix, $\mathbf{D}^{(t)}\in\mathbb{R}^{SN\times SN}$ is the undersampling
mask diagonal matrix. Eq. (1) can be solved by using general conjugate-
gradient SENSE (CG-SENSE) [36] reconstruction which is performed $T$ times to
reconstruct these $T$ cardiac frames. However, this general MR reconstruction
method optimizes every cardiac frame $x^{(t)}$ separately regardless of the
adequate temporal information across the cardiac sequence. Therefore, its
reconstruction performance is limited with respect to the undersampling ratio.
In this work, we use this general CG-SENSE as an initialization step
(Reconstruction Initialization in Fig. 2) for the following MCMR task.

###  II-B Motion-compensated MR reconstruction with a varying number of input
neighboring frames

As mentioned above, leveraging the temporal information in the cardiac
sequence can facilitate the CMR reconstruction. The spatial-temporal redundant
information is bridged by the cardiac motion. Following the work of Batchelor
_et al._ [5], motion is embedded into the MR forward model and information
from other temporal frames can be leveraged as complements:

$\displaystyle\min\limits_{x^{(t)}}\left\|\mathbf{A}^{(K)}\mathbf{U}^{(t\rightarrow
K)}x^{(t)}-\mathbf{y}^{(K)}\right\|^{2}_{2},\ t=1,\dots,T$ (2)

where $K=2k+1$ denotes the neighboring $\pm k$ frames of the frame $t$. The
k-spaces
$\mathbf{y}^{(K)}=[y^{(t-k)},\dots,y^{(t)},\dots,y^{(t+k)}]\in\mathbb{C}^{SNK}$
are used as complementary neighboring data to reconstruct the frame $x^{(t)}$.
We assume periodicity in the cardiac cycle, i.e. the previous frame to
$x^{(0)}$ is regarded as $x^{(T)}$. $\mathbf{U}^{(t\rightarrow
K)}\in\mathbb{R}^{NK\times N}$ denotes the cardiac motion matrix and warps
$x^{(t)}$ to the $K$ cardiac frames. By means of $\mathbf{U}^{(t\rightarrow
K)}$, the redundancy and correlation of the neighboring cardiac frames of
$x^{(t)}$ are instilled for the $t$-th frame reconstruction. It should be
noted that our MCMR framework differs from the original MCMR framework [5]
which applied all temporal frames $K=T$ to conduct the reconstruction, while
in our case we choose $K<T$ as detailed in Section V-A. Analogously to
$\mathbf{A}^{(t)}$,
$\mathbf{A}^{(K)}=\mathbf{D}^{(K)}\mathbf{F}\mathbf{S}\in\mathbb{C}^{SNK\times
NK}$ denotes the CMR forward model for these $K$ frames.

##  III Method

In this work, we propose a deep-learning-based framework to reconstruct the
dynamic CINE images. This framework consists of two parts: a Motion Estimation
Block which tries to estimate cardiac motion and a Motion-Compensated
Reconstruction Block which is purposed for carrying out the motion-compensated
reconstruction, depicted in Fig. 2. In contrast to all the previously proposed
MCMR works, our framework can be trained end-to-end which regards the motion
estimation and reconstruction processes as a single entity instead of
splitting them into two sub-tasks. Furthermore, unrolling the iterative
procedure of motion estimation and reconstruction prolongs the processing time
and renders itself inefficient. In this work we aim to estimate precise motion
directly from the undersampled data by using one-shot prediction with a motion
estimator $\mathcal{G}$ and then solve the inverse problem with an $\ell_{2}$
regularizer using the initial sequence $\mathbf{x}_{u}$ provided by
Reconstruction Initialization block, read as:

$\displaystyle\hat{\mathbf{U}}$
$\displaystyle=\mathcal{G}\left(\mathbf{x}_{u}\right)$ (3a)
$\displaystyle\hat{\mathbf{x}}$
$\displaystyle=\arg\min_{\mathbf{x}}\left\|\mathbf{A}\hat{\mathbf{U}}\mathbf{x}-\mathbf{y}^{(TK)}\right\|_{2}^{2}+\lambda\left\|\mathbf{x}-\mathbf{x}_{u}\right\|_{2}^{2},$
(3b)

where $\lambda$ presents the weighting factor of the applied $\ell_{2}$ term.
$\hat{\mathbf{U}}\in\mathbb{R}^{NTK\times NT}$ denotes the estimated cardiac
motion which is used inside the MCMR, $\hat{\mathbf{x}}$ denotes the final
reconstructed image for all cardiac frames.
$\mathbf{y}^{(TK)}=[y^{(1-k)},\dots,y^{(1+k)},\dots\dots,y^{(T-k)},\dots,y^{(T+k)}]$
extends from $\mathbf{y}^{(K)}$ presenting the adopted complementary
neighboring frames to reconstruct every $x^{(t)}$ of the cardiac sequence
$\mathbf{x}$.

###  III-A Motion Estimation Block

We utilize a learning-based motion estimation network $\mathcal{G}$ with
trainable parameters $\theta$ to predict the non-rigid cardiac motion. The
backbone of GRAFT [24] is applied to model $\mathcal{G}$. GRAFT is a group-
wise motion estimation network that takes the undersampled cardiac sequence
$\mathbf{x}_{u}$ as input and predicts the motion between the frames. Its
inherent Temporal Information Enhancement Block consists of convolutional
layers which take the target frame along with its one previous and subsequent
cardiac frame as input and extract the spatial-temporal information from them.
By means of that, the problem of through-plane motion and occlusion can be
alleviated. Afterward, a Feature Encoder is incorporated which processes the
embedding from Temporal Information Enhancement Block and extracts the
information from the image sequence. Subsequently, a 4D-Correlation layer is
performed to compute the correlation of the 2D spatial plane and a Gated
Recurrent Unit (GRU) is employed to conduct an iterative motion estimation.
Finally, the motion is upsampled $4\times$ to the original image size. This
process is carried out $K$ times and a motion field $\hat{\mathbf{U}}$ mapping
from dimension $NT$ to $NTK$ is produced by GRAFT at this end.

Usually, a warping similarity measurement $\mathcal{L}_{w}$ is utilized to
drive the learning of the motion estimation network:
$\mathcal{L}_{w}(\mathbf{x}^{(K)},\hat{\mathbf{U}}^{(t\rightarrow K)}x^{(t)})$
with $\mathbf{x}^{(K)}=[x^{(t-k)},\dots,x^{(t)},\dots,x^{(t+k)}]$ the target
frames from $\mathbf{x}$ and $\hat{\mathbf{U}}^{(t\rightarrow K)}x^{(t)}$ its
corresponding warped estimation. However, $\mathcal{L}_{w}$ is just an
intermediate motion-warping loss function in the context of MCMR. As mentioned
in Section I, the loss’ effectiveness is undermined by the increase of the
undersampling rate (more aliasing and severe intensity inconsistency) whose
goal diverges from the goal of improving the final reconstruction quality.
Furthermore, the utilization of $\mathcal{L}_{w}$ after the Motion Estimation
Block breaks the original MCMR optimization into two sub-tasks, introducing
the drawbacks as mentioned in Section I. In this work, we do not calculate
$\mathcal{L}_{w}$ at this intermediate position but forward the output motion
$\hat{\mathbf{U}}$ of $\mathcal{G}$ to the subsequent Motion-Compensated
Reconstruction Block. Since no network loss function is applied yet, the
motion prediction $\hat{\mathbf{U}}$ with learnable parameters $\theta$ are
still pending and the complete forward chain of the applied deep learning
model is to be established by the subsequent Motion-Compensated Reconstruction
Block.

###  III-B Motion-Compensated Reconstruction Block

The Motion-Compensated Reconstruction Block is a complex-valued operator which
executes the CINE reconstruction and serves as the forward pass for the
network prediction $\hat{\mathbf{U}}$ to reach the final loss function. This
block solves Eq. (3b) by finding the stationary point utilizing the normal
function:

$\displaystyle\underbrace{\left(\hat{\mathbf{U}}^{H}\mathbf{A}^{H}\mathbf{A}\hat{\mathbf{U}}+\lambda\mathbf{I}\right)}_{\mathcal{V}}\hat{\mathbf{x}}=\underbrace{\left(\hat{\mathbf{U}}^{H}\mathbf{A}^{H}\mathbf{y}^{(TK)}+\lambda\mathbf{x}_{u}\right)}_{\mathbf{b}}.$
(4)

Since the inverse of matrix $\mathcal{V}$ is computationally prohibitive to
calculate, we adopt Conjugate Gradient (CG) [37] to solve this problem in an
iterative manner until the process converges. Therefore, we present the final
output of the Motion-Compensated Reconstruction Block in the basis of a set of
conjugate vectors $\mathcal{P}=\\{\mathbf{p}_{0},...,\mathbf{p}_{I}\\}$:

$\displaystyle\hat{\mathbf{x}}=\sum^{I}_{i=0}\alpha_{i}\mathcal{V}\mathbf{p}_{i}.$
(5)

Here, $(\mathbf{p}_{i})^{H}\mathcal{V}\mathbf{p}_{j}=0$ for all $i\neq j$ and
$\alpha_{i}$ are their corresponding basis coefficients. In the following we
will describe the conjugate gradient algorithm, to comprehend the end-to-end
nature of our proposed approach. Details of the applied CG are shown in Alg.
1. $\mathbf{p}_{i}$ and $\alpha_{i}$ can be obtained during the iteration
update of CG and $I$ is its predefined total iteration number. All
$\alpha_{i}$ and $\mathbf{p}_{i}$ are calculated based on $\mathcal{V}$ while
$\mathcal{V}$ is computed based on the motion matrix $\hat{\mathbf{U}}$. In
contrast to conventional works, it is important to note that
$\hat{\mathbf{U}}$ here is not a static fixed matrix but still a pending
variable from $\mathcal{G}(\mathbf{x}_{u})$. Its trainable parameters $\theta$
still wait for updates through back-propagation on a higher-level loss
function for network training. To express this more clearly, we reformulate
Eq. (5) with Eq. (4) as:

$\displaystyle\hat{\mathbf{x}}=\sum^{I}_{i=0}\alpha_{i}\left(\mathcal{G}^{H}(\mathbf{x}_{u})\mathbf{A}^{H}\mathbf{A}\mathcal{G}(\mathbf{x}_{u})+\lambda\mathbf{I}\right)\mathbf{p}_{i}.$
(6)

Note that $\alpha_{i}$ and $\mathbf{p}_{i}$ also depend on the motion
estimation network $\mathcal{G}(\mathbf{x}_{u})$ (refer to Alg. 1) which in
turn depends on the trainable parameters $\theta$.

Finally, we define our loss function $\mathcal{L}_{r}$ as the mean squared
error between the reconstruction estimation $\hat{\mathbf{x}}$ and the
reference reconstruction target $\mathbf{x}_{\text{ref}}$. Thus, the final
learning-based optimization function can be represented as:

$\displaystyle\mathcal{L}_{r}=\left\|\sum^{I}_{i=0}\alpha_{i}\left(\mathcal{G}^{H}(\mathbf{x}_{u})\mathbf{A}^{H}\mathbf{A}\mathcal{G}(\mathbf{x}_{u})+\lambda\mathbf{I}\right)\mathbf{p}_{i}-\mathbf{x}_{\text{ref}}\right\|_{2}^{2}.$
(7)

Now, the complete deep-learning forward chain is established and $\theta$ can
be updated by gradient back-propagation. An end-to-end MCMR framework is cast
without employing any intermediate motion-warping loss. In this respect, the
motion estimation process is directly guided and driven by feedback from the
final reconstruction performance but not by the motion
estimation/registration. The goal of motion compensation is now aligned with
the final reconstruction goal.

Algorithm 1 Algorithm of Conjugate Gradient

1:$\mathbf{r}_{0}:=\mathbf{b}$, $\mathbf{x}_{0}=\mathbf{0}$ $\triangleright$
$\mathbf{r}$ is the residual

2:$\mathbf{p}_{0}:=\mathbf{r}_{0}$, $i:=0$

3:while $i<I$ do

4:
$\alpha_{i}:=(\mathbf{r}_{i}^{H}\mathbf{r}_{i}/((\mathbf{p}_{i})^{H}\mathcal{V}\mathbf{p}_{i})$

5: $\mathbf{x}_{i+1}:=\mathbf{x}_{i}+\alpha_{i}\mathbf{p}_{i}$

6: $\mathbf{r}_{i+1}:=\mathbf{r}_{i}-\alpha_{i}\mathcal{V}\mathbf{p}_{i}$

7:
$\beta_{i}:=(\mathbf{r}_{i+1})^{H}\mathbf{r}_{i+1}/((\mathbf{r}_{i+1})^{H}\mathbf{r}_{i})$

8: $\mathbf{p}_{i+1}:=\mathbf{r}_{i+1}+\beta_{i}\mathbf{p}_{i}$

9: $i:=i+1$

10:end while

##  IV Experiments

###  IV-A Dataset and Implementation Details

43 subjects (27 patients and 16 healthy volunteers) were scanned with a 2D
cardiac CINE sequence. The data is acquired in-house on a 1.5T MRI scanner
(Magnetom Aera, Siemens Healthineers) with an acquisition sequence of 2D
balanced steady-state free precession (bSSFP) equipped with multi-channel body
and spine coil resulting in 30, 34 and 38 MR receiver coil channels. A
$2\times$ GRAPPA acceleration generated the CINE data with an in-plane
resolution of $1.9\times 1.9\text{mm}^{2}$, a slice thickness of 8mm, echo
time (TE) of 1.06ms, and repetition time (TR) of 2.12ms. Retrospective gating
is used to bin the data into 25 cardiac phases with a temporal resolution of
40ms. Matrix size varies from the smallest size 176 (frequency-encoding)
$\times$ 132 (phase-encoding) to the largest size 192 $\times$ 192\. A stack
of 10 to 15 slices for each subject along the long axis was acquired from base
to apex under multiple breath-holds (2 slices per breath-hold). Slices without
clear cardiac anatomy were discarded, resulting in a total of 366 cardiac
motion-resolved image sequences. Retrospective undersampling is performed by
Cartesian VISTA [38] sampling with varying acceleration factors.

The proposed framework was implemented in PyTorch (v1.9.0) and trained on an
NVIDIA A40 GPU. The AdamW [39] optimizer combined with a one-cycle learning
rate scheduler (max. learning rate 0.0001) was used to optimize Eq. (7). The
network parameters for the Motion Estimation Block follows [24] and the hyper-
parameters $K$, $\lambda$ and $I$ are set to $9$, $0.01$ and $10$ for training
and test. Regarding network training, we adopt either a fixed undersampling
rate for training or a mixed training procedure with $R=8,R=12,R=16$ and
$R=20$ undersampled data with a random selection with the same probability
(dubbed as mixed $R$ training). During inference, we test our approach on an
arbitrary undersampling rate. The undersampled raw k-space data is first
reconstructed by the Reconstruction Initialization block and then fed to the
proposed network.

###  IV-B Ablation study

We investigate the benefits of using Eq. (7) as the loss function in
comparison to the widely used motion-warping loss $\mathcal{L}_{w}$ (refer to
III-A) which breaks the MCMR into two sub-tasks for the MCMR reconstruction.
To this respect, we conduct a set of experiments in which 4 trainings are
carried out: 1. training only uses $\mathcal{L}_{r}$ (Eq. (7)); 2. training
only uses $\mathcal{L}_{w}$; And we use a combined loss function
$\mathcal{L}=\alpha\mathcal{L}_{r}+\beta\mathcal{L}_{w}$ only for this
ablation study. We conduct the 3. training with $\alpha=100,\beta=1$ and 4.
training with $\alpha=10,\beta=1$. We use reference images instead of the
undersampled images in $\mathcal{L}_{w}$ during the network training as
proposed by [22, 34] to mitigate being affected by aliasing artifacts. All of
these 4 experiments are trained and tested on $R=12$.

We further investigate the impact of using different amounts of neighboring
frames $K$ for the dynamic CINE reconstruction during training and test. In
the ideal case, the motion across the whole cardiac cycle can be estimated
precisely, therefore all $N$ temporal frames should be used to exploit the
temporal redundancy. However, non-rigid contraction and expansion of the heart
are challenging to estimate and given the 2D acquisition nature through-plane
motion and occlusion (especially towards basal slices) can occur. Thus, the
residual frame-to-frame warping error cannot be suppressed completely to zero
even with SOTA motion estimators. The more neighboring frames considered, the
larger the accumulated residual motion error can occur. We therefore
investigate the optimal number of neighboring frames to use for the CINE
reconstruction. We run experiments using neighbouring $k=\pm 1$ ($K=3$), $\pm
2$, $\pm 4$, $\pm 6$, $\pm 8$, $\pm 12$ ($K=25$) frames with mixed $R$
training and test on different acceleration rates.

Finally in order to test the method’s generalizability, we implement a 5-fold
cross-validation (35 subjects for training and 8 for testing). We repeat the
experiments five times with the same parameter settings.

###  IV-C Baseline comparisons

We compare our method with six baseline methods. Two SOTA MCMR methods are
considered in which the cardiac motion is estimated explicitly prior to the
reconstruction. One is GRAFT-Recon [24], which applies GRAFT to predict the
cardiac motion by using $\mathcal{L}_{w}$ loss and then conducts the follow-up
reconstruction task separately. The second one is Unrolled-MCMR [34], which
performs an iterative unrolled joint optimization of cardiac motion estimation
and reconstruction but its motion is also calculated from $\mathcal{L}_{w}$.
Moreover, CG-SENSE [36], L+S [6], MoDL [40] and CTF-Net [12] are adopted as
non-MCMR reconstruction methods for comparison. L+S solves the reconstruction
by leveraging the decomposed low-rank and sparse matrix, MoDL uses an unrolled
scheme with a dealiasing network and a data-consistency term, while CTF-Net
tackles the problem by exploiting the $k-t$ domain redundancy using recurrent
networks.

###  IV-D Evaluation

We apply Structural Similarity Index (SSIM) [41], Peak Signal-to-Noise Ratio
(PSNR) and Normalized Mean Squared Error (NMSE) to evaluate the reconstruction
performance quantitatively. Besides these three metrics, we also employ
Learned Perceptual Image Patch Similarity (LPIPS) [42] which has been verified
to be closer to human perception. Furthermore, we use a cardiac segmentation
network [43] to obtain a bounding box around the heart to focus evaluation on
the cardiac anatomy. An offset value of $10$ pixels is set to extend the
bounding box region. All metrics (SSIM, PSNR, NMSE and LPIPS) are evaluated
within this heart region.

##  V Results

###  V-A Ablation Study

We foremost delve into the ablation study to find out the best training
setting for our proposed approach. The final training performance is shown in
Table I. It can be seen that the more $\mathcal{L}_{w}$ participates in the
training, the worse the reconstruction quality becomes. The training fails
(does not converge) if only $\mathcal{L}_{w}$ is used since the undersampled
$R=12$ images differ from the reference images severely and the motion network
cannot find a correlation between them.

TABLE I: The impact of using end-to-end training with loss $\mathcal{L}_{r}$ and the motion-warping loss $\mathcal{L}_{w}$ with training on $R=12$ and test on $R=16$ accelerated data. PSNR is employed here as the evaluation metric. The failed training is marked with ’N.A.’. Metric | only $\mathcal{L}_{r}$ |  $\alpha$=100,$\beta$=1 |  $\alpha$=10,$\beta$=1 | only $\mathcal{L}_{w}$
---|---|---|---|---
PSNR | 30.00 | 29.69 | 29.39 | N.A.
TABLE II: The reconstruction performance of using different neighboring frames amount $K$. PSNR is employed here as the evaluation metric and calculated on the whole range of images. The top two results are marked in bold. K | $R=8$ | $R=12$ | $R=16$ | $R=20$
---|---|---|---|---
$3$ | 40.79 $\pm$ 2.25 | 37.31 $\pm$ 2.09 | 33.14 $\pm$ 2.44 | 29.52 $\pm$ 2.69
$5$ | 42.39 $\pm$ 2.82 | 39.73 $\pm$ 2.32 | 36.73 $\pm$ 2.03 | 33.64 $\pm$ 2.97
$9$ | 41.27 $\pm$ 3.74 | 39.85 $\pm$ 3.03 | 38.26 $\pm$ 2.57 | 36.58 $\pm$ 2.76
$13$ | 40.20 $\pm$ 3.72 | 39.16 $\pm$ 3.38 | 37.92 $\pm$ 3.02 | 36.95 $\pm$ 2.83
$17$ | 39.31 $\pm$ 3.31 | 38.46 $\pm$ 3.21 | 37.33 $\pm$ 3.01 | 36.61 $\pm$ 2.99
$25$ | 37.99 $\pm$ 2.85 | 37.26 $\pm$ 2.77 | 36.21 $\pm$ 2.69 | 35.77 $\pm$ 2.90

Figure 3: Reconstruction error maps between reconstructed and reference image for using different neighboring frames amount $K$ on a test sample with acceleration rate $R=20$. TABLE III: 5 fold cross-validation of the proposed network with mixed $R$ training and test on $R=8$ and $R=16$ cases. PSNR and SSIM values are shown. Fold | $R=8$ | $R=16$
---|---|---
| PSNR | SSIM | PSNR | SSIM
Fold 1 | 31.57 $\pm$ 5.38 | 0.92 $\pm$ 0.04 | 28.88 $\pm$ 3.98 | 0.87 $\pm$ 0.06
Fold 2 | 31.45 $\pm$ 4.04 | 0.92 $\pm$ 0.04 | 29.22 $\pm$ 3.86 | 0.88 $\pm$ 0.06
Fold 3 | 32.81 $\pm$ 5.00 | 0.93 $\pm$ 0.04 | 30.42 $\pm$ 4.09 | 0.89 $\pm$ 0.06
Fold 4 | 32.42 $\pm$ 3.22 | 0.93 $\pm$ 0.03 | 30.16 $\pm$ 3.12 | 0.89 $\pm$ 0.05
Fold 5 | 31.59 $\pm$ 3.46 | 0.93 $\pm$ 0.04 | 29.40 $\pm$ 3.24 | 0.89 $\pm$ 0.05
Avg. | 31.96 $\pm$ 4.28 | 0.93 $\pm$ 0.04 | 29.66 $\pm$ 3.69 | 0.88 $\pm$ 0.06

The quantitative evaluation for the optimal amount of neighboring frames is
shown in Table II. A qualitative analysis of the reconstruction error for the
$R=20$ case is illustrated in Fig. 3. It can be seen that adopting fewer
neighboring frames in low acceleration rates is preferable. Yet with the
increase of acceleration rate for a small number of neighboring frames, no
sufficient temporal redundancy is captured, resulting in increased
reconstruction errors in the image background as shown in the $K=3(\pm 1)$ and
$5(\pm 2)$ cases. However, if a high number of neighboring frames is chosen, a
performance drop is observed because the advantage of using more redundant
information from other frames is overcome by the suboptimal large and through-
plane motion errors. In this case, motion-warping errors accumulate around the
heart region while the background error is suppressed (refer to the $K=25$
case). We found that choosing $K=9$ is an optimal trade-off across all
acceleration rates as indicated in Table II. Therefore, we applied this value
for the reconstruction in all experiments.

For the evaluation of the generalization, the averaged PSNR and SSIM across
the 5 folds are displayed in Table III. Regarding both PSNR and SSIM metrics,
the performance difference between these 5 folds is smaller than the standard
deviation within one specific fold. Therefore, we can conclude that the
proposed method demonstrates robust behavior toward subject domain shifts.

TABLE IV: Quantitative comparison of the proposed framework, CG-SENSE [36], L+S [6], MoDL [40], CTF-Net [12], GRAFT-Recon [24] and Unrolled-MCMR [34] during inference for $R=8,12,16$ and $R=20$. Peak Signal to Noise Ratio (PSNR), Structure Similarity (SSIM), Learned Perceptual Image Patch Similarity (LPIPS) and Normalized Mean Squared Error (NMSE) are adopted as evaluation metrics and their averaged value with corresponding standard deviation across all test samples are shown. Methods’ average execution time of reconstructing the sample with spatial resolution $192\times 156$ is logged. The best results are marked in bold. | Metrics | CG-SENSE | L+S | MoDL | CTF-Net | GRAFT-Recon | Unrolled-MCMR | Proposed
---|---|---|---|---|---|---|---|---
$R=8$ | PSNR | 23.88 $\pm$ 2.01 | 26.40 $\pm$ 2.11 | 28.80 $\pm$ 2.04 | 27.41 $\pm$ 2.80 | 29.02 $\pm$ 3.08 | 30.46 $\pm$ 6.23 | 31.57 $\pm$ 5.38
SSIM | 0.76 $\pm$ 0.05 | 0.84 $\pm$ 0.04 | 0.88 $\pm$ 0.03 | 0.88 $\pm$ 0.05 | 0.89 $\pm$ 0.05 | 0.90 $\pm$ 0.05 | 0.92 $\pm$ 0.04
LPIPS | 0.08 $\pm$ 0.03 | 0.06 $\pm$ 0.02 | 0.02 $\pm$ 0.01 | 0.04 $\pm$ 0.02 | 0.03 $\pm$ 0.02 | 0.03 $\pm$ 0.02 | 0.02 $\pm$ 0.01
NMSE | 0.25 $\pm$ 0.06 | 0.18 $\pm$ 0.04 | 0.14 $\pm$ 0.03 | 0.16 $\pm$ 0.05 | 0.14 $\pm$ 0.05 | 0.13 $\pm$ 0.05 | 0.11 $\pm$ 0.05
$R=12$ | PSNR | 17.54 $\pm$ 2.47 | 20.87 $\pm$ 2.65 | 22.05 $\pm$ 2.78 | 25.27 $\pm$ 2.64 | 25.19 $\pm$ 2.53 | 29.41 $\pm$ 4.76 | 30.10 $\pm$ 4.38
SSIM | 0.51 $\pm$ 0.09 | 0.70 $\pm$ 0.07 | 0.70 $\pm$ 0.08 | 0.84 $\pm$ 0.05 | 0.80 $\pm$ 0.07 | 0.88 $\pm$ 0.06 | 0.89 $\pm$ 0.05
LPIPS | 0.18 $\pm$ 0.06 | 0.09 $\pm$ 0.04 | 0.08 $\pm$ 0.04 | 0.05 $\pm$ 0.03 | 0.07 $\pm$ 0.05 | 0.03 $\pm$ 0.02 | 0.02 $\pm$ 0.01
NMSE | 0.52 $\pm$ 0.13 | 0.34 $\pm$ 0.08 | 0.31 $\pm$ 0.09 | 0.20 $\pm$ 0.05 | 0.21 $\pm$ 0.06 | 0.14 $\pm$ 0.06 | 0.13 $\pm$ 0.05
$R=16$ | PSNR | 15.09 $\pm$ 2.53 | 16.81 $\pm$ 2.96 | 17.22 $\pm$ 3.40 | 23.67 $\pm$ 2.62 | 23.55 $\pm$ 2.44 | 28.44 $\pm$ 4.20 | 28.88 $\pm$ 3.98
SSIM | 0.35 $\pm$ 0.12 | 0.50 $\pm$ 0.13 | 0.47 $\pm$ 0.14 | 0.80 $\pm$ 0.06 | 0.73 $\pm$ 0.08 | 0.86 $\pm$ 0.07 | 0.87 $\pm$ 0.06
LPIPS | 0.24 $\pm$ 0.07 | 0.16 $\pm$ 0.06 | 0.16 $\pm$ 0.07 | 0.07 $\pm$ 0.03 | 0.11 $\pm$ 0.06 | 0.03 $\pm$ 0.02 | 0.03 $\pm$ 0.02
NMSE | 0.69 $\pm$ 0.17 | 0.54 $\pm$ 0.15 | 0.54 $\pm$ 0.19 | 0.23 $\pm$ 0.06 | 0.26 $\pm$ 0.07 | 0.15 $\pm$ 0.06 | 0.15 $\pm$ 0.06
$R=20$ | PSNR | 14.24 $\pm$ 2.44 | 15.27 $\pm$ 2.77 | 15.43 $\pm$ 3.23 | 23.37 $\pm$ 2.68 | 22.66 $\pm$ 2.52 | 27.71 $\pm$ 3.95 | 27.90 $\pm$ 3.81
SSIM | 0.29 $\pm$ 0.11 | 0.39 $\pm$ 0.13 | 0.36 $\pm$ 0.14 | 0.78 $\pm$ 0.07 | 0.68 $\pm$ 0.09 | 0.84 $\pm$ 0.08 | 0.85 $\pm$ 0.07
LPIPS | 0.26 $\pm$ 0.07 | 0.20 $\pm$ 0.06 | 0.21 $\pm$ 0.08 | 0.08 $\pm$ 0.04 | 0.13 $\pm$ 0.08 | 0.05 $\pm$ 0.03 | 0.04 $\pm$ 0.03
NMSE | 0.75 $\pm$ 0.17 | 0.64 $\pm$ 0.16 | 0.65 $\pm$ 0.19 | 0.25 $\pm$ 0.06 | 0.29 $\pm$ 0.08 | 0.17 $\pm$ 0.07 | 0.17 $\pm$ 0.07
Avg. Time (s) |  | 0.29 $\pm$ 0.01 | 1.73 $\pm$ 0.01 | 2.48 $\pm$ 0.01 | 3.00 $\pm$ 0.01 | 5.37 $\pm$ 0.01 | 14.89 $\pm$ 0.03 | 5.49 $\pm$ 0.01

###  V-B Baseline comparisons

We compare our method to six baseline methods (see IV-C). The deep learning
approaches included the proposed network, MoDL [40] and CTF-Net [12] are
trained with mixed $R$ training procedure (see IV-A). These methods can
achieve their best performance by using this training procedure. The mixed $R$
training procedure is not applied to GRAFT-Recon [24] and Unrolled-MCMR [34]
since including highly undersampled data gave rise to unstable training and
poor reconstruction performance. In practice we found that GRAFT-Recon can
achieve the best inference results when using the fixed $R=8$ training
compared to all other fixed $R$ trainings. For Unrolled-MCMR, the fixed $R=12$
training is the best training strategy. Thus, for GRAFT-Recon a fixed $R=8$
training is conducted, while for Unrolled-MCMR training we only use $R=12$
data. Besides, we also set their temporal neighborhood to $K=9$ with $\pm 4$
neighboring frames, whilst in their original work they employed all temporal
frames which can cause higher warping errors. After training, all six methods
are tested on an arbitrary undersampling rate. The quantitative performance
evaluated by metrics PSNR, NMSE, SSIM, and LPIPS is shown in Table IV.

Figure 4: Qualitative comparison of the proposed method to non-MCMR methods
including CG-SENSE [36], L+S [6], MoDL [40] and CTF-Net [12] in the $R=8$
(patient with myocarditis) and $R=16$ (healthy subject) accelerated
acquisition. The respective PSNR values of the heart region are depicted in
the image. Reference images, reconstructed images and their corresponding
error maps are demonstrated. The spatial ($x-y$) images are depicted next to
the temporal traces ($y-t$) through the middle of the left ventricle. The
selected y-axis is marked with a blue line in the reference image.

The superior and consistent performance of the proposed method is shown across
every single undersampling rate compared to all other baseline methods and
regardless of the evaluation metric. It can be noted that learning-based
methods e.g. GRAFT-Recon, Unrolled-MCMR, CTF-Net and MoDL outperform
conventional methods like CG-SENSE and L+S. Moreover, Unrolled-MCMR
consistently demonstrates the second-best performance because of its usage of
reference images in the motion-warping loss function and its unrolled
iterative optimization mechanism. The advantage of iterative optimization
becomes more prominent for higher acceleration rates.

The qualitative comparison of two test subjects (healthy subject and patient)
between the proposed network and the non-MCMR methods is illustrated in Fig. 4
for undersampling rates of $R=8$ and $R=16$. The corresponding error maps are
displayed as well. The proposed network presents a consistent performance in
both subjects with the highest PSNR score and lowest residual error. Temporal
traces are in good agreement with the fully-sampled reference and cardiac
dynamics were recovered by the proposed network. Diagnostic features like
papillary muscles are restored clearly without blurring in both cases.
Moreover, it is apparent that MoDL does not leverage any temporal information
to reconstruct the cardiac frames. While CTF-Net and MoDL still show
comparable results for $R=8$, a severe performance drop for MoDL is perceived
in $R=16$. The same conclusion can also be drawn from Table IV. This indicates
the importance of leveraging temporal redundancy during the reconstruction,
especially in highly accelerated cases.

Further qualitative comparison of two test subjects (healthy subject and
patient) between the proposed framework and other MCMR methods (GRAFT-Recon
and Unrolled-MCMR) are demonstrated in Fig. 5. The proposed network
outperforms the two compared MCMR methods in both $R=12$ and $R=20$. While the
proposed framework is only trained with one loss term (without any smoothness
terms), it predicts a more meaningful and dense motion field even for $R=20$.
The motion estimation from Unrolled-MCMR is sparse and non-smooth, in spite of
the usage of smoothness terms during training. The GRAFT-Recon reveals
inferior reconstruction due to the motion estimation being artifact-affected
resulting in error propagation amongst frames, while the proposed method
yields a reconstruction image without any aliasing in both cardiac region and
background.

Figure 5: Qualitative comparison of the proposed network to CG-SENSE [36],
GRAFT-Recon [24] and Unrolled-MCMR [34] in the $R=12$ (left side, healthy
subject) and $R=20$ (right side, patient with myocarditis) accelerated
acquisition. The respective PSNR values of the heart region are shown in the
image. Reference images, reconstructed images, corresponding error maps and
color-wheel-encoded [44] motion field visualization are shown. The spatial
($x-y$) images are depicted next to the temporal traces ($y-t$) through the
left ventricle. The selected y-axis is marked with a blue line in the
reference image.

##  VI Discussion

MCMR is a powerful and straightforward concept that has been demonstrated for
the reconstruction of cardiac CINE [45, 33, 25, 34]. However, a wide range of
MCMR implementations for CINE is precluded by two major unsolved challenges:
high-speed MCMR processing and precise artifact-suppressed cardiac motion
estimation. In this work, we proposed a learning-based MCMR framework for CINE
imaging that copes with these two problems at once. The fast MCMR is achieved
by leveraging the trained network to accelerate the estimation process in
inference time, whilst the artifact-suppressed motion estimation is achieved
using reconstruction-driven motion estimation. We treat the two sub-tasks as a
single entity, in which the training loss is back-propagated end-to-end from
the final reconstructed images to the motion estimation input.

We performed an ablation study in which the use of an intermediate warping
similarity loss is compared to a final reconstruction loss. Results indicate
that if the optimization is driven by the final reconstruction loss, not only
the reconstruction performance is enhanced, but also the motion prediction is
ameliorated. Furthermore, we investigated the aspect of the optimal number of
neighboring frames in the cardiac cycle to be used for MCMR. The conducted
experiments demonstrate that the higher the acceleration rate, the more
neighboring frames are preferable. There is a trade-off between the static
reconstruction error which is incurred by the lack of redundant information,
and the dynamic reconstruction error around the heart which is caused by
residual warping errors from neighboring frames. It is important to note that
this phenomenon is not only occurring in our proposed approach but is generic
for any MCMR method. Based on these results, we set a fixed number of
neighboring frames in this work. In the future, we plan to make this important
hyper-parameter learnable so it can self-adapt to the optimal value for
different application scenarios.

We further conduct a systematic analysis of the proposed approach compared to
baseline methods. The consistent and superior performance of our proposed
method is found throughout this work. We conclude from the experiments that
learning-based methods usually outperform non-learning-based methods.
Furthermore, methods that leverage the whole cardiac cycle e.g. CTF-Net
usually outperform methods that leverage less temporal redundancy e.g. MoDL,
especially in the high acceleration case. Furthermore, Unrolled-MCMR showed
similar but inferior performance results to the proposed method. This is
attributed to two major reasons. First, we carry out just a single but more
effective optimization instead of applying alternating updates of motion
fields and image reconstructions. It should be noted that our proposed method
can also be extended as an iterative unrolled optimization but at the cost of
prolonged training and test time. Second, we only have one simple
reconstruction loss term $\mathcal{L}_{r}$ in the final stage, while Unrolled-
MCMR employs a conventional motion-warping loss $\mathcal{L}_{w}$ together
with two regularization terms, requiring non-trivial weighting factor tuning.

The superior results of the proposed method can be attributed to the artifact-
suppressed motion estimation (refer to the motion fields in Fig. 5). The
proposed Motion-Compensated Reconstruction Block can be regarded as a
transformation operator which extends the motion estimation procedure from
image space to k-space. Although Eq. (7) presents a loss function that forces
the framework to generate a reference resembled reconstruction, it can also be
interpreted as a warping loss function which warps a set of undersampled
images by the estimated motion to the target images while ensuring consistency
to acquired k-space samples.

Furthermore, our proposed approach provides another perspective on solving the
cardiac motion estimation/registration problem. Cardiac motion
estimation/registration can not only be used inside the MCMR framework for
reconstruction but can also be applied for cardiac feature tracking to
evaluate myocardial strain and functional analysis [46, 47] or to facilitate
cardiac segmentation tasks [48]. Our proposed method can be recast to a motion
estimation/registration method with two major benefits compared to the
conventional motion estimation/registration methods. First, we only need a
single loss term (Eq. (7)) to generate smooth and realistic motion fields
avoiding further hyper-parameter tuning. Second, we can predict high-quality
cardiac motion directly from highly-undersampled MR data. It is also
conceivable that we do not need visually appealing MR images for the
extraction and quantifícation of clinical parameters (e.g. left ventricular
function). A potential synergistic approach for jointly reconstructing,
analyzing (e.g. segmentation or motion tracking) and interpreting the cardiac
CINE imaging will be developed further based on this study.

In the end, we also acknowledge some limitations of our work. First, the
motion estimation is based on the backbone of GRAFT. It conducts $N\times K$
computations to reconstruct the cardiac cycle with $N$ frames which are
suboptimal regarding estimation speed and memory usage. In future work, we
will attempt to build a more efficient and lightweight group-wise motion
estimator to accelerate the reconstruction process further. Moreover,
currently our MCMR framework only learns motion prediction and uses only one
non-learnable regularizer. This has been proven in this work as a simple yet
effective approach, but we have not yet integrated our motion estimates in
learnable denoising regularizers [49, 40], which will be subject to future
work. Finally, the proposed work is solely evaluated for retrospective
undersampling. The test performance on a larger cohort including prospectively
collected data will also be evaluated in future work.

##  VII Conclusion

In this work, we proposed a learning-based MCMR framework for CINE imaging. We
introduce a mechanism that solves the MCMR problem as a single entity and
drives the motion estimation directly from the final reconstruction
perspective. The training loss is back-propagated through the whole pipeline
and the framework is optimized end-to-end without breaking into two sub-tasks.
We find out that using a smaller neighboring frames number to conduct MCMR can
achieve better results than using all sequence frames. Our method shows
consistent performance throughout all conducted experiments and outperforms
all baseline methods. We have confidence that the developed method for cardiac
CINE imaging can also be generalized and applied to other reconstruction
applications.

## References

  * [1] P. Kellman, F. H. Epstein, and E. R. McVeigh, “Adaptive sensitivity encoding incorporating temporal filtering (tsense)†,” _Magnetic Resonance in Medicine_ , vol. 45, no. 5, pp. 846–852, 2001. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/mrm.1113
  * [2] F. A. Breuer, P. Kellman, M. A. Griswold, and P. M. Jakob, “Dynamic autocalibrated parallel imaging using temporal grappa (tgrappa),” _Magnetic Resonance in Medicine_ , vol. 53, no. 4, pp. 981–985, 2005. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/mrm.20430
  * [3] H. Jung, K. Sung, K. S. Nayak, E. Y. Kim, and Y. J. C., “k-t focuss: A general compressed sensing framework for high resolution dynamic mri,” _Magnetic Resonance in Medicine_ , vol. 61, no. 1, pp. 103–116, 2009.
  * [4] R. Otazo, D. Kim, L. Axel, and D. K. Sodickson, “Combination of compressed sensing and parallel imaging for highly accelerated first-pass cardiac perfusion mri,” _Magnetic Resonance in Medicine_ , vol. 64, no. 3, pp. 767–776, 2010. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/mrm.22463
  * [5] P. G. Batchelor, D. Atkinson, P. Irarrazaval, D. L. G. Hill, H. J., and L. D., “Matrix description of general motion correction applied to multishot images,” _Magnetic Resonance in Medicine_ , vol. 54, p. 1273–1280, 2005\.
  * [6] R. Otazo, E. Candès, and D. K. Sodickson, “Low-rank plus sparse matrix decomposition for accelerated dynamic mri with separation of background and dynamic components,” _Magnetic Resonance in Medicine_ , vol. 73, no. 3, pp. 1125–1136, 2015. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/mrm.25240
  * [7] W. Huang and et al., “Deep low-rank plus sparse network for dynamic mr imaging,” _Medical Image Analysis_ , vol. 73, p. 102190, 2021.
  * [8] T. Küstner, N. Fuin, K. Hammernik, and et al., “CINENet: deep learning-based 3D cardiac CINE MRI reconstruction with multi-coil complex-valued 4D spatio-temporal convolutions,” _Scientific Reports_ , vol. 10, no. 1, pp. 1–13, 2020.
  * [9] C. M. Sandino, P. Lai, S. S. Vasanawala, and J. Y. Cheng, “Accelerating cardiac cine mri using a deep learning-based espirit reconstruction,” _Magnetic Resonance in Medicine_ , vol. 85, no. 1, 2021.
  * [10] S. G. Lingala, Y. Hu, E. DiBella, and M. Jacob, “Accelerated dynamic mri exploiting sparsity and low-rank structure: k-t slr,” _IEEE Transactions on Medical Imaging_ , vol. 30, no. 5, pp. 1042–1054, 2011.
  * [11] C. Qin and et al., “k-t next: Dynamic mr image reconstruction exploiting spatio-temporal correlations,” in _Medical Image Computing and Computer Assisted Intervention_ , 2019, pp. 505–513.
  * [12] ——, “Complementary time-frequency domain networks for dynamic parallel mr image reconstruction,” _Magnetic Resonance in Medicine_ , vol. 86, no. 6, pp. 3274–3291, 2021.
  * [13] S. Klein, M. Staring, K. Murphy, and J. Pluim, “Elastix: a toolbox for intensity-based medical image registration,” _IEEE Transactions on Medical Imaging_ , vol. 29, no. 1, pp. 196–205, 2009.
  * [14] M. Modat and et al., “Fast free-form deformation using graphics processing units,” _Computer Methods and Programs in Biomedicine_ , vol. 98, no. 3, pp. 278–284, 2010.
  * [15] T. Vercauteren, X. Pennec, A. Perchant, and N. Ayache, “Diffeomorphic demons: Efficient non-parametric image registration,” _Neuro Image_ , vol. 45, no. 1, pp. S61–S72, 2009.
  * [16] X. Chen, A. Diaz-Pinto, N. Ravikumar, and A. F. Frangi, “Deep learning in medical image registration,” _Progress in Biomedical Engineering_ , vol. 3, no. 1, p. 012003, feb 2021. [Online]. Available: https://dx.doi.org/10.1088/2516-1091/abd37c
  * [17] G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, and A. Dalca, “An unsupervised learning model for deformable medical image registration,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9252–9260.
  * [18] ——, “Voxelmorph: A learning framework for deformable medical image registration,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 8, pp. 1788–1800, 2019.
  * [19] M. A. Morales, D. Izquierdo-Garcia, I. Aganj, J. Kalpathy-Cramer, B. R. Rosen, and C. Catana, “Implementation and validation of a three-dimensional cardiac motion estimation network.” in _Radiol Artificial Intelligence_ , vol. 4, no. 1, 2019.
  * [20] H. Yu, X. Chen, H. Shi, T. Chen, T. S. Huang, and S. Sun, “Motion pyramid networks for accurate and efficient cardiac motion estimation,” _CoRR_ , vol. abs/2006.15710, 2020.
  * [21] J. Pan, D. Rueckert, T. Küstner, and K. Hammernik, “Efficient image registration network for non-rigid cardiac motion estimation,” in _Machine Learning for Medical Image Reconstruction_ , 2021, pp. 14–24.
  * [22] H. Qi and et al., “End-to-end deep learning nonrigid motion-corrected reconstruction for highly accelerated free-breathing coronary mra,” _Magnetic Resonance in Medicine_ , vol. 86, no. 1, pp. 1983–1996, 2021.
  * [23] T. Küstner and et al., “Lapnet: Non-rigid registration derived in k-space for magnetic resonance imaging,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 12, pp. 3686–3697, 2021.
  * [24] K. Hammernik, J. Pan, D. Rueckert, and T. Küstner, “Motion-guided physics-based learning for cardiac mri reconstruction,” _Asilomar Conference on Signals, Systems, and Computers_ , 2021.
  * [25] J. Yang, T. Küstner, P. Hu, P. Liò, and H. Qi, “End-to-end deep learning of non-rigid groupwise registration and reconstruction of dynamic mri,” _Frontiers in Cardiovascular Medicine_ , vol. 9, 2022. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fcvm.2022.880186
  * [26] G. Cruz, D. Atkinson, M. Henningsson, R. Botnar, and C. Prieto, “Highly efficient nonrigid motion-corrected 3d whole-heart coronary vessel wall imaging,” _Magnetic Resonance in Medicine_ , vol. 77, no. 5, pp. 1894–1908, 2017.
  * [27] A. Bustin and et al., “3d whole-heart isotropic sub-millimeter resolution coronary magnetic resonance angiography with non-rigid motion-compensated prost,” _Journal of Cardiovascular Magnetic Resonance_ , vol. 22, no. 1, 2020\.
  * [28] F. Odille, P. A. Vuissoz, P. Y. Marie, and J. Felblinger, “Generalized reconstruction by inversion of coupled systems (grics) applied to free-breathing mri,” _Magnetic Resonance in Medicine_ , vol. 60, pp. 146–157, 2008.
  * [29] F. Odille and et al., “Joint reconstruction of multiple images and motion in mri: Application to free-breathing myocardial ${\rm t}_{2}$ quantification,” _IEEE Transactions on Medical Imaging_ , vol. 35, no. 1, pp. 197–207, 2016\.
  * [30] J. Royuela-del Val, L. Cordero-Grande, F. Simmross-Wattenberg, M. Martín-Fernández, and C. Alberola-López, “Nonrigid groupwise registration for motion estimation and compensation in compressed sensing reconstruction of breath-hold cardiac cine mri,” _Magnetic Resonance in Medicine_ , vol. 75, no. 4, pp. 1525–1536, 2016. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/mrm.25733
  * [31] N. Zhao, D. O’Connor, A. Basarab, D. Ruan, and K. Sheng, “Motion compensated dynamic mri reconstruction with local affine optical flow estimation,” _IEEE Transactions on Biomedical Engineering_ , vol. 66, no. 11, pp. 3050–3059, 2019.
  * [32] A. I. Aviles-Rivero, N. Debroux, G. Williams, M. J. Graves, and C. Schönlieb, “Compressed sensing plus motion (cs + m): A new perspective for improving undersampled mr image reconstruction,” _Medical Image Analysis_ , vol. 68, p. 101933, 2021.
  * [33] T. Schmoderer, A. I. Aviles-Rivero, V. Corona, N. Debroux, and C. Schönlieb, “Learning optical flow for fast MRI reconstruction,” _Inverse Problems_ , vol. 37, no. 9, p. 095007, aug 2021.
  * [34] J. Pan, D. Rueckert, T. Küstner, and K. Hammernik, “Learning-based and unrolled motion-compensated reconstruction for cardiac mr cine imaging,” in _Medical Image Computing and Computer Assisted Intervention_ , 2022, pp. 686–696.
  * [35] T. Küstner and et al., “Deep-learning based motion-corrected image reconstruction in 4d magnetic resonance imaging of the body trunk,” in _2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_ , 2020, pp. 976–985.
  * [36] K. P. Pruessmann, M. Weiger, P. Börnert, and P. Boesiger, “Advances in sensitivity encoding with arbitrary k-space trajectories,” _Magnetic Resonance in Medicine_ , no. 46, pp. 638–651, 2001.
  * [37] W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, _Numerical Recipes in C_ , 2nd ed. Cambridge, USA: Cambridge University Press, 1992.
  * [38] R. Ahmad, H. Xue, S. Giri, Y. Ding, J. Craft, and O. P. Simonetti, “Variable density incoherent spatiotemporal acquisition (vista) for highly accelerated cardiac mri,” _Magnetic Resonance in Medicine_ , vol. 74, no. 5, pp. 1266–1278, 2015.
  * [39] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , no. 19, 2017.
  * [40] H. K. Aggarwal, M. P. Mani, and M. Jacob, “Model based image reconstruction using deep learned priors (modl),” in _IEEE International Symposium on Biomedical Imaging (ISBI)_ , 2018, pp. 671–674.
  * [41] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transaction of Image Process_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [42] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [43] C. Chen and et al., “Deep learning for cardiac image segmentation: A review,” _Frontiers in Cardiovascular Medicine_ , vol. 7, 2020. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fcvm.2020.00025
  * [44] S. Baker and et al., “A database and evaluation methodology for optical flow,” in _2007 IEEE 11th International Conference on Computer Vision_ , 2007, pp. 1–8.
  * [45] G. Cruz, K. Hammernik, T. Kuestner, D. Rueckert, R. Botnar, and C. Prieto, “One-heartbeat cardiac cine imaging via jointly regularized non-rigid motion corrected reconstruction,” in _Proc. International Society for Magnetic Resonance in Medicine (ISMRM)_ , 2021, p. 0070.
  * [46] R. J. Taylor and et al., “Myocardial strain measurement with feature-tracking cardiovascular magnetic resonance: normal values,” _European Heart Journal - Cardiovascular Imaging_ , vol. 16, no. 8, pp. 871–881, 02 2015. [Online]. Available: https://doi.org/10.1093/ehjci/jev006
  * [47] G. Pedrizzetti, P. Claus, P. J. Kilner, and E. Nagel, “Principles of cardiovascular magnetic resonance feature tracking and echocardiographic speckle tracking for informed clinical use,” _Journal of Cardiovascular Magnetic Resonance_ , vol. 18, no. 1, p. 51, Aug 2016. [Online]. Available: https://doi.org/10.1186/s12968-016-0269-7
  * [48] C. Qin and et al., “Joint learning of motion estimation and segmentation for cardiac mr image sequences,” in _Medical Image Computing and Computer Assisted Intervention_ , 2018.
  * [49] K. Hammernik and et al., “Learning a variational network for reconstruction of accelerated mri data,” _Magnetic Resonance in Medicine_ , vol. 79, no. 6, pp. 3055–3071, 2018.