# Deep Learning for Accelerated and Robust MRI Reconstruction: a Review

Reinhard Heckel  Technical University of Munich, Department of Computer
Engineering, Munich, Germany  Mathews Jacob  University of Iowa, Department of
Electrical and Computer Engineering, Iowa City, IA, USA  Akshay Chaudhari
Stanford University, Department of Radiology and Department of Biomedical Data
Science, Stanford, CA, USA  Or Perlman  Tel Aviv University, Department of
Biomedical Engineering, Tel Aviv, Israel  Tel Aviv University, Sagol School of
Neuroscience, Tel Aviv, Israel  Efrat Shimron

###### Abstract

Deep learning (DL) has recently emerged as a pivotal technology for enhancing
magnetic resonance imaging (MRI), a critical tool in diagnostic radiology.
This review paper provides a comprehensive overview of recent advances in DL
for MRI reconstruction. It focuses on DL approaches and architectures designed
to improve image quality, accelerate scans, and address data-related
challenges. These include end-to-end neural networks, pre-trained networks,
generative models, and self-supervised methods. The paper also discusses the
role of DL in optimizing acquisition protocols, enhancing robustness against
distribution shifts, and tackling subtle bias. Drawing on the extensive
literature and practical insights, it outlines current successes, limitations,
and future directions for leveraging DL in MRI reconstruction, while
emphasizing the potential of DL to significantly impact clinical imaging
practices.

_Keywords_ MRI, deep learning, machine learning, image reconstruction

##  1 Introduction

Magnetic Resonance Imaging (MRI) has long been at the forefront of medical
imaging, in that it offers unparalleled ability to visualize the human body’s
internal structures and functions. Because it is noninvasive and has
exceptional soft tissue contrast, it has become an indispensable tool in
modern diagnostic medicine. From detecting subtle pathological changes to
guiding therapeutic interventions, MRI’s versatility is unmatched. However,
the full potential of MRI is often constrained by inherent limitations in
imaging speed and resolution, which are crucial for accurate diagnosis and
patient comfort [1].

One of the fundamental challenges facing MRI is the inherent trade-off between
image quality and acquisition time. High-resolution images, which are
essential for precise diagnosis, traditionally require longer scan times,
which can be burdensome for patients and increase susceptibility to motion
artifacts. This limitation is particularly pronounced in dynamic imaging
scenarios such as cardiac and abdominal imaging, where rapid physiological
movements can cause artifacts such as blurring [2].

Significant research efforts have been aimed at accelerating MRI. Central to
these endeavors is the development of methods for image reconstruction from
under-sampled data. Techniques based on parallel imaging (PI) introduced in
the 1990s were an important watershed [3, 4, 5, 6, 7, 8]. These techniques
leverage the spatial diversity of multiple coil arrays to reconstruct images,
thus allowing for reduced scan times by acquiring less data. The early 2000s
saw the emergence of compressed sensing (CS) methods, which constituted a
novel approach to MRI reconstruction [9, 10, 11, 12, 13, 14, 15, 16]. CS
overcomes the sparsity of MRI images by enabling the reconstruction of high-
quality images from a much smaller set of measurements than traditionally
required. The development of these techniques represented major milestones in
MRI, and contributed to substantial reductions in scan duration and
improvements in image quality. However, both parallel imaging and compressed
sensing methods have their limitations, such as the need for specific
acquisition schemes or the long computation times required for solving
optimization problems, especially for non-Cartesian imaging trajectories.

In recent years, the advent of deep learning (DL) has heralded a new era in
MRI by offering promising solutions to these longstanding challenges. DL, a
subset of machine learning characterized by algorithms based on computational
neural networks, has had remarkable success in extracting complex patterns
from large datasets [17, 18, 19, 20].In the realm of MR image reconstruction,
DL methods focus on learning from vast amounts of data to transform under-
sampled or noisy data into high-fidelity images.These methods have
demonstrated their ability to mitigate artifacts, enhance resolution, and
accelerate the imaging process [21, 22, 23, 24, 25, 26, 27, 28, 29]. There are
several datasets [30, 31, 32, 33] and community challenges [34, 35, 36, 37]
for MRI reconstruction research.

This review provides a comprehensive overview of recent advances and
applications of deep learning (DL) to the reconstruction of magnetic resonance
(MR) images. Given the rapid pace at which this field is evolving,
encapsulating the entirety of the published literature is a formidable
challenge. Previous reviews have laid the groundwork by detailing the
fundamental components of DL architectures [25, 26, 27, 28, 29, 38, 39, 40,
41]. Here, we cover the diversity of approaches, by spotlighting not only the
emergence of novel methods such as self-supervised learning methods and
diffusion models, but also by addressing areas where DL encounters significant
hurdles, including susceptibility to distribution shifts, instabilities, and
inherent biases. Drawing on our hands-on experience, we propose actionable
strategies to enhance the robustness of DL models to mitigate these challenges
effectively.

##  2 Background on MRI reconstruction

###  2.1 Image formation & forward model

The acquisition process in many imaging schemes can be modeled by an operator
$\mathbf{A}$ applied on the continuous domain image $\mathbf{x}$, where the
process of collecting measurements is described by
$\mathbf{y}=\mathbf{A}(\mathbf{x})+\mathbf{n}$. In MRI, the measurement
operator $\mathbf{A}$ commonly corresponds to a multi-coil Fourier sampling
operator. Although the acquisition is continuous, the general practice is to
discretize the problem. Thus, we consider the reconstruction of an image
vector $\mathbf{x}$ from linear measurements, modeled by a matrix
$\mathbf{A}$, by:

$\mathbf{y}=\mathbf{A}\mathbf{x}+\mathbf{n}.$ (1)

The above equation is a numerical model for the imaging device, and is often
referred to as the _forward model_. In many imaging methods, the forward model
is known precisely. However, there are many applications where the forward
model is unknown or only partially known. Examples include imaging in the
presence of motion during the acquisition, trajectory errors, and field
inhomogeneity effects in MRI acquisitions.

Due to MRI’s long scan duration, many scans are accelerated by sampling _k-
space_ at a sub-Nyquist rate. In these cases, the forward model $\mathbf{A}$
is often rank-deficient, making the recovery of $\mathbf{x}$ an ill-posed
problem.

####  2.1.1 Conventional model-based image recovery

When the recovery of $\mathbf{x}$ is ill-posed, many MRI schemes such as SENSE
[5] pose the recovery as an optimization problem
$\mathbf{x}=\arg\min_{\mathbf{x}}\mathcal{L}(\mathbf{x})$ with an objective
function

$\mathcal{L}(\mathbf{x})=\underbrace{\|\mathbf{A}\,\mathbf{x}-\mathbf{y}\|_{2}^{2}}_{\mbox{data
consistency}}+{\lambda}~{}\underbrace{\mathcal{R}(\mathbf{x})}_{\mbox{regularization}}$
(2)

where the first term is often called a _data consistency_ term, and the second
term is called a _regularization_ prior. The objective (2) is sometimes called
a _variational objective_ [42].

The prior $\mathcal{R}:\mathbb{C}^{n}\rightarrow\mathbb{R}_{+}$ is used to
restrict the solutions to the space of desirable images. The prior
$\mathcal{R}(\mathbf{x})$ has a large value when $\mathbf{x}$ is an
undesirable image and is small for a desirable image. A common prior used in
compressive sensing methods is wavelet-domain sparsity, where the number of
non-zero wavelet coefficients or their surrogates are used as priors [9, 10].
In this case, the optimization algorithm facilitates the recovery of an image
$\mathbf{x}$ that has few non-zero wavelet coefficients.

From a Bayesian perspective, the above formulation can be viewed as an
a-posterior estimate [27, 43], where the goal is to find an image $\mathbf{x}$
that maximizes the posterior distribution
$p(\mathbf{x}|\mathbf{y})=\frac{p(\mathbf{y}|\mathbf{x})p(\mathbf{x})}{p(\mathbf{y})}$.
The estimate is obtained by minimizing the negative log posterior

$-\log p(\mathbf{x}|\mathbf{y})=\underbrace{-\log
p(\mathbf{y}|\mathbf{x})}_{\mbox{data consistency}}-\underbrace{\log
p(\mathbf{x})}_{\mbox{prior}}.$ (3)

Here, the first term is the data consistency term. It yields the mean-squared
error in equation (1) if the noise vector $\mathbf{n}$ has i.i.d. Gaussian
entries. Data consistency terms appear in both compressed sensing [9] and DL
methods [44], as they ensure that the reconstructed images adhere closely to
the acquired data. The second term incorporates prior information on the
images [45, 43].

Over the past few decades, substantial research efforts have been dedicated to
crafting effective priors. Tikhonov regularization, for instance, employs a
Gaussian prior on $\mathbf{x}$, resulting in a regularization term
$\mathcal{R}(\mathbf{x})=\|\mathbf{x}\|^{2}$, and compressed sensing methods
mentioned earlier promote sparsity.

####  2.1.2 Optimization algorithms

The loss in equation (2) is typically minimized by applying iterative first-
order optimization algorithms such as gradient descent. Starting from
$\mathbf{x}_{0}$ with a stepsize of $\eta_{t}$, iteration $t+1$ of gradient
descent is described by

$\displaystyle\mathbf{x}_{t+1}$
$\displaystyle=\mathbf{x}_{t}-\eta_{t}\mathcal{L}(\mathbf{x}_{t})$ (4)
$\displaystyle=\mathbf{x}_{t}-\eta_{t}\left({\mathbf{A}}^{T}(\mathbf{A}\mathbf{x}_{t}-\mathbf{y})+\nabla\mathcal{R}(\mathbf{x}_{t})\right).$

The above algorithm depends on the gradient of the regularizer
$\nabla\mathcal{R}$. When $\mathcal{R}(\mathbf{x})=\log p(\mathbf{x})$ is the
log-prior as mentioned earlier, $\nabla R(\mathbf{x})=\nabla\log
p(\mathbf{x})$, which is often referred to as the _score_ of the distribution.
This term can be seen as an estimate of the noise. Subtracting it from the
original image will give a cleaner image. Hence, the above gradient descent
can be seen as a noise removal step, where the alias components specified by
${\mathbf{A}}^{T}(\mathbf{A}\mathbf{x}_{t}-\mathbf{y})$ and the noise
components specified by $\nabla R(\mathbf{x})$ are removed from the current
estimate $\mathbf{x}_{t}$.

Other popular fast iterative algorithms for minimizing the objective (2)
include the alternating direction method of multipliers (ADMM) [46] and the
fast iterative shrinkage thresholding algorithm (FISTA) [47]. For example, the
ADMM scheme considers the equivalent problem,

$\mathbf{x}=\arg\min_{\mathbf{x}}\min_{\mathbf{v}}\|\mathbf{A}\,\mathbf{x}-\mathbf{y}\|_{2}^{2}+{\lambda}~{}\mathcal{R}(\mathbf{v})~{}\mbox{such
that}~{}~{}\mathbf{v}=\mathbf{x}$ (5)

The above problem is solved by alternating between the following steps

$\displaystyle{\mathbf{x}}_{n+1}$ $\displaystyle=$
$\displaystyle\arg\min_{\mathbf{x}}\|\mathbf{A}\,\mathbf{x}-\mathbf{y}\|_{2}^{2}+\beta\|\mathbf{x}-(\mathbf{v}_{n}-\mathbf{u}_{n})\|^{2}$
(6) $\displaystyle{\mathbf{v}}_{n+1}$ $\displaystyle=$
$\displaystyle\beta\|\mathbf{v}_{n}-\underbrace{\left(\mathbf{x}_{n+1}-\mathbf{u}_{n}\right)}_{\overline{\mathbf{x}}}\|^{2}+\lambda
R(\mathbf{v})$ (7) $\displaystyle\mathbf{u}_{n+1}$ $\displaystyle=$
$\displaystyle\mathbf{u}_{n}+(\widehat{\mathbf{x}_{n+1}}-\widehat{\mathbf{v}}_{n})$
(8)

The second step of the above optimization scheme

$\displaystyle{\mathbf{v}}$ $\displaystyle=$
$\displaystyle\arg\min_{\mathbf{v}}\beta\|\mathbf{v}-\overline{\mathbf{x}}\|^{2}+\lambda\mathcal{R}(\mathbf{v})=\mathcal{D}_{\beta}(\overline{\mathbf{x}})$
(9)

can be viewed as a _denoising_ step to clean the current solution
$\overline{\mathbf{x}}$, thus yielding $\mathbf{v}$. For many penalties (e.g.
$\ell_{1}$ norm), the solution to (9) can be evaluated as proximal mapping.
Here, $\beta$ is a continuation parameter that can be interpreted as
$\frac{1}{\sigma_{\beta}^{2}}$, where $\sigma_{\beta}$ is the variance of
_noise_ in $\overline{\mathbf{x}}$ that decreases with the iterations. The
first step (6) involves an inversion step to reduce the cost function composed
of the linear combination of the data consistency error and the deviation from
the _denoised_ image $\mathbf{v}$. This provides an iterative denoising
interpretation, which is used in plug-and-play algorithms (discussed below, in
Section 3.2). One challenge with the above convex optimization schemes is
their high computational complexity, which is due to the numerous iterations
required for convergence. In particular, the data consistency step (6)
involves the evaluation of the forward model and its adjoint, which is often
computationally expensive.

##  3 DL reconstruction: approaches and architectures

There are at least four different approaches to deep learning based MRI: (i)
neural networks trained end-to-end; (ii) approaches based on pre-trained
denoisers, often called plug-and-play methods; (iii) approaches based on
generative models; and (iv) un-trained neural networks. In this section, we
review these four approaches. It is worth mentioning that in the 2020 FastMRI
challenge, the top-performing methods were all neural networks trained end-to-
end [48]. However, relative comparisons of algorithms depend on the problem
setup and metrics. In addition, the field is progressing rapidly, so that new
and diverse benchmarking studies would be valuable.

###  3.1 Neural networks trained end-to-end

Neural networks trained end-to-end are commonly trained to map the acquired
data, which is often noisy and degraded by undersampling artifacts, to a
target, ground-truth image. Their training hence requires such paired data.

Let $f_{\theta}$ be a neural network, which receives the measurements as input
and produces clean, reconstructed images as output. Given a training set
consisting of pairs of measurements and target images
$\\{(\mathbf{y}_{1},\mathbf{x}_{1}),\ldots,(\mathbf{y}_{n},\mathbf{x}_{n})\\}$,
the network is trained by minimizing the loss between the prediction of the
network and the target images, i.e.,

$\displaystyle\mathcal{L}({\bm{\theta}})=\sum_{i=1}^{n}\text{loss}(f_{\theta}(\mathbf{y}_{i}),\mathbf{x}_{i}).$
(10)

####  3.1.1 Networks mapping a noisy measurement to a clean image

Many architectures have been developed for mapping a noisy measurement to a
clean image. Here we review some of the most well-known approaches.

One of the early architectures is known as AUTOMAP (Automated Transform by
Manifold Approximation) [21]. It utilizes a fully connected network followed
by a convolutional network as the architecture $f_{\theta}$ in equation (10).
This architecture does not incorporate the known forward model directly.
Instead, it maps data from _k-space_ to image domain, and learns the forward
model from the data.

Other architectures commonly utilize image-to-image neural networks and
incorporate the forward map. This class of approaches maps a coarse
reconstruction of the image, for example a zero-filled image, to a target
image. In the notation above, the architecture $f_{\theta}$ consists of a
linear map computing the zero-filled image followed by application of an
image-to-image neural network.

The architecture of the image-to-image network is most commonly a
convolutional neural network (CNNs). One of the pioneering works in this area
was by Wang et al. [49], which demonstrated that using a CNN as the image-to-
image network enables substantial improvements in both speed and image
quality. Another pioneering work was by [50], which showed that this approach
is applicable to a wide range of linear forward models.

Additionally, numerous other studies have embedded UNET, ResNet or recurrent
neural networks as the backbone architecture [25, 26, 27, 28, 29, 38, 39, 40,
41]. More recently, vision transformers have been utilized instead of a CNN as
the image-to-image network. Several studies demonstrated that transformers can
provide improvements [51, 52, 53]. However, they are computationally more
expensive. In practice, a UNET provides a very good trade-off between image
quality and computational performance.

####  3.1.2 Unrolled network architectures

Currently, some of the best-performing neural networks are based on unrolled
architectures [48]. These networks are obtained by unrolling an iterative
algorithm such as gradient descent. The idea of unrolled networks was first
introduced by [54], and several pioneering works applied it in the context of
MRI reconstruction [55, 23, 56, 24]. These architectures iterate between two
types of blocks: (i) data-consistency blocks, which can be computed using
different algorithms [44], and (ii) blocks that remove noise and artifacts,
which are commonly implemented by a deep neural network.

One of the early works in this context was by Hammernik et al. [23], who
introduced the _variational network_. This approach relies on a gradient
descent algorithm to minimize the variational objective (2) where the
regularizer $\mathcal{R}$ is taken as the total-variation norm. In this case,
the gradient of the regularizer in the gradient descent iterations (4) takes
the form of a convolution. Thus, the gradient descent iterations can be
interpreted as a neural network that applies data consistency operations
(originating from the gradient of the least-squares loss) and the application
of a convolutional network (originating from the gradient of the regularizer).
Motivated by this observation, these so-called variational networks initialize
$\mathbf{x}_{0}={\mathbf{A}}^{T}\mathbf{y}$ and then perform the following
computations with a neural network:

$\displaystyle\mathbf{x}_{t+1}=\mathbf{x}_{t}-\eta_{t}{\mathbf{A}}^{T}(\mathbf{A}\mathbf{x}_{t}-\mathbf{y})+\mathrm{CNN}_{t}(\mathbf{x}_{t}).$
(11)

Here, both the parameter $\eta_{t}$ and the parameters of the CNN are
learnable. The original variational network [23] used a relatively shallow
CNN, inspired by the parameterization provided by the total variation norm and
its generalizations; specifically, the fields-of-experts-model. This yields a
well-performing network with very few parameters. Later work has also shown
that using a UNET within the unrolled network can improve the overall
performance. [24, 57].

Other unrolled methods adopted alternate algorithms to minimize the
variational loss (4) (described in Section 2.1.2). Those replaced the CNN in
equation (11) with other image-to-image architectures; see [55, 56, 24, 58,
59] for a few examples. Furthermore, other architectures replaced the image-
domain CNN with either a _k-space_ CNN, a dual-domain (_k-space_ and image
domain) network [60, 61, 62] or a transformer [58].

_Computational considerations._ The unrolling step requires multiple physical
realizations of the CNN block during training, which translates into a high
memory demand during training. This restricts their application in higher
dimensional (e.g. 3D, 4D) applications. Programming solutions such as gradient
check-pointing are now available to reduce the memory demand, at the expense
of increased computational complexity. An alternative approach relies on deep
equilibrium models [63, 64]. These models use a single CNN block and iterate
the steps (6)-(8) until convergence to a fixed point, similar to PnP methods.
These methods then implement the fixed point iterations for back propagation.
These methods thus enable the evaluation of the forward and back propagation
using a single physical CNN block, thus reducing the memory demand. The MOL
[65] method also imposes a local Lipschitz constraint on the CNN block, which
offers theoretical guarantees and robustness without sacrificing performance.

###  3.2 Pretrained plug-and-play (PnP) methods

Early CS methods relied on convex priors $\mathcal{R}(\mathbf{x})$, such as
the total-variation norm. Plug-and-play (PnP) methods make it possible to
solve inverse problems with pre-trained denoisers. Another benefit is that
they work with arbitrary forward models, where the prior incorporates
information about the image. One class of PnP methods replaces the proximal
operator $\mathcal{D}_{\beta}$ in (9) with a pre-trained denoiser. While early
methods relied on off-the-shelf image denoisers such as BM3D [66], pre-trained
CNN denoisers are now considered to be more effective [67, 68, 69]. Note that
the proximal step in (9)

$\displaystyle\mathcal{D}_{1/2\sigma^{2}}({\mathbf{x}})$ $\displaystyle=$
$\displaystyle\arg\min_{\mathbf{v}}\frac{1}{2\sigma^{2}}\|\mathbf{v}-\mathbf{x}\|^{2}+\lambda~{}\mathcal{R}(\mathbf{v})$
(12)

can be seen as the maximum a-posteriori (MAP) estimate of $\mathbf{x}$ from
its noise corrupted measurements

$\mathbf{v}=\mathbf{x}+\sigma~{}\mathbf{n}.$ (13)

Here, $\mathbf{n}\sim\mathcal{N}(0,\mathbf{I})$ is a sample from a Gaussian
distribution with variance $\sigma^{2}$. The CNN modules are hence pre-learned
from training data as MAP denoisers, where noise- corrupted images are fed as
input and the model is trained to yield noise-free images. During inference,
steps (6)-(8) are iterated until the algorithm converges to a fixed point.
Similar to CS methods, several iterations are often needed for convergence,
which translates into higher computational complexity than the unrolled
approaches described in section 3.2.

###  3.3 Generative priors

Another successful approach to DL-based MRI reconstruction is to learn an
image prior parameterized by a generative neural network. Several major
classes of generative methods have emerged, based on variational autoencoders
[70, 71], Generative Adversarial Networks (GANs) [72, 73, 74, 75, 76, 77, 78],
and very recently, diffusion models [79, 80, 81, 82, 83, 84, 85, 86, 87, 88].
Here we focus on the two latter ones, which have attracted substantial
attention.

One of the major advantages of generative approaches for image reconstruction
is that they are flexible with regard to changes of the forward model, and at
the same time perform well for reconstructing high-quality images from
undersampled data. Furthermore, their probabilistic nature provides measures
for uncertainty quantification, which is highly important for clinical imaging
[89, 90, 91].

####  3.3.1 GANs

GANs [72] are a framework for generative modeling. A GAN consists of two
competing neural networks: a generator, which aims to produce data
indistinguishable from a given dataset of real images, and a discriminator,
whose role is to distinguish between the generator’s output and the real data.
GANs are trained using an adversarial loss [72]; this process enables the
generator to learn to generate high-quality realistic images. After training,
the generator can be used either to generate images that look similar to those
in the training set, or as a prior for image reconstruction.

In the context of MRI reconstruction, GANs have attracted substantial
attention over the last few years [73, 74, 75, 76, 77, 78]. For example, DAGAN
(Deep De-Aliasing Generative Adversarial Networks) [73] was a pioneering work
that proposed a conditional GAN with a refinement-learning stage, and used a
loss function comprised of an adversarial and a perceptual component. Mardani
et al. [92] proposed a reconstruction framework where GANs were used for
learning the low-dimensional manifold that underlies high-quality MR images.
However, images generated by the generator are not necessarily consistent with
the acquired measurements. To ensure such consistency, they included an affine
projection operation, conducted by a layer placed between the generator and
discriminator. Another approach for tackling this was proposed by Quan et al.
[75], who introduced a novel cyclic loss in their GAN architecture to enforce
data consistency. These methods, and many others [77, 93] showcased the
potential of GANs to produce clinically viable MRI reconstructions.

####  3.3.2 Diffusion models

Diffusion models, a class of generative models that have garnered substantial
attention in recent years, are making an impact in a variety of fields,
including MRI reconstruction [79, 80, 81, 82, 83, 84, 85, 86, 87, 88]. These
models operate by learning to reverse a diffusion process that gradually
transforms random noise into structured images, and have shown a remarkable
capability to generate high-quality, detailed images.

Diffusion models have been derived using different approaches [79, 80],
including discretized corruptions, e.g., denoising diffusion probabilistic
models (DDPMs) [88], denoising score matching [94], and continuous
formulations based on stochastic differential equations (SDEs) [95].

For a general probability density function $p(x)$, these approaches
approximate the _score function_ , defined by $\nabla_{x}\log p(x)$, using a
neural network $s_{\theta}(x)$. To do so, the network is used to approximate a
series of conditional score functions, $s_{t}(x_{t})=\nabla_{x_{t}}\log
p(x_{t}|x_{t+1})$, which guide the denoising process from pure noise, i.e.,
$x_{T}$ drawn from a normal Gaussian distribution for some maximum iteration
value $T$, to a clean sample $x_{0}\sim p(x)$. Once trained, these models can
be used to sample unconditionally from the prior distribution by running the
reverse diffusion process, and hence generating new samples.

In the context of inverse problems in general, and MRI reconstruction in
particular, the diffusion process can be hijacked to approximately sample from
the conditional posterior distribution, $p(y|x)$ instead. One method involves
conditioning on the _k-space_ measurements $y$ and applying Bayes’ rule to the
series of score functions, i.e.,

$\displaystyle\nabla{\log p(x_{t}|y,x_{t+1})}=\nabla{\log
p(y|x_{t})}+\nabla{\log p(x_{t}|x_{t+1})}.$ (14)

The second term, corresponding to the prior conditioned on the denoising
process, is unchanged from the original diffusion model and can be learned by
training on clean, fully sampled images. The first term, corresponding to the
likelihood conditioned on the denoising process, can be approximated through
various approaches [85, 88, 96]. In a naive approximation,

$\displaystyle\nabla{\log p(y|x_{t})}\approx A^{H}(Ax_{t}-y),$ (15)

given the MRI forward model.

A growing body of work demonstrates that diffusion models work well for
accelerated MRI and exhibit flexibility when handling various sampling
patterns [85, 91, 87, 89, 86, 97]. For example, in a pioneering work, Jalal et
al. [98] demonstrated that training a score-based generative model using
Langevin dynamics, without making any assumptions on the measurement system,
could yield competitive reconstruction results for both in-distribution and
out-of-distribution data. Chung et al. [89] demonstrated that score-based
diffusion models trained solely on magnitude images can be utilized for
reconstructing complex-valued data. Luo et al. [90] described a comprehensive
approach using data-driven Markov chains for MRI reconstruction which not only
facilitates efficient image reconstruction across variable sampling schemes,
but also enables the generation of uncertainty maps.

The flexibility afforded by explicitly decoupling the image prior (which is
learned with diffusion models) and the statistical measurement model has also
enabled other extensions. These include incorporating errors into the forward
model, e.g., due to motion [99] and field inhomogeneity [100] and
incorporating multiple image contrasts [101].

###  3.4 Un-trained neural networks

Un-trained methods are DL models that do not rely on training data apart from
hyper-parameter tuning. Instead of conventional training on large datasets,
these methods are typically based on fitting a randomly initialized neural
network to a specific measurement. Here we discuss two types of methods: un-
trained neural networks based on CNNs, and methods based on coordinate-wise
implicit neural networks.

####  3.4.1 Un-trained CNNs for single image recovery

CNNs can be used as an image prior by fitting a randomly initialized CNN with
gradient descent to a measurement. This approach, termed the deep image prior
(DIP), was introduced in a pioneering work by Ulyanov et al. [102]. The
optimization problem is formulated by,

$\mathbf{x}^{*}=\arg\min_{\theta}\|\mathbf{A}\mathbf{x}-\mathbf{y}\|^{2}~{}~{}\mbox{such
that}~{}~{}\mathbf{x}=\mathcal{G}_{\theta}(z),$ (16)

where $\mathcal{G}_{\theta}$ is a CNN generator whose input $\mathbf{z}$ is a
noise vector drawn from some noise distribution. The optimization is performed
using gradient descent or ADAM [103], starting with random initialization of
the network weights, and early stopping is used for regularization. The image
quality first improves with the number of iterations, and then degrades as the
network begins to fit the measurement noise in $\mathbf{y}$. This behavior is
caused by the implicit bias of CNN networks to natural images: when trained
with gradient descent, CNNs fit the smooth images before the noise, as
formalized in [104].

Un-trained networks perform very well for denoising [102, 105] and compressive
sensing (e.g., accelerated MRI). These methods can provably denoise smooth
signals [104] and can provably reconstruct undersampled smooth images [106].
Un-trained networks also work quite well for accelerated MRI; they provide
significant improvement over sparsity-based methods for 2D accelerated MRI
[107].

One key benefit of un-trained networks is that they do not need training data.
However, this benefit comes at the expense of performance; the images produced
by DIP methods are commonly not comparable to those from the pre-trained
networks discussed above. In addition, DIP often suffers from longer run times
compared to the unrolled and direct inversion approaches because of the need
for ADAM or gradient descent optimization during reconstruction.

####  3.4.2 Un-trained CNNs for joint recovery of multiple images

Recently, the DIP framework was extended to dynamic imaging applications [108,
109] where the images in a time series are modeled as the output of a
generator

$\gamma_{t}(\mathbf{r})=\mathcal{G}_{\theta}\left(\mathbf{z}_{t}\right).$ (17)

Unlike the fixed noisy input used in the original DIP work [102], here
$\mathbf{z}_{t}$ are low dimensional latent vectors at a specific time point
$t$. $\mathcal{G}_{\theta}$ is a deep CNN generator, whose weights $\theta$
are independent of $t$. For example, in a free-breathing cardiac MRI, the
images in the time series at a specific time $t$ can be viewed as non-linear
functions of cardiac and respiratory phases captured by $\mathbf{z}_{t}$. This
model (17) can be viewed as a non-linear mapping/lifting from a low-
dimensional subspace $\mathbf{Z}$ to the image space. The low-dimensional
nature of the latent vectors enables the exploitation of the non-local
redundancies between images at different time points, thus facilitating the
fusion of information between them as in [110, 111].

The network parameters $\theta$ and the latent variables $\mathbf{z}$ are
jointly optimized for by minimizing the cost function

$\mathcal{C}(\mathbf{z},\theta)=\sum_{t=1}^{N}\|\mathcal{A}_{t}\left(\mathcal{G}_{\theta}[\mathbf{z}_{t}]\right)-\mathbf{y}_{t}\|^{2}+\lambda_{1}\underbrace{\|\nabla_{\mathbf{z}}\mathcal{G}_{\theta}\|^{2}}_{\mbox{network
regularization}}+{{\lambda}}_{2}\underbrace{\mathcal{R}(\mathbf{z})}_{\mbox{latent
regularization}}.$ (18)

The network regularization is an $\ell_{2}$ penalty on the weights $\theta$,
which was shown to minimize the need for early stopping and provide improved
performance. The latent vector regularization term involves a smoothness
regularization to capitalize on the temporal smoothness of the images in the
time series.

The above approach can also be extended to 3D applications, where the joint
alignment and recovery of data from different slices obtained using different
acquisitions may differ in cardiac/respiratory motion. Different set of latent
vectors are used for different slices to account for differences in breathing
patterns and cardiac motion. In this case, a Kullback-Leibler divergence term
is used to encourage the latent vectors of all the slices to follow a zero-
mean Gaussian distribution, thus facilitating the alignment of data from
different slices.

####  3.4.3 Coordinate-based networks

Coordinate-based neural representations, also known as implicit neural
representations (NeRF-type networks), have recently emerged as an efficient
way to represent and work with images, 3D shapes, and other signals [112].
They are commonly used for representing scenes and performing view syntheses
in vision [113, 114, 115]. To represent a 2D or 3D object, these models map a
coordinate input (e.g., $(x,y)$-coordinate for 2D or $(x,y,z)$-coordinate for
3D) to a pixel value, e.g., a real number for a gray-scale image and two real
numbers for a complex-valued image.

Coordinate-based networks can be used in an analogous fashion to un-trained
CNNs to reconstruct an image [116, 117]. Specifically, they can replace the
CNN in an un-trained network and be fitted to measurement data. Networks with
Fourier-feature input (like NerF [113], SIREN [118], and Fourier Feature
inputs [116]) impose a smoothness prior similar to the un-trained CNN
discussed in the previous section and work reasonably well. However, they
perform slightly worse in terms of image quality. In the context of MRI
reconstruction, coordinate networks are useful for high-dimensional objects
such as 3D volumes and scenarios with motion. For example, [119] and [120]
used coordinate networks successfully for free-breathing cardiac MRI
reconstruction and achieved good results.

###  3.5 Self-supervised methods

Neural networks, such as the end-to-end networks discussed in Section 3.1 are
usually trained in a supervised manner (see equation (10)). This requires
pairs of measurement and target (ground-truth) images. However, in practice,
such pairs cannot always be acquired, e.g., due to scan time constraints,
signal decay effects along echo trains, or physiological motion. Therefore,
self-supervised methods are attracting increased research interest. These
methods make it possible to train networks without target or ground-truth data
by either making assumptions on the measurements or using additional noisy or
partial measurements. A plethora of approaches has been developed, including
methods for learning from under-sampled data [121, 122], unpaired data [123],
or limited-resolution data [124]. Here we describe several approaches that are
architecture-agnostic. For recent reviews on this topic see [125, 126, 123].

####  3.5.1 Learning of algorithms based on Stein’s Unbiased Risk Estimate
(SURE)

We start with a method that is based on assumptions on the noise distribution,
called Stein’s Unbiased Risk Estimate (SURE) [127]. We consider the estimation
of $\mathbf{x}$, denoted by $\widehat{\mathbf{x}}$ from its noisy measurements
$\mathbf{v}=\mathbf{x}+\mathbf{n}$. Here $\mathbf{n}$ is zero-mean Gaussian
noise with a variance of $\sigma^{2}$. In practice, the estimate
$\widehat{\mathbf{x}}$ is derived from the noisy measurements $\mathbf{v}$
using a deep network as $\widehat{\mathbf{x}}=f_{\phi}(\mathbf{v})$. When the
noiseless reference image $\mathbf{x}$ is available, the true mean-square
error (MSE), denoted by

${\rm MSE}=\mathbb{E}_{\mathbf{x}}~{}\|\widehat{\mathbf{x}}-\mathbf{x}\|^{2}$
(19)

can be used.

By contrast, the SURE [127] approach uses the loss function

${\rm
SURE}(f_{\phi}(\mathbf{v}),\mathbf{v})=\|f_{\phi}(\mathbf{v})-\mathbf{v}\|^{2}_{2}+2\sigma^{2}\nabla_{\mathbf{v}}\cdot
f_{\Phi}(\mathbf{v})-N\sigma^{2},$ (20)

which is an unbiased estimate of (19). Note that the expression in (20) does
not depend on the noise-free images $\mathbf{x}$; it only depends on the noisy
images $\mathbf{v}$ and the network parameters $\phi$. In (20),
$\nabla_{\mathbf{u}}\cdot f_{\Phi}(\bm{u})$ represents the network divergence,
which is often estimated using Monte-Carlo simulations [128]. Several
researchers have adapted SURE as a loss function for the unsupervised training
of deep image denoisers [129, 130] and demonstrated performance approaching
that of supervised methods.

The SURE approach was extended to inverse problems with a rank-deficient
measurement operator known as the generalized SURE (GSURE) [131]. The GSURE
provides an unbiased estimate of the projected MSE, which is the expected
error of the projections in the range space of the measurement operator. The
GSURE approach was recently used for inverse problems in [129]. The
experiments in [129] showed that the GSURE-based projected MSE was a poor
approximation of the actual MSE in the highly undersampled setting. To improve
performance, the authors trained the denoisers at each iteration in a message-
passing algorithm in a layer-by-layer fashion using classical SURE, which was
termed LDAMP-SURE [129]. This approach approximates the residual aliasing
errors at each iteration to be Gaussian random noise. As this assumption is
violated in many inverse problems, the performance of this layer-by-layer
training approach is not as good as supervised methods.

The ENSURE framework circumvents the poor approximation of the true MSE by
GSURE by considering different sampling operators for different images.
Similar to classical SURE metrics [132, 131], the ENSURE loss metric has a
data consistency term and a divergence term. The data consistency term in
ENSURE is the sum of the weighted projected losses [131] from multiple
subjects; the weighting depends on the class of sampling operators. When
different sampling patterns from different subjects fully cover _k-space_ ,
the ENSURE metric is an unbiased estimate of the true image-domain MSE and
hence is a superior loss function than projected SURE [131]. The comparison of
the above methods shows that the the ENSURE approach can provide performance
comparable to that of supervised training.

####  3.5.2 Self-supervised DL based on Noise2noise

Noise2noise [133] is a well-established framework, which constructs a self-
supervised loss based on independent noisy measurements of the same object. In
the context of MRI, suppose we are given two undersampled measurements
$\mathbf{y}=\mathbf{M}\mathbf{F}\mathbf{x}$ and
$\mathbf{y}^{\prime}=\mathbf{M}^{\prime}\mathbf{F}\mathbf{x}$, where
$\mathbf{x}$ is the image of interest, $\mathbf{F}$ is the Fourier transform,
$\mathbf{M}$ is a random undersampling mask and $\mathbf{M}^{\prime}$ is a
different random mask, with each diagonal entry chosen independently. From
these measurements, we can construct the self-supervised loss

$\displaystyle\ell_{\text{SS}}(f_{\bm{\theta}}(\mathbf{y}),\mathbf{y}^{\prime})={\left\|\mathbf{M}^{\prime}\mathbf{F}f_{\bm{\theta}}(\mathbf{y})-\mathbf{y}^{\prime}\right\|}_{2}^{2}.$
(21)

It can be shown that in expectation over the random measurements, a minimizer
of the self-supervised loss is also a minimizer of the expectation of the
supervised loss (see Prop. 2 in [134]). Thus, with enough training examples,
such self-supervised training can approach the performance of supervised
training [134].

One notable method that has implemented this approach successfully for MRI
reconstruction is Self-Supervised Learning via Data Undersampling (SSDU)
[121]. This method partitions available _k-space_ measurements into two
disjoint sets; the first set is used in the data consistency units of the
unrolled network, i.e., for the forward pass, and the other one is used for
computing the loss, i.e., for supervision. SSDU can hence be trained using
under-sampled data alone. In their work, Yaman et al. [121] demonstrated that
SSDU achieved comparable performance to fully supervised learning methods
while offering practical advantages in real-world MRI applications.

Recently, Millard and Chiew [135] introduced a general theoretical framework
that extends Noiser2Noise [136] and also explains SSDU. Unlike the SSDU
formulation, where one set is recovered from the other, they applied two
subsampling masks to the data. They proposed a weighted $\ell_{2}$ loss,
computed in _k-space_ , with a weighting that compensates for the sampling and
sampling-partitioning densities. They derived the framework analytically and
showed that when the weighting matrix $W$ is rank-deficient and fulfils
certain conditions, the method boils down to SSDU. They showed analytically
that SSDU with an $\ell_{2}$ _k-space_ loss approximates fully sampled
reconstruction, on expectation. It is worth mentioning that their analysis was
done for an $\ell_{2}$ _k-space_ loss, while the original SSDU method was
trained with a mixed $\ell_{1}/\ell_{2}$ loss.

####  3.5.3 Self-supervised DL using limited-resolution data (_k-space_ bands)

The self-supervised methods described above focused on learning from under-
sampled data acquired with variable-density or parallel-imaging schemes.
Although such datasets have undersampling artifacts, they effectively
constitute high-resolution data, because the sampling masks commonly cover the
entire _k-space_ extent (note that under-sampling creates artifacts but does
not necessarily reduce the resolution). However, the acquisition of high-
resolution data can be challenging. In dynamic MRI, for example, there is
often a trade-off between the spatial and temporal resolutions, which requires
acquisition compromises.

Recently, the _k-band_ framework was proposed for self-supervised learning
from partial, _limited-resolution_ data [124, 137]. This framework is based on
the acquisition of _k-space_ bands, where each band acquires data with high
resolution in the MRI readout dimension and limited resolution in the phase
encoding (PE) dimension. The authors suggested acquiring different bands from
different subjects, and randomizing the bands’ orientation across subjects;
fundamentally, this randomization serves to expose the network to all _k-
space_ areas across the training iterations (Fig. 1). Thus, even though the
network does not get a full _k-space_ from any single subject, it can learn
connections across all the _k-space_ regions. To enable self-supervised
learning from limited-resolution data without limiting the resolution during
inference, the authors introduced an optimization method dubbed _stochastic
gradient descent (SGD) over _k-space_ subsets_.

In this framework, the loss is computed in _k-space_ and formulated by

$\displaystyle\ell_{k\\_band}=\|\mathbf{W}\mathbf{B}(\mathbf{F}f_{\theta}(\mathbf{y})-\mathbf{F}\mathbf{x})\|_{1}$
(22)

where $\mathbf{B}\in\\{\mathbf{B}_{i}\\}_{i=1,...,180}$ is a binary band
sampling operator that samples a band with angle $i$, and $\mathbf{W}$ is the
loss weighting mask

$\displaystyle\mathbf{W}=180(\sum_{i=1}^{180}\mathbf{B}_{i})^{-1}.$ (23)

This loss-weighting compensates for the over-exposure of the network to low-
frequency _k-space_ data and enhances learning in the _k-space_ periphery.
This is beneficial because in the _k-band_ acquisition setting, the center of
_k-space_ is included in all bands (Fig. 1), unlike the periphery. The authors
showed analytically that when this loss-weighting mask is applied, the self-
supervised training process stochastically approximates fully supervised
training, on expectation. They demonstrated that learning from limited-
resolution data can hence result in performance comparable to supervised and
self-supervised methods trained on high-resolution data, and hence offers a
practical solution for cases where such data are unavailable.

Figure 1: Example of the input _training_ data for three DL reconstruction
methods. The fully-supervised MoDL method [24] receives var-dens sampled data
as input and uses the entire _k-space_ for supervision. The self-supervised
SSDU method [121] receives var-dens data as input, splits it into two subsets,
and uses one set for data consistency and the other for supervision. In this
example, the var-dens data were sampled from parallel-imaging (equispaced)
acquired data, as in [121]. The _k-band_ method [124] receives var-dens
sampled data from a _k-space_ band, and uses data from the whole band for
supervision, without any supervision outside the band. Different bands are
acquired from different subjects, with random orientations. At _inference_ ,
the input to all three methods is var-dens data from the entire _k-space_ ,
similar to that shown here for MoDL.

####  3.5.4 Loss-weighting

Both [124] and [122] demonstrated that applying loss-weighting in _k-space_
can enhance the performance of self-supervised DL methods. Interestingly,
these studies reached similar conclusions while analyzing different
reconstruction frameworks. Millard and Chiew [122] analyzed SSDU with an
$\ell_{2}$ _k-space_ loss, while Wang et al. [124] analyzed _k-band_ with an
$\ell_{1}$ _k-space_ loss. Another difference is that Millard and Chiew [122]
used variable-density masks that cover the entire _k-space_ , while Wang et
al. [124] focused on _k-space_ band acquisition. However, both studies derived
loss-weighting masks that weigh down the loss in the center of _k-space_ and
enhance it in the periphery. In other words, these masks inhibit the learning
of low-frequency data and facilitate the learning of high-frequency details,
so that eventually all frequencies are weighted equally. Given that different
settings were analyzed, this strategy may be effective for other methods as
well.

###  3.6 Recent architectures: transformers and dual-domain networks

In addition to the training methods described above, much progress has also
been made in the development of advanced architectures. For example, two
architectures that recently garnered substantial attention are transformers
and dual-domain networks. Transformers [138, 139] have powerful computational
capabilities due to their use of an attention mechanism [140] that makes it
possible to weigh the importance of different parts of the input data and
capture long-range dependencies. Transformers first made a substantial impact
in the field of natural language processing [141, 142] and then became highly
influential in computer vision [138].

In the context of MRI reconstruction, recent studies demonstrated that
transformers offer excellent performance and ability to deliver improved
structural and textural fidelity. For example, Korkmaz et al. [143] developed
an unsupervised MRI reconstruction method based on a generative vision
transformer. Their method utilizes cross-attention transformer blocks, which
receive both global and local latent variables as input and progressively map
them to MR images with increasing spatial resolution. This style-generative
architecture enhances representational learning and improves model
invertibility. Feng et al. [144] introduced the $T^{2}Net$ for simultaneous
MRI reconstruction and super-resolution. This network has two branches
dedicated to these two tasks, and incorporates a task transformer module to
facilitate effective feature sharing between them. Guo et al. [53] introduced
the ReconFormer, an architecture that leverages recurrent pyramid transformer
layers and scale-wise attention mechanisms. It effectively captures multi-
scale information and deep feature correlations, leading to efficient, high-
quality image reconstruction and computational efficiency.

Another emerging type of architecture is known as dual-domain networks, which
integrate information from the image and _k-space_ domains [145, 61, 146, 147,
148]. This approach, exemplified by MD-Recon-Net [147], leverages the
complementary strengths of these two domains to enhance reconstruction
quality. A study by Souza et al. [145] demonstrated the effectiveness of such
networks in multi-channel MRI reconstructions. Singh et al. [148] demonstrated
that layers utilizing joint learning of image and frequency domain features
can directly replace standard convolutional layers. This is useful for
numerous tasks, including image reconstruction, motion correction and
denoising.

Transformers and dual-domain networks have recently been integrated, leading
to state-of-the-art architectures. For example, Zhao et al. [149] introduced
SwinGAN, a dual-domain Swin Transformer-based GAN. This network combines
frequency-domain and image-domain generators, both utilizing Swin Transformer
backbones. This design allows for effective capture of long-distance
dependencies in MR images. SwinGAN also features a contextual image relative
position encoder, which enhances its ability to capture local information.
Wang et al. introduced DCT-Net, a dual-domain transformer network for MRI
reconstruction [62], which integrates image and frequency domain information
through its cross-attention and fusion-attention blocks. DCT-Net is designed
to enhance MRI reconstruction performance, particularly under low sampling
rates, by leveraging the complementary strengths of both domains. In summary,
these recent architectures offer high computational power to improve image
reconstruction quality.

##  4 DL for acquisition optimization

In this section we review two applications in which DL is utilized to improve
MRI acquisition. We first explore methods that optimize the _k-space_ sampling
trajectories in tandem with the reconstruction, and then turn to advances in
harnessing DL to refine MRI pulse sequences.

###  4.1 Optimizing _k-space_ trajectories

The computational design of sampling patterns has a long history in MRI.
Generally, two types of approaches have been taken. _Algorithm-agnostic_
methods, e.g. [150, 151, 152, 153, 154], consider specific image properties
(e.g., the Cramér-Rao bound or image support) and optimize the sampling
pattern to improve the measurement diversity for that class. _Algorithm-
dependent_ methods, on the other hand, e.g. [155, 156, 157, 158], optimize the
sampling pattern assuming specific reconstruction algorithms. These are
typically CS algorithms, which employ regularizers such as TV, wavelet-domain
sparsity, or pre-trained diffusion models [158].

The main challenge with the above computational approaches is their high
computational complexity. In particular, algorithm-dependent schemes need to
solve the CS problem for each image in the dataset, to evaluate the loss for a
specific sampling pattern. The design of sampling pattern thus involves a
nested optimization strategy; the optimization of the sampling patterns is
performed in an outer loop, while image recovery is performed in the inner
loop to evaluate the cost associated with the sampling pattern.

DL provides an opportunity to speed up the computational design, because DL
inference schemes enable rapid evaluation of the loss for each sampling
pattern. This enables a joint strategy that simultaneously optimizes the
acquisition scheme and the reconstruction algorithm. Early DL-based joint
optimization schemes solved for a binary sampling mask [159, 160]. The PILOT
approach, for example, solved for the sampling density, assuming a sampling
layer similar to variational auto-encoders [159]. The LOUPE method, on the
other hand, learned the optimal sampling density in tandem with a
reconstruction network [160]. It was first developed for 2D Cartesian imaging
[160] and later extended to non-Cartesian sampling [161]. Other studies have
focused on 3D Cartesian sampling with a variational reconstruction network
[162].

More recent work represents the sampling locations $\phi$ as continuous
variables and jointly solves for them and for the parameters of the DL
algorithms. These methods consider a forward model $\mathbf{A}_{\phi}$, where
$\phi$ denotes the sampling locations. This forward model may be represented
either by using an analytical Fourier transform [163] or a non-uniform Fourier
transform [164]. We denote the reconstruction algorithm (which can be
unrolled, direct inversion, or plug-and-play) by

$\hat{\mathbf{x}}=\mathcal{M}_{\theta,\phi}(\mathbf{y}),$ (24)

where $\theta$ denotes the parameters of the reconstruction algorithm and
$\phi$ are the sampling locations corresponding to the forward model. Joint
optimization schemes, e.g. [163, 164], are designed to optimize the sampling
pattern $\Phi$ and the CNN parameters $\phi$ in tandem, i.e.,

$\\{\theta^{*},\phi^{*}\\}=\arg\min_{\theta,\phi}\sum_{i=1}^{N}\|\mathcal{M}_{\theta,\phi}\left(\mathbf{A}_{\phi}(\mathbf{x}_{i})\right)-\mathbf{x}_{i}\|_{2}^{2}.$
(25)

Several methods have been developed within this framework. For example, J-MODL
focuses on a model-based reconstruction and utilizes an unrolled network
[163]. In a different work, Wang et al. [164] parameterized trajectories with
quadratic B-spline kernels, and performed optimization under penalties
describing realistic MRI hardware constraints, e.g. the slew rate and gradient
amplitude. This work was later extended to a generalized Stochastic
optimization framework for 3D NOn-Cartesian samPling trajectorY (SNOPY) [165],
which can accomodate several optimization objectives. Chaithya and Ciuciu
[166] introduced the PROJeCTOR framework, which enables joint learning of non-
Cartesian trajectories and reconstruction networks by using a projected
gradient descent algorithm. Alkan et al. [167] introduced joint sampling and
reconstruction optimization through variations in information maximization,
where they used an encoder to represent non-uniform sampling and a decoder in
an unrolled neural network. Xie et al. [168] introduced the PUERT method for
learning probabilistic sampling patterns along with an interpretable
reconstruction method; their learning module incorporated a dynamic gradient
estimation strategy. Finally, Zou et al. [169] demonstrated that joint
optimization can reduce the bias and uncertainty of pharmacokinetic parameter
estimation in dynamic contrast enhanced MRI, and hence contribute to higher
diagnostic value . Altogether, these methods have shown significant benefits
from jointly optimizing the sampling pattern and the reconstruction algorithm.

###  4.2 Pulse sequence design

The previous section focused on accelerating MRI scans via _k-space_ sub-
sampling. The complementary element to this effort is the optimization of the
remaining pulse sequence parameters by a set of radio-frequency (RF) powers,
shapes, and duration that enable the shortest possible scan time, while
retaining sufficient contrast, SNR, and consistency with conventional (and
lengthy) alternatives.

The pulse sequence design task was traditionally hand-crafted by MR experts,
who combined strong intuitions and an understanding of spin physics with
mathematical solutions of the Bloch equations. While a remarkable number of
contrast mechanisms and imaging schedules have been developed since the
invention of MRI, the reliance on solvable differential equations severely
limits our ability to reach a globally optimized schedule and reduce the scan
time. Recent developments in DL architectures and computational frameworks
have created new opportunities for the automatic and efficient optimization of
rapid acquisition protocols.

Zhou et al. [170, 171] introduced the representation of the Bloch equations as
a computational graph. By treating each of the acquisition parameters as a
neural network node weight, an efficient gradient-descent-based optimization
was realized, where simulated signal trajectories were fed into the network,
enabling an automatic generation of pulse sequences. The resulting protocols
were characterized by non-intuitive gradient waveforms, where continuous off-
resonant excitation applied as the receive channel was continuously and
simultaneously recorded. This approach yielded an ultra-short scan time for
T1/T2 mapping at 1D. By expanding for 2D imaging, Lee et al. [172] used
automatic differentiation to optimize the Cramér-Rao Lower Bound (CRLB) of
multiple-echo spin echo T2 mapping, driven equilibrium single pulse
observation of T1 (DESPOT1) mapping, and the MRF IR-FISP sequence.

Loktyushin et al. [173] developed a supervised learning framework termed MR-
Zero where a target contrast of interest is used for learning the optimal set
of RF events, the gradient moment, and the delay times (Fig. 2). One important
feature of this approach is the use of a task-driven cost function that
provides the user with the flexibility to prioritize the characteristics
required from the output protocol, such as high data fidelity, short scan
time, or the specific absorption rate (SAR) limits. In a later study, the same
group used this approach to optimize the refocusing flip angles and minimize
T2-induced blurring in accelerated spin echo sequences [174].

Figure 2: Automated discovery of MRI acquisition protocols using supervised
learning. A differentiable MR scanner utilizes the Bloch equations for in-
silico signal generation and the later reconstruction of the target contrast
of interest from real, acquired data. Reproduced from Loktyushin et al. [173].

In the molecular MRI field, an end-to-end DL-based framework was developed for
the discovery of rapid, quantitative chemical exchange saturation transfer
(CEST), semisolid MT acquisition and reconstruction protocols [175]. The
system was based on a computational graph representation of the Bloch-
McConnell analytical solution which receives the molecular imaging scenario of
interest as input, and outputs an optimized set of acquisition parameters and
the corresponding reconstruction network that translates the raw data into
quantitative parameter maps. In vivo experiments showed it could acquire in-
vivo data in merely 35 seconds and reconstruct parameter maps in less than 1
second. The use of recurrent neural networks and training over a wide range of
saturation pulse frequency offsets have further increased the robustness of
this conceptual approach for B0 and B1 inhomogeneity [176].

All these approaches exploit DL-based strategies to optimize and derive novel
acquisition routes offline. Recently, a different optimization paradigm was
suggested where the acquisition parameters are modified and adapted on the
fly, during data acquisition [177]. By combining a Bayesian framework, CRLB
calculation, and model-based reconstruction, the acquisition parameters for a
series of images can be optimized in real-time based on the previous image
history. This concept has demonstrated up to a 3.3 fold acceleration of multi-
echo sequences in human subjects and molecular imaging phantoms.

##  5 Advanced techniques and applications

In this section we discuss DL methods for quantitative MRI and dynamic MRI.

###  5.1 DL methods for quantitative MRI

The goal of quantitative MRI is to extract one or more tissue parameter maps
from a series of qualitative images [178]:

$\mathbf{I}_{m}=\mathbf{\Phi}_{m}(T_{param})\mathbf{\rho}$ (26)

where Im denotes the contrast-weighted images for m=1,…,M acquisitions, $\rho$
denotes the spin density, Tparam denotes the tissue parameters (T1, T2, etc.),
and $\Phi_{m}$ is the biophysical function connecting the acquisition
parameters with the resulting contrast-weighted images. The mapping of tissue
properties enables de-biasing imaging protocols and harmonization of the final
diagnosis across sites, vendors, and physicians. It thus provides sensitive
and standardized tools for reproducible interpretation of MRI-based
information [179]. The classical approach to MR property mapping mandates
repeated acquisition, where all the protocol parameters are held fixed, and
only a single parameter is slowly and gradually varied (e.g., the flip angle
or the repetition time across different M acquisitions). The resulting long
scan time hinders the widespread use of quantitative MRI in clinical settings
[180]. The reconstruction of the acquired raw data series demands a lengthy
parameter-fitting procedure that is computationally intensive and slow.

The development of powerful DL architectures such as CNNs, UNets, GANs,
ResNets and recurrent neural networks has been leveraged to accelerate and
enhance the performance of quantitative MRI [181]. To accelerate relaxometry
studies, Liu et al. [182] developed a model-augmented neural network that
receives a series of incoherently-sampled multi-echo images and uses a CNN to
reconstruct the T2 parameter maps. The supervised learning was guided by a
parameter-space loss, which compares the reconstructed T2 maps to the ground
truth reference, and a _k-space_ loss. The latter was designed to ensure that
the physical-model-based synthetic undersampled _k-space_ measurements matched
the originally acquired _k-space_ information. In a later work, the same group
developed a model-guided self-supervised DL framework for rapid T1/T2 mapping
[183], to obviate the need for fully sampled training references.

While many DL-based quantitative mapping strategies are focused on _k-space_
sub-sampling, a further acceleration potential lies in reducing the number of
contrast-weighted images acquired. In a very recent work, Li et al. [184]
trained a deep residual CNN network to receive just three _k-space_ under-
sampled contrast weighted images and output the corresponding T1rho and T2
parametric maps (which are particularly useful for the study of
osteoarthritis).

Magnetic resonance fingerprinting (MRF), which was first reported in a 2013
Nature paper [185] and increasingly studied since then, constitutes a paradigm
shift in MRI-based tissue characterization. Unlike traditional relaxometry
studies, MRF starts with the acquisition of tens or hundreds of images, using
a pseud-random acquisition pattern accommodating a series of short repetition
times, small flip angles, and heavily under-sampled _k-space_ data (e.g., via
a single variable density spiral trajectory). Although each of the resulting
raw images is extremely noisy, the temporal evolution of the signal at each
pixel entails a unique fingerprint. By comparing the experimental trajectory
to a Bloch-equation-derived dictionary of simulated signals, the inverse
problem can be solved to uncover the associated parameter maps (Tparam).

Figure 3: Deep learning reconstruction of quantitative magnetic resonance
fingerprinting (MRF) information. (a) A fully connected neural network is
trained using simulated signal trajectories. During inference, it receives a
series of raw MRF images pixel-wise, as well as auxiliary maps, yielding
quantitative parameter maps. (b) A further acceleration in scan time can be
achieved by training a generative adversarial network (GAN) using a smaller
subset of raw input data to yield the same quantitative output maps.
Reproduced and modified from Weigand-Whittier et al. [186].

While the resulting acquisition times are incredibly short (e.g., $<$13 s
[187, 185]) the quantitative image reconstruction step (via pattern matching)
may take hours, because the similarity between each acquired signal trajectory
and all possible dictionary entries needs to be calculated.

In recent years, several DL-based strategies have been suggested to overcome
this challenge. Cohen et al. [188] trained a fully connected neural network
using synthetic signal dictionaries to reconstruct MRF data in less than 100
ms. To take advantage of the inherent dependencies between adjacent image
pixels, Balsiger et al. [189] designed a spatiotemporal CNN where the time-
evolution dimension is the third dimension of the CNN patch kernel. In-vivo
human brain validation studies using this approach demonstrated improved
performance compared to alternative MRF networks [188, 190]. To accommodate 3D
imaging, Gomez et al. [191] combined fully connected reconstruction networks
with radial and spiral readout trajectories, and achieved whole-brain
reconstruction in less than 7 min (compared to $>1.5$ hours using traditional
reconstruction).

Another MRF-associated bottleneck relates to the time required to generate the
synthetic signal dictionary, which increases exponentially with the number of
simulated parameters [192]. Even when high-end computer clusters are used for
this task, the computation time may reach hours/days for complex multi-pool
imaging [193, 194]. Recently, this challenge was addressed by training a fully
connected neural network using a variety of dictionaries to learn the
nonlinear relations embedded in the physical model. The resulting system
enabled the rapid generation of simulated signals for various protocols and
imaging scenarios [195]. NN-based simulators can be further combined with
reconstruction networks to provide a unified rapid method for MRF analysis
[196]. A different approach to circumvent the need for exhaustive dictionary
generation involves the direct synthesis of multi-contrast images (e.g.,
T1-weighted, T2-weighted, and FLAIR) from raw MRF data. While synthetic images
can be derived from quantitative MRF data by forward model activation using
the desired acquisition parameters [197, 198], Wang et al. [199] showed that a
dictionary-free conditional GAN (trained on MRF raw data and paired ground
truth weighted images) can perform the same task much faster. For cases where
full quantitative information is required, a different work demonstrated that
multi-parameter maps can still be extracted with GANs, even when merely 30$\%$
of the acquired MRF data is used [186] (Fig. 3).

###  5.2 Dynamic MRI

Deep learning has become a transformative force in the realm of dynamic MRI,
particularly in addressing the challenges related to limited acquisitions and
motion correction, which constitute substantial hurdles in clinical imaging
[2, 41, 200, 201, 202]. DL excels at learning signal evolution [203, 146], a
critical factor when aiming to accurately visualize and interpret dynamic
changes in the body.

####  5.2.1 Motion-resolved reconstruction

Motion-resolved algorithms can effectively learn spatio-temporal correlations
and reconstruct images from highly undersampled sequential data [204, 205].
These methods have primarily been developed in the context of cardiac MRI
[200, 56, 111, 206, 204, 207]. For instance, supervised unrolled algorithms
have been used to recover cardiac cine MRI from breath-held MRI using using 4D
(3D+Time spatial) convolutions [200, 56]. In other studies, architectures
included unrolled algorithms that combine manifold [111] or low-rank priors
[206, 204], and joint learning of motion estimation and segmentation in
cardiac MRI [207]. From a clinical perspective, DL has been found useful for
measuring myocardial displacement [208], noninvasive diagnosis of myocardial
ischemia [209], and evaluation of cardiac function in pediatric imaging [210].

To tackle the scarcity of training data, _unsupervised_ implicit learning
approaches have recently been introduced for dynamic MRI [205, 211] (for more
information see Section 3.4.2). These methods were also extended to multi-
slice dynamic MRI data [212], where the dynamic data from each slice
$\mathbf{x}_{i}(t)$ are acquired sequentially at different time points. The
above model has been generalized to recover a pseudo-3D reconstruction by
modeling the data as
$\mathbf{x}_{i}(t)=\mathcal{G}_{\theta}\Big{(}\mathbf{z}_{i}(t)\Big{)}$, where
$\mathbf{z}_{i}(t)$ are allowed to vary for different slices.

####  5.2.2 Motion-compensated reconstruction

The development of DL techniques for motion estimation and correction is a
highly active research field, as DL can accurately detect and compensate for
both rigid and non-rigid motion artifacts, which leads to more diagnostically
valuable images. A thorough review of motion estimation and correction
techniques is beyond the scope of this manuscript Here we highlight some of
the main applications, and more information can be found in recent reviews
[213, 41, 40, 214, 215].

One of the main applications where DL is highly effective for motion
correction is brain MRI, which is characterized by rigid-body motion [214,
215, 216, 217]. One of the early works in this field, by Johnson and Dragnova
[202], proposed conditional GANs to infer clean images from motion-corrupted
data. More recent techniques include co-optimization for jointly estimating
the motion parameters and reconstructed image [216, 123], methods for
detection and correction of motion-corrupted _k-space_ lines [218, 217] and
the use of score-based generative models [219].

DL approaches are also highly useful for tackling _non-rigid_ , irregular
motion. Applications include imaging of the trunk [220], fetal MRI [221, 222],
abdominal MRI [223, 224], and MR angiography [225]. Unsupervised implicit
learning methods that represent the deformable motion fields at each time
point have also been introduced and found to be effective in motion-
compensated recovery [226]. DL is also making strides in the field of real-
time interventional MRI. Here, the rapid processing capabilities of DL
algorithms enable real-time feedback and guidance during medical procedures,
thus enhancing both the safety and efficacy of interventions. [227, 228, 229]

###  5.3 Multi-task pipelines

The fundamental goal motivating the acquisition of diagnostic-quality MR
images is to extract clinically-used insights to further clinical care or to
interrogate disease activity. Consequently, efficient, high-quality image
acquisition is just the first step (typically referred to as _upstream DL_) in
the imaging workflow, which is followed by image analysis and insight
extraction (typically referred to as _downstream DL_) [230]. In many
applications, these upstream and downstream processes are disconnected,
leading to insufficient insights as to whether a novel MRI acceleration and
reconstruction technique can reliably produce the requisite diagnostic
information [231]. As a result, there is a substantial need to combine the
upstream and downstream processes to ultimately harness advances in MRI
physics, hardware, and DL for end-to-end acquisition-to-analysis workflows.

Conventional and DL-based reconstruction techniques can potentially be
combined with downstream task of clinical utility to guide useful model
development. Specifically, MRI reconstruction workflows can be combined with
three different downstream tasks that use whole images as inputs: (i) _image
classification_ , which performs binary identification (via a yes/no) to
identify the presence of one or multiple disorders; (ii) _abnormality
detection_ , which performs localization via bounding boxes to accurately
depict where one or multiple disorders are present in images; (iii) _image
segmentation_ , which performs image classification at the voxel level to
distinguish voxels that belong to a particular tissue or disease class. The DL
sub-field of multi-task learning can learn multiple tasks simultaneously with
_positive task transfer_ , where learning one task improves the performance of
other tasks.

In the context of accelerated MRI, combining the upstream task of MRI
reconstruction with the downstream tasks of classification, detection, or
segmentation can improve performance on all tasks. It also contributes to
optimizing reconstruction techniques with clinically informed metrics. One of
the greatest challenges in doing so, however, is the lack of available
datasets that can merge both sets of tasks. The fastMRI raw-data dataset was
recently supplemented with the fastMRI+ dataset that includes classification
and detection bounding box annotations for knee and brain abnormalities at the
slice level [232]. Such datasets can enable the design of end-to-end
techniques to optimize reconstruction, subject to high performance on lesion
detection [233]. Similarly, even beyond end-to-end methods, such abnormality
labels can be used to design clinical task-specific undersampling trajectories
[234].

Beyond fastMRI+, SKM-TEA datasets include raw _k-space_ data as well as
classification labels, detection bounding boxes, segmentation masks, and
quantitative T2 relaxation time maps [33]. The original work profiled how
different reconstruction approaches combined with different segmentation tools
affected a common musculoskeletal biomarker of cartilage T2 relaxation time.
Despite differences in the performance of the individual DL blocks, the
overall impact on regional cartilage T2 values was small, a surprising finding
that has been replicated for cartilage morphology and T2 tasks [235, 236].
Recent work has evaluated new approaches that combine generic pre-training
tasks such as image reconstruction with fine-tuning for different clinically-
relevant downstream tasks [237]. This approach achieves high performance in
image acceleration as well as segmentation. Similar to these findings, the K2S
challenge at MICCAI 2022 combined knee MRI reconstruction with bone/cartilage
segmentation and bone shape analysis [37]. Yet again, there were only weak
correlations between the metrics of reconstruction and segmentation quality,
with one of the best segmentation models producing highly artifactual
reconstructions but high quality segmentations.

###  5.4 Joint estimation of sensitivity maps and reconstruction

The power of DL has also been harnessed for improving parallel multi-coil MRI,
where the coil sensitivity maps must be estimated and incorporated in the
image reconstruction process. Early DL methods commonly utilized the ESPIRIT
algorithm [238] for computing the sensitivity maps prior to the reconstruction
process. Nevertheless, joint estimation of the sensitivity maps and
reconstructed data could contribute to improving image quality, as indicated
in different studies, first with classical approaches[239, 240] and later
using DL [241, 242, 243]. DL frameworks were hence recently developed for
joint estimation of the sensitivity maps and reconstruction data. For example,
the well-known E2E-VarNet method [241] included a module for sensitivity maps
estimation and incorporated it into a larger unrolled network, trained end-to-
end. A similar approach was taken by Jun et al., who proposed the IC-Net
[242]. Luo et al. suggested using a deep image prior [243], and Istanet et al.
proposed a zero-shot learning method which is trained solely on data from a
specific subject and jointly estimates the sensitivity maps and temporal data
[244].

###  5.5 Other applications

The powerful capabilities of DL have also been exploited for other
computational tasks in the MRI workflow aside from image reconstruction. A
detailed review of these applications is beyond the scope of this manuscript,
which focuses on MRI reconstruction. Specific examples include the joint
recovery of multi-contrast MRI data [245, 101], the synthesis of missing
contrasts or synthesis of quantitative maps based on anatomical data [245,
246], super-resolution [247, 248], B0 estimation and off-resonance correction
[249]), enhancement of low-field MRI data, where the low SNR degrades image
quality [250, 251], and automated scan prescription [252].

##  6 Datasets and software

###  6.1 Datasets

The availability of public datasets and open-source code repositories has
played a crucial role in the rapid development of DL techniques [253]. In the
MRI reconstruction field, several major databases such as fastMRI [30], SKM-
TEA [33], mridata.org [31], and Calgary-Campinas [32] catalyzed development by
making available large amounts of raw _k-space_ data, which are useful for
developing and benchmarking methods [254, 44]. Other resources provide
valuable data for specific applications, including MR imaging of speech
production [255], cardiovascular imaging [256], and low-field MRI [257]. Many
other MRI datasets are also available on the web, but those were commonly
designed for downstream, non-reconstruction tasks, hence they do not always
contain raw _k-space_ data. Examples include the Human Connectome project
[258], IXI [259], BRaTS [260, 261], ADNI [262], UK-biobank [263] and OASIS
[264]

###  6.2 Open-source software

The adoption of open-source frameworks has significantly accelerated the
development of DL methods as they proide researchers robust, flexible
platforms to develop new algorithms. The two most prominent general-purpose DL
frameworks are PyTorch [265] and TensorFlow [266], which offer extensive
libraries that facilitate the design, training, and deployment of DL models.

Several open-source software frameworks have been developed specifically for
MRI. These offer useful computational tools for handling raw _k-space_ data,
implementation of algorithms and computation of MRI-related metrics. For
example, BART (Berkeley Advanced Reconstruction Toolbox) [267] is a large and
highly popular software package. It enables efficient data processing and
contains implementations of different iterative reconstruction algorithms. The
recent versions of BART also contain general-purpose tools that are highly
useful for the development of DL reconstruction models, e.g., an automated
differentiation framework compatible with complex-valued data, and
implementations of well-established DL models [268]. Gadgetron [269] is
another popular package, which offers extensive tools for image
reconstruction, data management, and implementations of iterative solvers.
Another example is Sigpy [270], which offers a set of operators, blocks and
algorithms that are highly suitable for iterative reconstruction. Unlike other
toolboxes, Sigpy is written entirely in Python and can hence be integrated
easily into frameworks such as PyTorch. The SNOPY (non-Cartesian sampling
trajectory) [165] framework has practical tools for optimizing _k-space_
sampling trajectories, including a differentiable MRI system model, and loss
functions corresponding to constraints on image quality, hardware (e.g.,
maximum slew rate and gradient strength), and peripheral nerve stimulation
(PNS). A different framework is Yarra (https://cai2r.net/resources/yarra/),
which provides tools for automated collection of raw _k-space_ data. These can
facilitate acquisition of datasets and the creation of new datasets.

Another key area is pulse sequence development. One of the main challenges in
reproducing complex pulse sequences across different sites and scanners is the
dedicated prototyping environment and software used by each vendor. Pulseq
[271] is a rapid, hardware-independent pulse sequence prototyping framework,
which enables intuitive high-level programming of acquisition protocols in
Matlab or Python. It enables easy deployment across different field strengths
and hardware. Importantly, the spin physics associated with the specific
acquisition protocol compiled at the scanner can be accurately simulated as
part of the Pulseq framework or its derivatives [272]. Techniques for data
harmonization can help mitigate challenges in transferring protocols across
different systems [273].

In the context of quantitative imaging, the qMRLab software was developed to
facilitate reproducibility across MRI systems [274]. It consists of practical
tools for analyzing and processing quantitative MRI data acquired by different
vendors. The user-friendly interface and modular design of qMRLab enable
researchers to easily implement and share quantitative MRI techniques.

Many other open-source codes can also be found on online platforms such as
GitHub, articles with code (https://paperswithcode.com/), and the two
dedicated websites of the ISMRM: MR-Hub (https://ismrm.github.io/mrhub/),
which hosts toolboxes, and MR-Pub (https://ismrm.github.io/mrpub/), which
hosts git repositories published together with articles.

The toolboxes and platforms described above are essential for enhancing
reproducible research in the MRI community. The development of a unified data
format, the ISMRMRD [275] can also facilitate easy translation of datasets and
methods across sites and research groups. The recent development of techniques
for federated learning [276, 277] are also useful for training algorithms
collaboratively without sharing the data; this can help address data-privacy
issues.

##  7 Robustness challenges

In this section we discuss challenges related to developing, evaluating, and
benchmarking DL reconstruction methods, and suggest approaches for mitigating
these challenges.

###  7.1 Distribution shifts

In deep learning, _generalization_ refers to the ability of a trained model to
accurately reconstruct images that it has never seen before, particularly when
these new images differ substantially from the training data. Good
generalization of DL-based MRI reconstruction models is critical for clinical
workflows. However, achieving a good generalization is challenging because MRI
data can vary substantially in terms of different factors, e.g. MRI hardware,
vendor-specific scanning protocols, patient populations, and the anatomical
regions being imaged. This variability can lead to a model that performs well
on data from one source but poorly on data from another, a phenomenon known as
_domain shift_ or _distribution shift_. Here we review some of the main
challenges related to this issue.

Domain shifts have been studied from several perspectives. [278] analyzed the
models submitted to the 2019 fastMRI challenge and found that many of them
were sensitive to distribution shifts. Darestani et al. [279] evaluated the
robustness of DL reconstruction methods with regard to out-of-distribution
data, and found that both trained and untrained networks were affected by
distribution shifts. Avidan et al. studied another type of distribution shift
[280], related to sampling; methods trained on specific sampling schemes may
not generalize well to other schemes. Altogether, distribution shifts can lead
to substantial performance drops in MRI and can hence be a major limiting
factor in practice.

_Potential mitigation strategies._ In cases where only a few training examples
from a target domain are available, pre-training a network on other data and
fine-tuning it to the target domain can improve performance [281, 282, 283].
In the challenging case where no target data are available for fine-tuning,
test-time-training, which involves adapting to a single training at inference,
is a viable performance-enhancing alternative [284]. Another good strategy
that can help mitigate the performance drop due to distribution shifts is to
train on broad and diverse data [284].

A growing body of work has pointed out the advantages of diffusion models in
robustifying networks to distribution shifts. As described above (section
3.3.2), diffusion models decouple the image prior from the statistical
measurement model. They can hence generalize easily to various anatomies and
sampling patterns [85, 91, 87, 89, 86, 97]. Another technique to robustify
networks to shifts in sampling patterns is to provide the network with the
undersampling mask, and train the network to generalize to various sampling
masks [280]. In addition, generative networks can be trained solely on
magnitude images and applied to complex-valued data with different sub-
sampling patterns and out-of-distribution data [91].

###  7.2 Bias and ”data crimes”

In the field of AI, the term _bias_ is often associated with gender-related or
population-related bias. This can occur when models are trained on datasets
that do not contain equal distributions of subjects having different genders,
different ethnicities, or even data that only contain a narrow set of medical
conditions [285]. This training can lead to algorithmic failure for under-
served populations [286, 287] or rare conditions [285].

However, when solving inverse problems, bias can also arise from a naive,
seemingly-appropriate use of open-access datasets. One of the primary
challenges in the development of DL reconstruction methods is the need for raw
_k-space_ data, which are scarce and difficult to acquire due to the high cost
of MRI scans and the long scan duration. While several databases offer such
data, e.g. [30, 31, 32, 33, 255, 237], there are many other databases that
offer non-raw MRI data. Those are generally designed for downstream tasks,
e.g., segmentation and classification, hence they are frequently preprocessed.
Nevertheless, due to their high availability, researchers sometimes download
and use them for synthesizing _k-space_ data training DL reconstruction
models. This has been referred to as ”off-label” data use, because those
datasets are used for a different task than the one they were were designed
for [288].

Surprisingly, training DL reconstruction algorithms using ”off-label” data
could give rise to good-looking results; nevertheless, those are often biased
and _overly optimistic_ , i.e. they are ”too good to be true” [288]. This is
due to subtle preprocessing steps which, although imperceptible to the naked
eye, impact algorithm performance. Common preprocessing steps include _k-
space_ zero-padding, coil combination, and JPEG data compression. The authors
of [288] demonstrated that CS, dictionary learning, and DL algorithms are all
sensitive to these preprocessing steps and yield biased results. This
underscores their potential risk, as these algorithms are aimed for clinical
purposes. The findings also demonstrated that DL algorithms trained on
preprocessed data may not be able to generalize well to real-world clinical
data, and could potentially eliminate crucial clinical details [288].

Another concerning finding is that popular error metrics, e.g., the normalized
root mean square error (NRMSE) and SSIM could be _blind_ to the preprocessing
[288], and miss the bias. This is because those metrics compare the
reconstructed and reference image, but those come from the same underlying
data, i.e. both are preprocessed. The error metrics thus cannot measure the
true image quality. Therefore, DL algorithms may achieve ”good” NRMSE and SSIM
scores even when their performance is poor. This makes the head-to-head
comparison of results across papers very difficult, because some papers report
experiments with raw _k-space_ data while others report results for
preprocessed data, where the metrics tend to be better. To raise awareness of
this phenomenon, the authors dubbed the publication of misleading results
”data crimes”.

_Potential mitigation strategies._ In the context of gender-related or
population-related biases, the best strategy is to train on large, diverse
datasets [286, 287, 285]. In the context of bias that stems from training on
preprocessed data, the optimal strategy is to train solely on raw _k-space_
data. When raw, zero-padded _k-space_ data are available, and such data have
not been subject to any other preprocessing steps, one simple correction step
is to crop _k-space_ to its original size. However, this scenario is quite
rare. In practice, other preprocessing steps are often applied, e.g. coil-
combination (e.g. using root-sum-of-squares operation) and JPEG compression.
Those steps are irreversible, hence there is no simple technique to remedy
them, and the bias cannot be easily prevented.

_Techniques for data synthesis_. When raw _k-space_ data are unavailable, one
possible strategy is to train on synthetic data [289, 290, 291]. When
magnitude data are available, synthetic complex-valued data can be obtained,
for example, by adding a synthetic phase to the magnitude images and training
a generative model to learn priors of complex-valued images [291]. Multi-coil
data can be synthesized by multiplying the phase-enhanced magnitude data with
sensitivity maps [289]. Another approach is to leverage the Bloch equations to
simulate realistic data [290, 292]. However, it should be emphasized that the
synthesis of complex-valued data does not guarantee good performance for real-
world data; i.e., it cannot automatically prevent the bias described above.
Models trained on synthetic data must therefore be tested using real-world,
raw _k-space_ data.

###  7.3 Hallucinations

The term _hallucinations_ refers to the generation of false, realistic-looking
features which are not present in the actual data. This can arise from the use
of inaccurate priors [293], e.g., when there is a distribution shift between
the training and test data, as described above. Strikingly, the team that
organized the second FastMRI challenge found that many of the top-performing
models produced hallucinations, and that these hallucinations were not
captured by image quality metrics such as SSIM [34]. They also noticed that
hallucinations could morph abnormal structures into seemingly normal ones.

Several studies have also highlighted and explored manifestations of
hallucinations. For example, Cohen et al. (2018) [294] discussed how
distribution matching losses in medical image translation can lead to
hallucinated features. Bhadra et al. [293] reported hallucinations in the
context of tomographic image reconstruction and introduced the concept of
hallucination maps to identify and understand the impact of prior information
in regularized reconstruction methods. Gottschling et al. [295] explored
hallucinations from a theoretical perspective, and highlighted problematic
scenarios of _in-distribution_ hallucinations. The issue of hallucinations is
a critical concern in DL-based image reconstruction, as it can lead to
misleading results and false diagnoses.

_Potential mitigation strategies._ At present, there is a pressing need for
techniques to mitigate hallucinations. Currently, the best strategy is to have
radiology experts evaluate the reconstructed images, in the hope that they
will be able to detect hallucinations. However, more research and development
are required.

###  7.4 Adversarial attacks and instabilities

Neural networks for image classification are known to be sensitive to small,
inperceptible, adversarially chosen perturbations. This has spurred concerns
that DL-based reconstructions could be sensitive to worst-case perturbations.
A number of studies [294, 296, 295, 297] have raised concerns regarding this
issue. For example, Antun et al. [297] provided simulations demonstrating that
small, adversarially selected perturbations in undersampled measurements can
result in severe reconstruction artifacts. They also showed that there is a
trade-off between robustness and performance, and that classical sparsity-
based reconstruction methods are also sensitive to adversarially selected
perturbations.

Robustness to adversarial attacks is still an open issue, which requires
further investigation. For example, at present there is no conclusive evidence
indicating that DL reconstruction methods are more sensitive than classical
methods such as sparsity-based reconstruction to worst-case perturbations.
[298] provided theoretical results on a worst-case optimal estimator. [299]
found that simple end-to-end variational networks are as sensitive to
perturbations as U-nets, and [279] showed that both un-trained and trained
networks are sensitive to such perturbations.

_Strategies for enhancing robustness._ Goujon et al. [300] demonstrated that
the robustness of reconstruction algorithms can be improved by constraining
the CNN module to be convex or using a monotone constraint [64]. However, a
global convexity or monotone constraint often translates into reduced
performance [300, 64], as predicted by [297]. A recent work showed that the
global monotone constraint described in [64] can be replaced by a local
constraint around the image manifold [301], to achieve improved robustness
without compromising on performance. Following the approach implemented in
non-convex algorithms, this algorithm is theoretically guaranteed to converge
to a minimum, provided it is initialized with SENSE reconstructions. From a
different perspective, [302] and [303] suggested the use of adversarial
attacks during training to reduce false negatives.

###  7.5 Benchmarking challenges

When evaluating various techniques comparing the performance of different
reconstruction and image analysis methods, it is crucial to have appropriate
benchmarks that can characterize true clinical utility. Conventionally, image
quality metrics such as the mean square error (MSE), peak signal-to-noise
ratio (PSNR), and the structural similarity metric (SSIM) [304] are used to
assess the image quality of reconstructed MR images. These metrics have been
described extensively in the computer vision literature and have a moderate
correspondence with human-perceived image quality [304]. However, as there is
a substantial domain shift between natural images and medical images,studies
have shown that such traditional image quality metrics do not correlate well
with radiologist-perceived metrics [305]. One likely reason for this
phenomenon is that not all pixels in a given image have similar diagnostic
value. Consider, for example, knee MRI scans, e.g. those contained in the
popular fastMRI database. Most abnormalities in knee scans are likely to be
located in small, subtle regions in the cartilage, meniscus, and ligaments,
while the tissues that are visible in the scans, such as bone and muscle, are
likely to have substantially fewer abnormalities. Computing traditional image
quality metrics that weigh all image pixels similarly may therefore not
reflect how a radiologist perceives the images. Thus, it is essential to
define metrics that correspond to downstream clinical utility so that the same
metrics can be used for benchmarking models.

Another issue of concern is that the performance of DL reconstruction
algorithms is typically evaluated using a relatively narrow test set, which is
similar in nature to the training data. The evaluation results can thus be
misleading, since they do not yield a reliable estimation of the model’s
generalization ability; i.e., the ability to perform well on test data that
deviate from the training data, which is of critical importance in clinical
settings.

Figure 4: Impact of common image perturbations on image quality metrics. A
variety of image perturbations applied to a sample image from the fastMRI
dataset (top row: noise addition, image blurring, pixel rolling (where an
image is shifted by a number of pixels), and physics-based subject motion. The
impact of these corruptions is shown for conventional image quality metrics
(SSIM, PSNR) and deep feature distance metrics (LPIPS - made for natural
images, SSFD - made for MR images). The deep feature metrics exhibit a larger
dynamic range to the noise, blurring, and motion corruptions, but present very
little change due to pixel rolling, since the image quality does not change.
These qualities of deep feature metrics are ideal for assessing MRI
reconstruction quality.

Yet another challenge is related to the common train/test data split. Even for
a single dataset, different research groups or studies may apply different
splits, and hence train and test the model with different data subsets. For
example, even if two identical studies used the same number of training
images, it is not always possible to verify whether they were the same images.
Previous work has shown that there can be a significant variation in how well
the same deep learning models perform on different imaging exams within the
same larger dataset [279]. Consequently, even if the number of training
examples is maintained across studies, some examples may be easier or more
challenging to train and evaluate. This makes a head-to-head comparison of
published literature difficult to interpret.

_Strategies for mitigating benchmarking challenges._ Initiatives such as the
large-scale fastMRI database and challenge [30] allow multiple different
methods to be compared on identical training, validation, and testing data
splits. This type of consistent evaluation platform allows for head-to-head
comparisons across different methods and can shed light on their pros and
cons.

More recent studies dealing with image quality assessment have gone beyond
PSNR and SSIM metrics and used perceptual metrics of image quality instead of
relying on handcrafted metrics [306]. Perceptual metrics utilize
representations extracted from pre-trained neural networks and have been used
for evaluating and optimizing MR reconstruction quality [307, 308]. For
example, Learned Perceptual Image Patch Similarity (LPIPS) is commonly used
for assessing the quality of natural images. A similar analog of Self-
Supervised Feature Distance (SSFD) has been proposed for assessing MR image
quality [309, 310]. When using these perceptual metrics, a reference image and
an evaluation image can be fed into the same pre-trained network, and the
differences in their representations compared to compute the similarity
between the two images. These perceptual metrics can either be computed using
pre-trained networks on natural images or medical images. A recent study
demonstrated that although there may not be considerable differences between
networks trained on natural images and medical images; i.e., between the
networks used for generating the representations, using perceptual metrics is
better than using traditional image quality metrics when evaluating against
radiologists’ perceived image quality [307]. The impact of common image
perturbations on such deep feature metrics is shown in Fig. 4.

Beyond the metrics of image quality, another approach for benchmarking
reconstruction models aims to directly benchmark the downstream value provided
by the MR image. For example, the SKM-TEA dataset is designed to enable the
evaluation of downstream tasks such as clinical classification, segmentation,
and articular cartilage T2 quantification [33]. This dataset emphasizes the
combination of automated cartilage segmentation with T2 quantification, a
known metric with clinical and research significance [311, 312]. Similarly,
the K2S dataset aims to optimize cartilage volume and thickness
quantification, whereas the fastMRI+ dataset assesses classification and
detection tasks [37, 233]. Such benchmarking metrics, which relate to clinical
significance, provide a promising path forward for assessing image quality
when used in conjunction with traditional or perceptual image quality metrics.
Doing so can contribute to optimizing both image quality and image value for
radiologists.

##  8 Uncertainty Estimation

Connecting both upstream and downstream DL tools with clinical practice first
requires promoting confidence among the users of these tools. Robustly
characterizing the uncertainty of DL models may help increase the trust of
downstream users in the outputs of DL algorithms prior to their integration
into routine clinical practice. Uncertainty quantification, a popular subfield
of DL research, can encourage building such trust among downstream users,
regarding both image reconstruction and automated image analysis algorithms.

Several works have proposed methods to evaluate the uncertainty of accelerated
MRI reconstructions [313, 314, 315, 316, 90, 317]. Given that MRI construction
is an ill-posed problem where several high-quality images can correspond to
the same low-quality image, quantifying the uncertainty of a sample
reconstruction can help guide the fidelity of the overall reconstruction
process. The developed uncertainty approaches leverage the variational
formulation of inverse image recovery for sampling not only a single output of
reconstruction, but a variety of different outputs using a learned model of
the posterior distribution. Sampling a multitude of image outputs makes it
possible to evaluate the variance at the voxel-level to compute uncertainties
within a given image. This technique of uncertainty quantification can be used
to determine whether specific regions of interest have high uncertainty values
around abnormal image findings.

For downstream image analysis tasks, techniques like Monte Carlo Dropout have
gained popularity for estimating output uncertainty [318]. Although dropout is
typically employed for model regularization and reducing overfitting during
training by randomly setting parameters to zero, it can also be applied at
inference time. This involves producing multiple outputs with different neural
network weights set to zero. This Monte Carlo Dropout approximates a Gaussian
process, which facilitates the computation of uncertainty across the variance
of all the generated inputs. Although straightforward to implement, this
approach requires multiple forward passes during inference.

Looking forward, despite our ability to generate voxel-level uncertainties,
the optimal utilization of these generated uncertainty maps still remains
unclear. This elicits a number of intriguing research questions concerning the
ways in which these uncertainties can be utilized beyond simply presenting
them to end-users. For instance, it is unlikely that a radiologist would
review both the output of a deep learning reconstruction and the corresponding
uncertainty map since doing so would nearly double clinical read durations,
and the radiologist may not necessarily know how to contextualize regions of
high and low uncertainty.

Alternatively, these uncertainty maps could be applied in iterative
reconstruction techniques [319]. In this approach, the uncertainty associated
with a specific step in an image reconstruction pipeline could serve as input
for the subsequent step. The reconstruction network’s objective would be to
reconstruct the same image but with the new constraint of reducing the
underlying uncertainty. This type of formulation holds potential for adaptive,
case-based sampling schemes tailored to individual patients.

Another promising area of research involves directly integrating uncertainties
from MR image reconstruction with those of the relevant downstream parameters
of interest [320]. This methodology could serve to estimate maximum
acceleration rates without compromising quantitative clinical parameters.
Achieving such end-to-end analysis necessitates datasets that can provide raw
case-based data alongside downstream image analysis datasets (e.g. SKM-TEA,
fastMRI+, K2S, etc).

##  9 Implementation issues

Training DL reconstruction models can be computationally challenging,
especially for large networks. For purposes of illustration, we focus on the
case of unrolled networks.

_Memory demands of unrolled algorithms._ PnP and score-based algorithms that
pre-train deep learning modules as denoisers are associated with low memory
demand. In contrast, unrolled algorithms, which offer state-of-the art
performance compared to PnP methods, involve a number of iterations and their
training is restricted by the memory of the GPU devices during training. This
often limits the applicability of unrolled algorithms to large-scale multi-
dimensional problems.

_Strategies for computational efficiency._ Several strategies have been
introduced to overcome the memory limitations of unrolled methods. For an
unrolled network with $N$ iterations and shared CNN modules across iterations,
the computational complexity and memory demands of backpropagation are
$\mathcal{O}(N)$ and $\mathcal{O}(N)$, respectively. The forward steps can be
recomputed during backpropagation, which reduces the memory demand to
$\mathcal{O}(1)$, while the computational complexity increases to
$\mathcal{O}(N^{2})$. Forward checkpointing [321] saves the variables for
every $K$ layers during forward propagation, which reduces the computational
demand to $\mathcal{O}(NK)$, while the memory demand is $\mathcal{O}(N/K)$.
Reverse recalculation has been proposed to reduce the memory demand to
$\mathcal{O}(1)$ and computational complexity to $\mathcal{O}(N)$ [322].
However, the approach in [322] requires multiple iterations to invert each CNN
block, resulting in high computational complexity in practical applications.
The deep equilibrium (DEQ) model [323] was recently adapted to inverse
problems to significantly improve the memory demand [63, 64] of unrolled
methods. Unlike unrolled methods, DEQ schemes run the iterations until
convergence, similar to PnP algorithms. This property makes it possible to
perform forward and backward propagation using fixed-point iteration involving
a single physical layer, which reduces the memory demand to $\mathcal{O}(1)$,
while the computational complexity is $\mathcal{O}(N)$; this offers better
tradeoffs than the alternatives discussed above [321, 322]. The runtime of DEQ
methods that are iterated until convergence are variable compared to unrolled
methods, which use a finite number of iterations. In addition, the convergence
of the iterative algorithm is crucial for the accuracy of backpropagation
steps in DEQ, unlike in unrolled methods. Convergence guarantees were
introduced in [63, 64].

##  10 Conclusion

The introduction and rapid development of deep-learning-based strategies for
MRI reconstruction have brought about a dramatic acceleration in acquisition
time, paved the way for rapid and accurate parameter mapping, and facilitated
automatic schedule optimization. While several key challenges still lie ahead,
especially in terms of robust generalization, careful considerations of the
training data diversity, ongoing model validation, and potentially, the
development of adaptive or continuous learning systems are expected to enable
adjustment to new data distributions over time.

## Acknowledgments

The authors thank Jonathan Tamir for useful comments regarding diffusion
models. M.J acknowledges support from NIH grants R01 AG067078, R01EB019961,
R01 EB031169, and Canon Medical Research. O.P. acknowledges funding support
from the Ministry of Innovation, Science and Technology, Israel, and the Tel
Aviv University Center for AI and Data Science (TAD). E.S. is a Horev Fellow
and acknowledges funding support from the Technion’s Leaders in Science and
Technology program. A.C. receives support from NIH grants R01 AR077604, R01
EB002524, and R01 AR079431; and from GE Healthcare, Philips, Amazon,
Microsoft, and Stability AI.

## References

  * [1] G. Constantine, K. Shan, S. D. Flamm, and M. U. Sivananthan, “Role of MRI in clinical cardiology,” _The Lancet_ , vol. 363, no. 9427, pp. 2162–2171, 2004.
  * [2] M. Zaitsev, J. Maclaren, and M. Herbst, “Motion artifacts in MRI: A complex problem with many partial solutions,” _Journal of Magnetic Resonance Imaging_ , vol. 42, pp. 887–901, 10 2015.
  * [3] D. K. Sodickson and W. J. Manning, “Simultaneous acquisition of spatial harmonics (SMASH): Fast imaging with radiofrequency coil arrays,” _Magnetic Resonance in Medicine_ , vol. 38, 1997.
  * [4] M. A. Griswold _et al._ , “Generalized autocalibrating partially parallel acquisitions (GRAPPA),” _Magnetic Resonance in Medicine_ , vol. 47, no. 6, pp. 1202–1210, 2002.
  * [5] K. P. Pruessmann, M. Weiger, M. B. Scheidegger, and P. Boesiger, “SENSE: Sensitivity encoding for fast MRI,” _Magnetic Resonance in Medicine_ , vol. 42, no. 5, pp. 952–962, 1999.
  * [6] N. Seiberlich, F. A. Breuer, M. Blaimer, K. Barkauskas, P. M. Jakob, and M. A. Griswold, “Non-Cartesian data reconstruction using GRAPPA operator gridding (GROG),” _Magnetic Resonance in Medicine_ , vol. 58, pp. 1257–1265, 12 2007.
  * [7] A. Deshmane, V. Gulani, M. Griswold, and N. Seiberlich, “Parallel MR imaging,” _Journal of Magnetic Resonance Imaging_ , vol. 36, no. 1, pp. 55–72, 2012.
  * [8] M. Lustig and J. M. Pauly, “SPIRiT: iterative self-consistent parallel imaging reconstruction from arbitrary k-space,” _Magn. Reson. Med._ , vol. 64, no. 2, pp. 457–471, 2010.
  * [9] M. Lustig, D. Donoho, and J. M. Pauly, “Sparse MRI: The application of compressed sensing for rapid MR imaging,” _Magnetic Resonance in Medicine_ , vol. 58, no. 6, pp. 1182–1195, 2007.
  * [10] D. L. Donoho, “Compressed Sensing,” _IEEE Transactions on information theory_ , vol. 52, no. 4, pp. 1289–1306, 2006.
  * [11] H. Jung, K. Sung, K. S. Nayak, E. Y. Kim, and J. C. Ye, “k-t FOCUSS: a general compressed sensing framework for high resolution dynamic MRI,” _Magnetic Resonance in Medicine_ , vol. 61, no. 1, pp. 103–116, 2009.
  * [12] L. Feng, L. Axel, H. Chandarana, K. T. Block, D. K. Sodickson, and R. Otazo, “XD-GRASP: golden-angle radial MRI with reconstruction of extra motion-state dimensions using compressed sensing,” _Magnetic Resonance in Medicine_ , vol. 75, no. 2, pp. 775–788, 2016.
  * [13] M. F. Duarte and Y. C. Eldar, “Structured compressed sensing: From theory to applications,” _IEEE Transactions on Signal Processing_ , vol. 59, pp. 4053–4085, 9 2011.
  * [14] J. C. Ye, “Compressed sensing MRI: a review from signal processing perspective,” _BMC Biomedical Engineering_ , vol. 1, no. 1, pp. 1–17, 2019.
  * [15] S. G. Lingala and M. Jacob, “Blind compressive sensing dynamic MRI,” _IEEE transactions on medical imaging_ , vol. 32, no. 6, pp. 1132–1145, 2013.
  * [16] J. A. Fessler, “Optimization methods for magnetic resonance image reconstruction: Key models and optimization algorithms,” _IEEE signal processing magazine_ , vol. 37, no. 1, pp. 33–40, 2020.
  * [17] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [18] L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaría, M. A. Fadhel, M. Al-Amidie, and L. Farhan, “Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions,” _Journal of big Data_ , vol. 8, pp. 1–74, 2021.
  * [19] A. Shrestha and A. Mahmood, “Review of deep learning algorithms and architectures,” _IEEE access_ , vol. 7, pp. 53 040–53 065, 2019.
  * [20] S. S. Chandra, M. Bran Lorenzana, X. Liu, S. Liu, S. Bollmann, and S. Crozier, “Deep learning in magnetic resonance image reconstruction,” _Journal of Medical Imaging and Radiation Oncology_ , vol. 65, no. 5, pp. 564–577, 2021.
  * [21] B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, and M. S. Rosen, “Image reconstruction by domain-transform manifold learning,” _Nature_ , vol. 555, no. 7697, pp. 487–492, 2018.
  * [22] S. Wang, H. Cheng, L. Ying, T. Xiao, Z. Ke, H. Zheng, and D. Liang, “DeepcomplexMRI: Exploiting deep residual network for fast parallel MR imaging with complex convolution,” _Magnetic Resonance Imaging_ , vol. 68, pp. 136–147, 5 2020.
  * [23] K. Hammernik _et al._ , “Learning a variational network for reconstruction of accelerated MRI data,” _Magnetic Resonance in Medicine_ , vol. 79, no. 6, pp. 3055–3071, 2018.
  * [24] H. K. Aggarwal, M. P. Mani, and M. Jacob, “MoDL: Model-based deep learning architecture for inverse problems,” _IEEE transactions on medical imaging_ , vol. 38, no. 2, pp. 394–405, 2018.
  * [25] A. S. Lundervold and A. Lundervold, “An overview of deep learning in medical imaging focusing on MRI,” _Zeitschrift für Medizinische Physik_ , vol. 29, no. 2, pp. 102–127, 2019.
  * [26] M. A. Mazurowski, M. Buda, A. Saha, and M. R. Bashir, “Deep learning in radiology: An overview of the concepts and a survey of the state of the art with focus on MRI,” _Journal of magnetic resonance imaging_ , vol. 49, no. 4, pp. 939–954, 2019.
  * [27] S. Ravishankar, J. C. Ye, and J. A. Fessler, “Image reconstruction: From sparsity to data-adaptive methods and machine learning,” _Proceedings of the IEEE_ , vol. 108, no. 1, pp. 86–109, 2019.
  * [28] F. Knoll, K. Hammernik, C. Zhang, S. Moeller, T. Pock, D. K. Sodickson, and M. Akcakaya, “Deep-learning methods for parallel magnetic resonance imaging reconstruction: A survey of the current approaches, trends, and issues,” _IEEE signal processing magazine_ , vol. 37, no. 1, pp. 128–140, 2020.
  * [29] D. Liang, J. Cheng, Z. Ke, and L. Ying, “Deep magnetic resonance image reconstruction: Inverse problems meet neural networks,” _IEEE Signal Processing Magazine_ , vol. 37, no. 1, pp. 141–151, 2020.
  * [30] F. Knoll _et al._ , “fastMRI: A publicly available raw k-space and DICOM dataset of knee images for accelerated MR image reconstruction using machine learning,” _Radiology: Artificial Intelligence_ , vol. 2, no. 1, 2020.
  * [31] F. Ong, S. Amin, S. Vasanawala, and M. Lustig, “Mridata.org: An open archive for sharing MRI raw data,” in _Proc. Intl. Soc. Mag. Reson. Med_ , vol. 26, 2018, p. 1.
  * [32] R. Souza, O. Lucena, J. Garrafa, D. Gobbi, M. Saluzzi, S. Appenzeller, L. Rittner, R. Frayne, and R. Lotufo, “An open, multi-vendor, multi-field-strength brain MR dataset and analysis of publicly available skull stripping methods agreement,” _NeuroImage_ , vol. 170, pp. 482–494, 2018.
  * [33] A. D. Desai _et al._ , “SKM-TEA: A Dataset for Accelerated MRI Reconstruction with Dense Image Labels for Quantitative Clinical Evaluation,” in _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2021.
  * [34] M. J. Muckley, B. Riemenschneider, A. Radmanesh, S. Kim, G. Jeong, J. Ko, Y. Jun, H. Shin, D. Hwang, M. Mostapha _et al._ , “Results of the 2020 fastMRI challenge for machine learning MR image reconstruction,” _IEEE transactions on medical imaging_ , vol. 40, no. 9, pp. 2306–2317, 2021.
  * [35] F. Knoll, T. Murrell, A. Sriram, N. Yakubova, J. Zbontar, M. Rabbat, A. Defazio, M. J. Muckley, D. K. Sodickson, C. L. Zitnick _et al._ , “Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge,” _Magnetic resonance in medicine_ , vol. 84, no. 6, pp. 3054–3070, 2020.
  * [36] M. J. Muckley, B. Riemenschneider, A. Radmanesh, S. Kim, G. Jeong, J. Ko, Y. Jun, H. Shin, D. Hwang, M. Mostapha _et al._ , “State-of-the-art machine learning MRI reconstruction in 2020: Results of the second fastMRI challenge,” _arXiv preprint arXiv:2012.06318_ , vol. 2, no. 6, p. 7, 2020.
  * [37] A. A. Tolpadi, U. Bharadwaj, K. T. Gao, R. Bhattacharjee, F. G. Gassert, J. Luitjens, P. Giesler, J. N. Morshuis, P. Fischer, M. Hein _et al._ , “K2S Challenge: From Undersampled K-Space to Automatic Segmentation,” _Bioengineering_ , vol. 10, no. 2, p. 267, 2023.
  * [38] Pal, Arghya and Rathi, Yogesh, “A review and experimental evaluation of deep learning methods for MRI reconstruction,” _The journal of machine learning for biomedical imaging_ , vol. 1, 2022.
  * [39] D. J. Lin, P. M. Johnson, F. Knoll, and Y. W. Lui, “Artificial intelligence for MR image reconstruction: an overview for clinicians,” _Journal of Magnetic Resonance Imaging_ , vol. 53, no. 4, pp. 1015–1028, 2021.
  * [40] J. A. Oscanoa, M. J. Middione, C. Alkan, M. Yurt, M. Loecher, S. S. Vasanawala, and D. B. Ennis, “Deep Learning-Based Reconstruction for Cardiac MRI: A Review,” _Bioengineering 2023, Vol. 10, Page 334_ , vol. 10, p. 334, 3 2023.
  * [41] V. Spieker, H. Eichhorn, K. Hammernik, D. Rueckert, C. Preibisch, D. C. Karampinos, and J. A. Schnabel, “Deep Learning for Retrospective Motion Correction in MRI: A Comprehensive Review,” _IEEE Transactions on Medical Imaging_ , 5 2023.
  * [42] K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson, T. Pock, and F. Knoll, “Learning a variational network for reconstruction of accelerated MRI data,” _Magnetic resonance in medicine_ , vol. 79, no. 6, pp. 3055–3071, 2018.
  * [43] B. Bilgic, V. K. Goyal, and E. Adalsteinsson, “Multi-contrast reconstruction with Bayesian compressed sensing,” _Magnetic resonance in medicine_ , vol. 66, no. 6, pp. 1601–1615, 2011.
  * [44] K. Hammernik, J. Schlemper, C. Qin, J. Duan, R. M. Summers, and D. Rueckert, “Systematic evaluation of iterative deep neural networks for fast parallel MRI reconstruction with sensitivity-weighted coil combination,” _Magnetic Resonance in Medicine_ , vol. 86, no. 4, pp. 1859–1872, 2021.
  * [45] G. Luo, N. Zhao, W. Jiang, E. S. Hui, and P. Cao, “MRI reconstruction using deep Bayesian estimation,” _Magnetic resonance in medicine_ , vol. 84, no. 4, pp. 2246–2261, 2020.
  * [46] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein _et al._ , “Distributed optimization and statistical learning via the alternating direction method of multipliers,” _Foundations and Trends® in Machine learning_ , vol. 3, no. 1, pp. 1–122, 2011.
  * [47] A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” _SIAM journal on imaging sciences_ , vol. 2, no. 1, pp. 183–202, 2009.
  * [48] M. J. Muckley, B. Riemenschneider, A. Radmanesh, S. Kim, G. Jeong, J. Ko, Y. Jun, H. Shin, D. Hwang, M. Mostapha _et al._ , “Results of the 2020 fastMRI challenge for machine learning MR image reconstruction,” _IEEE transactions on medical imaging_ , vol. 40, pp. 2306–2317, 2021.
  * [49] S. Wang _et al._ , “Accelerating magnetic resonance imaging via deep learning,” 2016, pp. 514–517.
  * [50] K. H. Jin, M. T. McCann, E. Froustey, and M. Unser, “Deep convolutional neural network for inverse problems in imaging,” _IEEE transactions on image processing_ , vol. 26, no. 9, pp. 4509–4522, 2017.
  * [51] K. Lin and R. Heckel, “Vision Transformers Enable Fast and Robust Accelerated MRI,” pp. 774–795, 12 2022. [Online]. Available: https://proceedings.mlr.press/v172/lin22a.html
  * [52] ——, “Vision transformers enable fast and robust accelerated MRI,” in _International Conference on Medical Imaging with Deep Learning_. PMLR, 2022, pp. 774–795.
  * [53] P. Guo, Y. Mei, J. Zhou, S. Jiang, and V. M. Patel, “ReconFormer: Accelerated MRI reconstruction using recurrent transformer,” _IEEE transactions on medical imaging_ , 2023.
  * [54] K. Gregor and Y. LeCun, “Learning Fast Approximations of Sparse Coding,” in _Proceedings of the 27th International Conference on International Conference on Machine Learning_ , ser. ICML’10. USA: Omnipress, 2010, pp. 399–406.
  * [55] J. Sun, H. Li, Z. Xu _et al._ , “Deep admm-net for compressive sensing mri,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [56] J. Schlemper, J. Caballero, J. V. Hajnal, A. N. Price, and D. Rueckert, “A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction,” _IEEE Transactions on Medical Imaging_ , vol. 37, pp. 491–503, 2 2018.
  * [57] A. Sriram, J. Zbontar, T. Murrell, A. Defazio, C. L. Zitnick, N. Yakubova, F. Knoll, and P. Johnson, “End-to-End Variational Networks for Accelerated MRI Reconstruction,” _arXiv:2004.06688_ , 2020.
  * [58] Z. Fabian, B. Tinaz, and M. Soltanolkotabi, “HUMUS-Net: Hybrid Unrolled Multi-scale Network Architecture for Accelerated MRI Reconstruction,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 25 306–25 319, 2022.
  * [59] M. Z. Darestani, V. Nath, W. Li, Y. He, H. R. Roth, Z. Xu, D. Xu, R. Heckel, and C. Zhao, “IR-FRestormer: Iterative Refinement With Fourier-Based Restormer for Accelerated MRI Reconstruction,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2024, pp. 7655–7664.
  * [60] A. Pramanik, H. K. Aggarwal, and M. Jacob, “Deep generalization of structured low-rank algorithms (Deep-SLR),” _IEEE transactions on medical imaging_ , vol. 39, no. 12, pp. 4186–4197, 2020.
  * [61] T. Eo, Y. Jun, T. Kim, J. Jang, H.-J. Lee, and D. Hwang, “KIKI-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images,” _Magnetic resonance in medicine_ , vol. 80, no. 5, pp. 2188–2201, 2018.
  * [62] B. Wang, Y. Lian, X. Xiong, H. Zhou, Z. Liu, and X. Zhou, “DCT-net: Dual-domain cross-fusion transformer network for MRI reconstruction,” _Magnetic Resonance Imaging_ , 2024.
  * [63] D. Gilton, G. Ongie, and R. Willett, “Deep equilibrium architectures for inverse problems in imaging,” _IEEE Transactions on Computational Imaging_ , vol. 7, pp. 1123–1133, 2021.
  * [64] A. Pramanik, M. B. Zimmerman, and M. Jacob, “Memory-efficient model-based deep learning with convergence and robustness guarantees,” _IEEE Transactions on Computational Imaging_ , vol. 9, pp. 260–275, 2023.
  * [65] A. Pramanik and M. Jacob, “Accelerated parallel MRI using memory efficient and robust monotone operator learning (MOL),” in _2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2023, pp. 1–4.
  * [66] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” vol. 16, no. 8, pp. 2080–2095, 2007.
  * [67] R. Ahmad, C. A. Bouman, G. T. Buzzard, S. Chan, S. Liu, E. T. Reehorst, and P. Schniter, “Plug-and-Play Methods for Magnetic Resonance Imaging: Using Denoisers for Image Recovery,” vol. 37, no. 1, pp. 105–116, 2020.
  * [68] U. S. Kamilov, C. A. Bouman, G. T. Buzzard, and B. Wohlberg, “Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imaging,” _arXiv:2203.17061 [eess]_ , 2022.
  * [69] E. Ryu, J. Liu, S. Wang, X. Chen, Z. Wang, and W. Yin, “Plug-and-play methods provably converge with properly trained denoisers,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 5546–5557.
  * [70] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [71] K. C. Tezcan, C. F. Baumgartner, R. Luechinger, K. P. Pruessmann, and E. Konukoglu, “MR image reconstruction using deep density priors,” _IEEE transactions on medical imaging_ , vol. 38, no. 7, pp. 1633–1642, 2018.
  * [72] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” _arXiv preprint arXiv:1406.2661_ , 2014.
  * [73] G. Yang, S. Yu, H. Dong, G. Slabaugh, P. L. Dragotti, X. Ye, F. Liu, S. Arridge, J. Keegan, Y. Guo _et al._ , “DAGAN: Deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction,” _IEEE transactions on medical imaging_ , vol. 37, no. 6, pp. 1310–1321, 2017.
  * [74] M. Mardani _et al._ , “Deep generative adversarial neural networks for compressive sensing MRI,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 1, pp. 167–179, 2018.
  * [75] T. M. Quan, T. Nguyen-Duc, and W.-K. Jeong, “Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss,” _IEEE transactions on medical imaging_ , vol. 37, no. 6, pp. 1488–1497, 2018.
  * [76] E. K. Cole, J. M. Pauly, S. S. Vasanawala, and F. Ong, “Unsupervised MRI reconstruction with generative adversarial networks,” _arXiv preprint arXiv:2008.13065_ , 2020.
  * [77] J. Lv, J. Zhu, and G. Yang, “Which GAN? A comparative study of generative adversarial network-based fast MRI reconstruction,” _Philosophical Transactions of the Royal Society A_ , vol. 379, no. 2200, p. 20200203, 2021.
  * [78] Y. Korkmaz, S. U. Dar, M. Yurt, M. Özbey, and T. Cukur, “Unsupervised MRI reconstruction via zero-shot learned adversarial transformers,” _IEEE Transactions on Medical Imaging_ , 2022.
  * [79] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang, “Diffusion models: A comprehensive survey of methods and applications,” _ACM Computing Surveys_ , vol. 56, no. 4, pp. 1–39, 2023.
  * [80] R. Po, W. Yifan, V. Golyanik, K. Aberman, J. T. Barron, A. H. Bermano, E. R. Chan, T. Dekel, A. Holynski, A. Kanazawa _et al._ , “State of the art on diffusion models for visual computing,” _arXiv preprint arXiv:2310.07204_ , 2023.
  * [81] F.-A. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vision: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [82] B. Kawar, M. Elad, S. Ermon, and J. Song, “Denoising Diffusion Restoration Models,” in _Neural Information Processing Systems (NeurIPS)_ , 2022.
  * [83] A. Bora, A. Jalal, E. Price, and A. G. Dimakis, “Compressed Sensing Using Generative Models,” in _International Conference on Machine Learning_ , 2017.
  * [84] M. Asim, M. Daniels, O. Leong, A. Ahmed, and P. Hand, “Invertible Generative Models for Inverse Problems: Mitigating Representation Error and Dataset Bias,” in _International Confernce on Machine Learning_ , 2020.
  * [85] A. Jalal, M. Arvinte, G. Daras, E. Price, A. G. Dimakis, and J. I. Tamir, “Robust Compressed Sensing MRI with Deep Generative Priors,” 2021.
  * [86] G. Daras, W.-S. Chu, A. Kumar, D. Lagun, and A. G. Dimakis, “Solving Inverse Problems with NerfGANs,” _arXiv:2112.09061_ , 2021.
  * [87] A. Güngör, S. U. Dar, Ş. Öztürk, Y. Korkmaz, H. A. Bedel, G. Elmas, M. Ozbey, and T. Çukur, “Adaptive diffusion priors for accelerated MRI reconstruction,” _Medical Image Analysis_ , p. 102872, 2023.
  * [88] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in neural information processing systems_ , vol. 33, pp. 6840–6851, 2020.
  * [89] H. Chung and J. C. Ye, “Score-based diffusion models for accelerated MRI,” _Medical Image Analysis_ , p. 102479, 2022.
  * [90] G. Luo, M. Blumenthal, M. Heide, and M. Uecker, “Bayesian MRI reconstruction with joint uncertainty estimation using diffusion models,” _Magnetic Resonance in Medicine_ , vol. 90, no. 1, pp. 295–311, 2023.
  * [91] M. Zach, F. Knoll, and T. Pock, “Stable deep MRI reconstruction using Generative Priors,” _IEEE Transactions on Medical Imaging_ , 2023.
  * [92] M. Mardani _et al._ , “Deep generative adversarial neural networks for compressive sensing MRI,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 1, pp. 167–179, 2018.
  * [93] H. Ali, M. R. Biswas, F. Mohsen, U. Shah, A. Alamgir, O. Mousa, and Z. Shah, “The role of generative adversarial networks in brain MRI: a scoping review,” _Insights into imaging_ , vol. 13, no. 1, p. 98, 2022.
  * [94] Y. Song and S. Ermon, “Improved Techniques for Training Score-Based Generative Models,” in _Advances in Neural Information Processing Systems_ , 2020.
  * [95] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” _arXiv preprint arXiv:2011.13456_ , 2020.
  * [96] H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye, “Diffusion posterior sampling for general noisy inverse problems,” _arXiv preprint arXiv:2209.14687_ , 2022.
  * [97] C. Yu, Y. Guan, Z. Ke, K. Lei, D. Liang, and Q. Liu, “Universal generative modeling in dual domains for dynamic MRI,” _NMR in Biomedicine_ , vol. 36, no. 12, p. e5011, 2023.
  * [98] A. Jalal, M. Arvinte, G. Daras, E. Price, A. G. Dimakis, and J. Tamir, “Robust compressed sensing MRI with deep generative priors,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 14 938–14 954, 2021.
  * [99] B. Levac, A. Jalal, and J. I. Tamir, “Accelerated Motion Correction for MRI Using Score-Based Generative Models,” in _IEEE International Symposium on Biomedical Imaging (ISBI)_ , 2023, pp. 1–5.
  * [100] C. Alkan, J. Oscanoa, D. Abraham, M. Gao, A. Nurdinova, K. Setsompop, J. M. Pauly, M. Mardani, and S. Vasanawala, “Variational Diffusion Models for Blind MRI Inverse Problems,” in _NeurIPS 2023 Workshop on Deep Learning and Inverse Problems_ , 2023.
  * [101] B. Levac, A. Jalal, K. Ramchandran, and J. Tamir, “MRI reconstruction with side information using diffusion models,” _arXiv preprint arXiv:2303.14795_ , 2023.
  * [102] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep Image Prior,” _International Journal of Computer Vision_ , 2020.
  * [103] D. P. Kingma and J. L. Ba, “Adam: a Method for Stochastic Optimization,” _International Conference on Learning Representations 2015_ , pp. 1–15, 2015.
  * [104] R. Heckel and M. Soltanolkotabi, “Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators,” in _International Conference on Learning Representations_ , 2020.
  * [105] R. Heckel and P. Hand, “Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks,” in _International Conference on Learning Representations_ , 2019.