# A Review on Deep Learning in Medical Image Reconstruction ††thanks: The work
of Haimiao Zhang is funded by China Postdoctoral Science Foundation under
grant No. 2018M641056. The work of Bin Dong is supported in part by National
Natural Science Foundation of China (NSFC) grant No. 11831002, and Beijing
Natural Science Foundation (No. Z180001).
$\\!{}^{\dagger}$Beijing International Center for Mathematical Research
(BICMR), Peking University, Beijing, 100871, P. R. China. Email:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
$\\!{}^{\ddagger}$Contact author: Bin Dong<EMAIL_ADDRESS>phone:
+86 10-62744091.

Haimiao Zhang† and Bin Dong† ‡

Abstract. Medical imaging is crucial in modern clinics to provide guidance to
the diagnosis and treatment of diseases. Medical image reconstruction is one
of the most fundamental and important components of medical imaging, whose
major objective is to acquire high-quality medical images for clinical usage
at the minimal cost and risk to the patients. Mathematical models in medical
image reconstruction or, more generally, image restoration in computer vision,
have been playing a prominent role. Earlier mathematical models are mostly
designed by human knowledge or hypothesis on the image to be reconstructed,
and we shall call these models handcrafted models. Later, handcrafted plus
data-driven modeling started to emerge which still mostly relies on human
designs, while part of the model is learned from the observed data. More
recently, as more data and computation resources are made available, deep
learning based models (or deep models) pushed the data-driven modeling to the
extreme where the models are mostly based on learning with minimal human
designs. Both handcrafted and data-driven modeling have their own advantages
and disadvantages. Typical handcrafted models are well interpretable with
solid theoretical supports on the robustness, recoverability, complexity,
etc., whereas they may not be flexible and sophisticated enough to fully
leverage large data sets. Data-driven models, especially deep models, on the
other hand, are generally much more flexible and effective in extracting
useful information from large data sets, while they are currently still in
lack of theoretical foundations. Therefore, one of the major research trends
in medical imaging is to combine handcrafted modeling with deep modeling so
that we can enjoy benefits from both approaches. The major part of this
article is to provide a conceptual review of some recent works on deep
modeling from the unrolling dynamics viewpoint. This viewpoint stimulates new
designs of neural network architectures with inspirations from optimization
algorithms and numerical differential equations. Given the popularity of deep
modeling, there are still vast remaining challenges in the field, as well as
opportunities which we shall discuss at the end of this article.
Keywords. Medical imaging, Deep learning, Unrolling dynamics, Handcrafted
modeling, Deep modeling, Image reconstruction
Subclass. 60H10, 92C55, 93C15, 94A08

##  1 Introduction

Medical image reconstruction can often be formulated as the following
mathematical problem

$\bm{f}=\bm{A}\bm{u}\oplus\bm{\eta},$ (1.1)

where $\bm{A}$ is a physical system modeling the image acquisition process.
Operator $\bm{A}$ can be a linear operator or nonlinear operator that depends
on the specific imaging modality. Variable $\bm{u}$ is the unknown image to be
reconstructed, and $\bm{f}$ is the measured data that might be contaminated by
noise $\bm{\eta}$ with known or partially known noise statistics, e.g.,
Gaussian, Laplacian, Poisson, Rayleigh, etc. The operator $\oplus$ is a
notation to denote addition when Gaussian noise is assumed, a certain
nonlinear operator when Poisson noise or Rician noise is assumed. In different
image reconstruction tasks, $\bm{A}$ takes different forms:

  * •

Denoising: $\bm{A}$ is an identity operator.

  * •

Deblurring: $\bm{A}$ is a convolution operator. When the convolution kernel is
unknown, the problem is called blind deblurring [120].

  * •

Inpainting: $\bm{A}$ is a restriction operator which can be represented by a
diagonal matrix with value 0 or 1 [12].

  * •

Magnetic resonance imaging (MRI): $\bm{A}$ is a sub-sampled Fourier transform
which is a composition of the Fourier transform and a binary sampling operator
[17].

  * •

X-Ray based computed tomography (CT): $\bm{A}$ is a sub-sampled Radon
transform, which is a partial collections of line integrations [23].

  * •

Quantitative susceptibility mapping (QSM) [39, 111, 45, 149]: $\bm{A}$ is the
dipole kernel

$A(X)=\frac{2z^{2}-x^{2}-y^{2}}{4\pi(x^{2}+y^{2}+z^{2})^{5/2}},\
X=(x,y,z)\in\mathbb{R}^{3}.$

The inverse problem (1.1) is in general challenging to solve due to the large-
scale and ill-posed nature of the problem in practice.

###  1.1 Image reconstruction models

The above inverse problem (1.1) covers a wide range of image restoration tasks
which are not limited to medical image reconstruction. To solve the inverse
problem (1.1), it is common practice to consider the following optimization
problem

$\min_{\bm{u}\in{\mathcal{D}}}\mathbb{\mathcal{L}}(\bm{u})=F(\bm{A}\bm{u},\bm{f})+\lambda\Phi(\bm{W},\bm{u}).$
(1.2)

The solution $\bm{u}^{\star}=\arg\min_{\bm{u}}\mathbb{\mathcal{L}}(\bm{u})$ is
an approximate solution to the inverse problem (1.1). In (1.2), the term
$F(\bm{A}\bm{u},\bm{f})$ is the data fidelity term that measures the
consistency of the approximate solution to the measured data $\bm{f}$. Its
specific form normally depends on the noise statistics. For example:

  * •

Gaussian noise:
$F(\bm{A}\bm{u},\bm{f})=\frac{1}{2}\|\bm{A}\bm{u}-\bm{f}\|_{2}^{2}$,

  * •

Poisson noise:
$F(\bm{A}\bm{u},\bm{f})=\langle\bm{1},\bm{A}\bm{u}\rangle-\langle\bm{f},log(\bm{A}\bm{u})\rangle$,
with $\langle\bm{a},\bm{b}\rangle=\sum_{i}\bm{a}_{i}\bm{b}_{i}$,

  * •

Impulse noise: $F(\bm{A}\bm{u},\bm{f})=\|\bm{Au}-\bm{f}\|_{1}$,

  * •

Multiplicative noise [129]:
$F(\bm{A}\bm{u},\bm{f})=\lambda_{1}\langle\frac{\bm{Af}}{\bm{u}},\bm{1}\rangle+\lambda_{2}\|\frac{\bm{Af}}{\bm{u}}-\bm{1}\|^{2}.$

The second term $\Phi(\bm{W},\bm{u})$ in (1.2) is the regularization term
encoding the prior knowledge on the image to be reconstructed. The
regularization term is often the most crucial part of the modeling, and what
people have mostly focused on in the literature. The parameter $\lambda$ in
(1.2) provides a balance between the data fidelity term and the regularization
term. Mathematical modeling has been playing a vital role in solving such
inverse problems. Interested reader can refer to [6, 32, 51, 133] for more
extensive reviews on mathematical models for image restoration.

Deep learning models can also be casted into the form of (1.2). However, there
are differences as well. In handcraft or handcraft $+$ data-driven modeling,
the transformation $\bm{W}$ is often a certain linear transformation that is
able to extract sparse features from the images. In handcraft models, $\bm{W}$
is often given by design (e.g. a differential operator or wavelet transform);
in handcraft $+$ data-driven models, $\bm{W}$ (or a portion of it) is often
learned from the given data. Sparsity is the key to the success of these
models. Deep learning models follow a similar modeling philosophy by
considering nonlinear sparsifying transformations rather than linear ones. In
general, we define a parameterized nonlinear mapping
$\mathcal{V}(\cdot,\bm{\Theta}):{\mathcal{F}}\to\mathcal{U},\bm{f}\mapsto\bm{u}$
that maps the input data $\bm{f}$ to a high quality output image $\bm{u}$. The
mapping $\mathcal{V}$ is parameterized by $\bm{\Theta}$ which is trained on a
given data set ${\mathcal{F}}\times\mathcal{U}$ by solving the following
optimization problem

$\min_{\Theta}\frac{1}{\\#({\mathcal{F}}\times\mathcal{U})}\sum_{(\bm{f},\bm{u})\in{\mathcal{F}}\times\mathcal{U}}\mathcal{C}(\mathcal{V}(\bm{f},\bm{\Theta}),\bm{u}),$

where $\mathcal{C}(\cdot,\cdot)$ is a metric of difference between the
approximated image $\mathcal{V}(\bm{f},\bm{\Theta})$ and the ground truth
image $\bm{u}$, and $\\#({\mathcal{F}}\times\mathcal{U})$ is the cardinality
of the data set ${\mathcal{F}}\times\mathcal{U}$. To prevent overfitting, we
can introduce a regularization term to the above optimization problem as in
(1.2). We then have the problem

$\min_{\bm{\Theta}}\mathcal{L}(\bm{f},\bm{u};\bm{\Theta})=\frac{1}{\\#({\mathcal{F}}\times\mathcal{U})}\sum_{(\bm{f},\bm{u})\in{\mathcal{F}}\times\mathcal{U}}\mathcal{C}(\mathcal{V}(\bm{f},\bm{\Theta}),\bm{u})+\mathcal{R}(\bm{\Theta}),$
(1.3)

where $\mathcal{R}(\cdot)$ is the regularization term that can be chosen as,
for example, the $\ell_{2}$ or $\ell_{1}$ norm. Good examples of the nonlinear
mapping $\mathcal{V}(\cdot)$ include the stacked denoising autoencoder (SDAE)
[147], the U-Net [128], the ResNet [72, 73], etc. We postpone a detailed
discussion on these networks and how to interpret them in mathematical terms
in later sections.

The development of modeling in image reconstruction for the past three decades
can be summarized to three stages:

  * •

Handcraft modeling (1990-). models are designed based on mathematical
characterizations on the desirable recovery of the image. For example, the
ideal function space a “good” image should belong to, ideal local or global
geometric properties the image should enjoy, or sparse approximation by
certain well-designed basis functions, etc. Successful handcraft models
include total variation (TV) [130] model, Perona-Malik diffusion [122, 121],
shock-filters [117, 4], nonlocal methods [19, 20, 21, 101, 42], wavelet [43,
105], wavelet frames [127, 48], BM3D [42], WNNM [66], etc. These models mostly
have solid theoretical foundations and high interpretability. They work
reasonably well in practice, and some of them are still the state-of-the-art
methods for certain tasks.

  * •

Handcraft $+$ data-driven modeling (1999-). Starting from around 1999, models
that combine data-driven or learning with handcraft modeling started to
emerge. These models rely on some general mathematical or statistical
framework by handcraft designs. However, the specific form of the model is
determined by the given image data or data set. Comparing to purely
handcrafted models, these models can better exploit the available data and
outperform their corresponding none data-driven counterparts. Meanwhile, the
handcrafted framework of the models grants certain interpretability and
theoretical foundation to the models. Successful examples include the method
of optimal directions [55], the K-SVD [3], learning based PDE design [98],
data-driven tight frame [26, 8], Ada-frame [141], low-rank models [154, 95,
27, 29, 24], piecewise-smooth image models [109, 25], and statistical models
[74], etc.

  * •

Deep learning models (2012-). 2012 is the year that signifies the uprise of
deep learning in computer vision with the introduction of a convolutional
neural network (CNN) called AlexNet [85] for image classification. Then,
various types of CNNs such as ResNet [72, 73] and generative adversarial
networks (GANs) [64] were introduced and applied in image reconstructions. We
shall refer to these models as deep learning based models (or deep models for
short). Most deep models have millions to billions of parameters. These
parameters are trained (optimized) on large data sets via parallel computing
(e.g., on graphics processing units (GPUs)). Deep models have greatly advanced
the state-of-the-art of many image reconstruction tasks and have changed the
research landscape of computer vision in general. The success of deep models
is mainly due to the presence of large image data sets with high-quality
labels, and the accessibility of massive computing resources. The reliance of
deep models on large labeled data sets limits, at least for the moment, the
application of deep learning in medical image reconstruction or healthcare in
general. The major focus of this review is to recall and discuss deep models
in medical image reconstruction, and the limitations, challenges, and
opportunities in this new and exciting research direction.

Note that what makes medical image reconstruction different from image
restoration in computer vision is quality metrics on the reconstructed image.
Although researchers use standard metrics such as the peak signal to noise
ratio (PSNR), mean square error, structure similarity (SSIM), etc., meaningful
quality metrics of a reconstructed medical image should be clinically relevant
and task dependent. Furthermore, most medical images are 3D arrays which poses
computation challenge as well.

###  1.2 Algorithm design for image reconstruction models

The difficulties of solving the image reconstruction models motivate the
optimization community to design highly efficient numerical algorithms for
large scale, nonsmooth and even nonconvex optimization problems.
Representative algorithms include the alternating direction method of
multipliers (ADMM) [16, 59, 61], primal-dual algorithm [177, 56, 31], split
Bregman algorithm [28, 62], linearized Bregman algorithm [163, 116] iterative
shrinkage-thresholding algorithm (ISTA) [44], and fast iterative shrinkage-
thresholding algorithm (FISTA) [10], among many others. Here, we briefly
review some of the algorithms that will be needed in later sections.

####  1.2.1 ISTA and FISTA

Consider the following optimization problem which is a special case of (1.2)

$\min_{\bm{\alpha}}\frac{1}{2}\|\bm{f}-\bm{W}^{\top}\bm{\alpha}\|_{2}^{2}+\lambda\|\bm{\alpha}\|_{1},$
(1.4)

where $\bm{W}^{\top}$ is a decoding operator that maps code $\bm{\alpha}$ back
to image domain. Then, the iterative soft-thresholding algorithm (ISTA)
solving (1.4) simply reads as

$\bm{\alpha}^{k+1}=\mathcal{T}_{\lambda\tau_{k}}\left(\bm{\alpha}^{k}-2\tau_{k}\bm{W}(\bm{W}^{\top}\bm{\alpha}^{k}-\bm{f})\right),$
(1.5)

where $\tau_{k}>0$ is an appropriate step size and the soft-thresholding
operator $\mathcal{T}_{\lambda}(\cdot)$ is defined component-wisely as
$\mathcal{T}_{\lambda}(x)=\text{\rm
sign}(x)\max(|x|-\lambda,0),\mbox{with}\;x\in\mathbb{R}.$ ISTA was explicitly
proposed in [44]. Its idea, however, can be traced back to the classical
proximal forward-backward algorithm [18, 119]. Later, an accelerated version
of ISTA, called fast iterative soft-thresholding algorithm (FISTA) was
introduced [10, 134] which is based on the idea of Nesterov’s [112]. FISTA
takes following form

$\displaystyle\bm{\alpha}^{k+1}$ $\displaystyle=$
$\displaystyle\mathcal{T}_{\lambda/L_{\rm{lip}}}\left(\bm{y}^{k}-\frac{1}{L_{\rm{lip}}}\bm{W}(\bm{W}^{\top}\bm{y}^{k}-\bm{f})\right),$
$\displaystyle t_{k+1}$ $\displaystyle=$
$\displaystyle\frac{1+\sqrt{1+4t_{k}^{2}}}{2},$ $\displaystyle\bm{y}^{k+1}$
$\displaystyle=$
$\displaystyle\bm{\alpha}^{k+1}+\frac{t_{k}-1}{t_{k+1}}(\bm{\alpha}^{k+1}-\bm{\alpha}^{k}),$
(1.6)

where $L_{\rm{lip}}$ is the Lipschitz constant of the quadratic term in (1.4).

####  1.2.2 ADMM/Split Bregman Algorithm

Consider the following special case of the optimization problem (1.2)

$\min_{\bm{u}}\mathbb{\mathcal{L}}(\bm{u})=\frac{1}{2}\|\bm{A}\bm{u}-\bm{f}\|_{2}^{2}+\lambda\|\bm{W}\bm{u}\|_{1},$

which can be written equivalently as

$\min_{\bm{u},\bm{d}}\mathbb{\mathcal{L}}(\bm{u},\bm{d})=\frac{1}{2}\|\bm{A}\bm{u}-\bm{f}\|_{2}^{2}+\lambda\|\bm{d}\|_{1},\quad
s.t.\quad\bm{W}\bm{u}=\bm{d}.$

The corresponding augmented Lagrangian function [114, Chapter 17] is defined
by

$\mathbb{\mathcal{L}}(\bm{u},\bm{d};\bm{b})=\frac{1}{2}\|\bm{A}\bm{u}-\bm{f}\|_{2}^{2}+\lambda\|\bm{d}\|_{1}+\langle\bm{W}\bm{u}-\bm{d},\bm{b}\rangle+\frac{\mu}{2}\|\bm{W}\bm{u}-\bm{d}\|_{2}^{2},$

with the Lagrangian multiplier $\bm{b}$. Then, the alternating direction
method of multipliers (ADMM) or split Bregman algorithm takes the form [62,
16]

$\displaystyle\bm{u}^{k+1}$ $\displaystyle=$
$\displaystyle\left(\bm{A}^{\top}\bm{A}+\mu\bm{W}^{\top}\bm{W}\right)^{-1}\left[\bm{A}^{\top}\bm{f}+\mu\bm{W}^{\top}(\bm{d}^{k}-\bm{\nu}^{k})\right],$
$\displaystyle\bm{d}^{k+1}$ $\displaystyle=$
$\displaystyle\mathcal{T}_{\lambda/\mu}\left(\bm{W}\bm{u}^{k+1}+\bm{\nu}^{k}\right),$
$\displaystyle\bm{\nu}^{k+1}$ $\displaystyle=$
$\displaystyle\bm{\nu}^{k}+(\bm{W}\bm{u}^{k+1}-\bm{d}^{k+1}),$ (1.7)

where $\mu$ is a tuning parameter.

####  1.2.3 The Primal-Dual Algorithm

Consider the following optimization problem

$\min_{\bm{u}}F(\bm{u})+\Phi(\bm{W}\bm{u}),$ (1.8)

where $F(\bm{u})$ is the data fidelity term and $\Phi(\bm{W}\bm{u})$ is the
regularization term appeared in (1.2). Assume
$F:\mathbb{R}^{n}\to(-\infty,+\infty]$ and
$\Phi:\mathbb{R}^{m}\to(-\infty,+\infty]$ are closed proper convex functions.
The problem (1.8) can be written equivalently as

$\min_{\bm{u}}\max_{\bm{w}}F(\bm{u})+\langle\bm{W}\bm{u},\bm{w}\rangle-\Phi^{\ast}(\bm{w}),$
(1.9)

Then, the primal-dual hybrid gradient (PDHG) algorithm [177, 56, 31] can be
written as

$\displaystyle\bm{w}^{k+1}$ $\displaystyle=$
$\displaystyle(I+\partial\Phi^{\ast})^{-1}\left(\bm{w}^{k}+\alpha_{k}\bm{W}\bm{u}^{k}\right),$
$\displaystyle\bm{u}^{k+1}$ $\displaystyle=$ $\displaystyle(I+\partial
F)^{-1}\left(\bm{u}^{k}-\beta_{k}\bm{W}^{\top}\bm{w}^{k+1}\right),$ (1.10)

where $\alpha_{k}$ and $\beta_{k}$ are tuning parameters. Note that in [31],
the authors introduced an additional correction update step,

$\bar{\bm{u}}^{k+1}=\bm{u}^{k+1}+\theta(\bm{u}^{k+1}-\bm{u}^{k}),$ (1.11)

to the original PDHG algorithm (1.2.3) and replaced $\bm{u}^{k}$ in
$\bm{w}^{k+1}$-step by $\bar{\bm{u}}^{k}$.

####  1.2.4 SGD

It is very common in machine learning that an optimization problem takes the
following form

$\min_{\bm{\Theta}}F_{N}(\bm{\Theta})=\frac{1}{N}\sum_{i=1}^{N}f_{i}(\bm{\Theta}).$
(1.12)

The main computation challenge, especially in deep learning, is that $N$ can
be huge, e.g., in the magnitude of millions to billions. Therefore, the
evaluation of the function value $F_{N}$ and its gradient can be rather slow.
In such case, stochastic gradient descent (SGD) algorithm [13, 125, 14, 170]
and its variants [113, 173, 84] are among the most popular algorithms in deep
learning.

The very basic form of (mini-batch) SGD is

$\bm{\Theta}^{k+1}=\bm{\Theta}^{k}-\alpha_{k}\frac{1}{|\mathcal{S}_{k}|}\sum_{i_{k}\in\mathcal{S}_{k}}\nabla
f_{i_{k}}(\bm{\Theta}^{k}),$

where $\alpha_{k}$ is the step size (or learning rate) and $\mathcal{S}_{k}$
is a random subset of $\\{1,2,\ldots,N\\}$. The evaluation of
$\frac{1}{|\mathcal{S}_{k}|}\sum_{i_{k}\in\mathcal{S}_{k}}\nabla
f_{i_{k}}(\bm{\Theta}^{k})$ provides an unbiased estimation of the full
gradient and is computationally cheap. Other than SGD, numerous randomized
algorithms are being used in deep learning, such as Adam [83], AdaGrad [52],
RMSProp [76], etc. A comprehensive review on optimization algorithms for large
scale machine learning problems can be found in [15].

###  1.3 When handcraft modeling meets deep learning

Both handcrafted models and deep models have their advantages and drawbacks
depending on the applications. Most handcrafted models are designed with a
solid mathematical foundation and can be very well interpreted. However,
handcrafted models are not flexible enough to fully leverage large data sets.
Deep models, on the other hand, are generally much more flexible and can
better extract useful information from large data sets. However, they are
generally more challenging to interpret. For the moment, they are also in lack
of theoretical foundations in contrast to handcrafted models. Therefore, there
has been an increasing effort in the community to combine handcrafted modeling
and deep modeling so that we can enjoy benefits from both approaches.

One of the most popular ways of such combination is the so-called unrolling
dynamics approach. It started with the work of [65] where the authors showed
that one could unroll the iterative soft-thresholding algorithm (ISTA) (1.5)
to create a feed-forward network. Then, one can train ISTA in an end-to-end
fashion to determine the parameters in ISTA so that they are best suitable for
the training data. They called the unrolled dynamics LISTA and demonstrated
its advantage over ISTA. This work showed that one could unroll a discrete
dynamic system to form a network for end-to-end training. More recently, more
and more examples showed that the unrolling dynamics approach seems a good
balance between model interpretability and efficacy. This includes unrolling
discrete forms of nonlinear diffusions for image restoration [98, 38] and
unrolling optimization algorithms for medical imaging and inverse problems
[139, 2, 136, 37, 97, 87, 168]. The unrolling dynamics approach can often
result in deep models that have better interpretability inherited from the
original dynamics.

Furthermore, these deep models normally have much fewer trainable parameters
than black-box deep neural networks, which are more suitable for learning on
relatively small data sets. On the other hand, we may interpret certain
classes of deep convolutional networks, such as ResNet, as discrete dynamics,
and hence relates deep learning with optimal control [151, 34]. Such viewpoint
not only provides elegant interpretation of deep neural networks [90], but
also enable us to design more effective deep networks for various tasks in
machine learning [33, 103, 88, 148, 131, 142, 166], computer vision [171],
inverse problems [100, 99], and natural language processing [102]. More
recently, intriguing relations between deep convolutional networks with
multigrid method are addressed [157] which lead to new interpretations to deep
models.

The remainder of this paper is organized as follows. In Section 2, we will
review some recently proposed deep neural networks which are popular in
medical imaging. Section 3 shows the understanding of deep neural networks
from the perspective of representation learning and differential equations.
Section 4 reviews some recently proposed deep modeling for medical imaging,
where Section 4.1 presents some examples of post-processing deep models,
Section 4.2 collects some models that are designed by combining handcrafted
modeling with deep modeling, and Section 4.3 reviews task-driven deep models.
To conclude, Section 5 summarizes the main challenge and opportunities in deep
learning based medical imaging.

##  2 Review of Deep Neural Networks

Deep neural networks (DNNs) are now proven to be powerful tools to represent
complex data. The main differences between DNNs and traditional machine
learning models are the composite nonlinearity of the DNNs and the end-to-end
training, which make DNNs very effectively in extracting features that are
most suitable for a given task. In recent years, DNNs are used in various
medical imaging tasks, including image reconstruction, segmentation, region-
of-interest detection, super-resolution, classification, etc. In this section,
we briefly recall some of the DNNs that are widely adopted in medical imaging.

###  2.1 ResNet

In computer vision, the residual network (ResNet) [72, 73] is one of the most
popular DNNs. The architecture of ResNet is shown in Figure 1 which can also
be formulated mathematically as

$\bm{u}_{k+1}=\bm{u}_{k}+{\mathcal{F}}_{k}(\bm{u}_{k}),$ (2.1)

where $\bm{u}_{k}$ (resp. $\bm{u}_{k+1}$) indicates the input (resp. output)
feature map of the $k$-th layer of the ResNet and
${\mathcal{F}}_{k}(\bm{u}_{k})$ is called a nonlinear residual block with
trainable parameters. The skip connection of ResNet is crucial in facilitating
stable training when the network is very deep. Other DNNs with the similar
skip connections include the learned diffusion model TRD [38], DenseNet [79]
and U-Net [128], among many others.

Figure 1: ResNet.

###  2.2 Autoencoder

Autoencoder (AE) [11, 124] is a type of neural network that is used to learn
data representation in an unsupervised manner. It aims to learn an encoder
from a set of data together with a decoder so that we do not lose any
essential information during the encoding and decoding process. Figure 2
presents a typical example of the AE architecture. For a given image $\bm{X}$,
the parameterized mapping $f_{\bm{\theta}}$ (e.g. a fully connected or a
convolutional neural network) is an encoder that extract feature maps from
$\bm{X}$. The encoded multi-channel feature maps are denoted by
$\bm{Y}=f_{\bm{\theta}}(\bm{X})$. The encoded feature maps $\bm{Y}$ is then
decoded by another parameterized mapping $g_{\bm{\theta}^{\prime}}$ to obtain
the reconstructed data $\bm{Z}$. The parameters $\bm{\theta}$ and
$\bm{\theta}^{\prime}$ are optimized on a data set so that a properly chosen
loss function that measures the average discrepancies between $\bm{X}$ and
$\bm{Z}$ is minimized. AE resembles linear representations such as Fourier and
wavelet transform if we regard encoding as the decomposition, decoding as the
reconstruction and feature maps as the coefficients of the representation.
However, the representation provided by AE is nonlinear and is learned from a
data set.

To learn a more effective and robust representation, [147] proposed the
stacked denoising autoencoder (SDAE). In SDAE, the encoder and decoder are
DNNs, and they are trained to recover $\bm{Z}\approx\bm{X}$ from noisy input
$\bm{X}$. Based on the encoder/decoder framework, [7] designed a DNN, called
SegNet, for image segmentation. In [106], the encoder/decoder framework is
adopted for image denoising and super-resolution. More recently, [35] designed
a residual encoder-decoder CNN to suppress the noise and preserve features in
low-dose CT images that are reconstructed using the filtered back projection
(FBP) algorithm.

Figure 2: Autoencoder.

###  2.3 U-Net

In [128], a U-shaped deep neural network, called U-Net, was proposed for
biomedical image segmentation which is by far one of the most successful deep
image segmentation models. The architecture of U-Net is shown in Figure 3. It
resembles the encoder/decoder architecture of AE if we view the left half of
the U-Net as an encoder and the right half as a decoder. The main difference
between the U-Net and AE is the use of skip connections of U-Net. Similar to
the U-Net, [108] designed a DNN, called V-Net, for 3D volumetric medical image
segmentation. Motivated by the U-Net and the convolutional framelets [162],
[161] designed a multi-resolution deep convolutional framelets. More recently,
U-Net is extended to image analysis tasks [57].

Figure 3: U-Net.

##  3 Interpretations of Deep Neural Networks

The development of traditional machine learning methods, such as support
vector machine, decision tree, random forest, etc., benefit tremendously from
theoretical studies in machine learning. However, existing machine learning
theory, such as PAC, VC-dimension, Rademacher complexity, etc., may not be
most suitable to analyze DNNs. Although DNNs are often composed of simple
functions, such as convolutions, pooling, element-wise activation functions,
etc., the entire networks are often difficult to analyze. Therefore,
theoretical deep learning has become a popular area in machine learning that
has attracted a lot of attention from theoretical computer scientists,
statisticians, and mathematicians. In this section, we shall review some
recent works on interpreting DNNs from two different perspectives, namely
representation learning and differential equations. We will see that function
approximation is a powerful tool in characterizing the efficacy of the given
representation. It provides a rigorous analysis of the capacity of DNNs and
how well they can approximate functions living in various function spaces. The
perspective through differential equations, on the other hand, is more
intuitive than function approximation and can explicitly guide the design of
architectures of DNNs and training algorithms. There are also several other
perspectives on the theoretical interpretations of deep learning. One may
refer to the course “Theories of Deep Learning” (STATS 385) hosted by David
Donoho at Stanford University and the references therein
(https://stats385.github.io/).

###  3.1 Representation Learning Perspective

Images, such as medical images or natural images, are usually assumed to have
sparse (or low dimensional) structures. The sparse structures can be
effectively extracted by transformations. Successful examples include the
(windowed) Fourier transform, wavelet transform, curvelet transform, etc, and
they are able to provide efficient representations to images. They are pre-
designed linear transformations and are independent of the given image data.
DNNs can also be viewed as sparse representations that are able to extract
sparse features from images. The difference is that DNNs are learned from a
set of images and are (highly) nonlinear.

The quality of a given representation can be measured by its ability to
approximate functions living in a certain function space. For example, let
$\Phi:=\\{\phi_{i}(\bm{x}):\bm{x}\in\mathbb{R}^{n},i\in\mathbb{N}_{+}\\}$ be a
set of atoms, and function $f(\bm{x})$ be an element in function space
$\mathcal{F}$ equipped with norm $\|\cdot\|$. One of the most basic and
important approximation properties states as follows: for any given
$\varepsilon>0$, there exists
$\tilde{f}_{\bm{\alpha},N}:=\sum_{i=1}^{N}\alpha_{i}\phi_{i}(\bm{x})$ with
$N\in\mathbb{N}_{+}$ and
$\bm{\alpha}=\\{\alpha_{1},\ldots,\alpha_{N}\\}\in\mathbb{R}^{N}$ such that

$\|f-\tilde{f}_{\bm{\alpha},N}\|<\varepsilon.$

A good representation requires fewer atoms (i.e. smaller $N$) to achieve an
$\varepsilon$-approximation. The representation of various types of $\Phi$
have been well studied in the literature, such as polynomials, splines,
Fourier basis and wavelets [47, 43, 105].

The neural network is a more efficient tool that can approximate a function
arbitrarily well under suitable conditions [77, 78, 123]. Both the depth and
width of a neural network are among the most important factors that affect its
approximation power. In the following, we will review some of the existing
characterizations of the approximation properties of shallow and deep neural
networks.

Consider a shallow neural network with only one hidden layer

$\tilde{f}_{N}(\bm{x};\bm{\Theta})=\sum_{i=1}^{N}a_{i}\sigma(\bm{w}_{i}^{\top}\bm{x}+b_{i}),$

where $\bm{x}\in\mathbb{R}^{n}$ is the input image data,
$\bm{\Theta}=\\{a_{i},\bm{w}_{i},b_{i}\\},i=1,\ldots,N,$ are trainable
parameters, and $\sigma(z)$ is an element-wisely applied nonlinear activation
function. Examples of $\sigma(z)$ are $\mbox{ReLU}(z)=\max(0,z)$,
$\mbox{tanh}(z)=\frac{e^{z}-e^{-z}}{e^{z}+e^{-z}}$,
$\mbox{sigmoid}(z)=\frac{1}{1+e^{-z}}$ and more generally a sigmoidal function
[41] that has the property:

$\sigma(z)=\begin{cases}1&\text{if}\;z\to+\infty,\\\
0&\text{if}\;z\to-\infty.\end{cases}$ (3.1)

A DNN is a neural network with multiple hidden layers. It can be viewed as a
successive composition of multiple shallow networks. A typical DNN (for
regression problems) with depth $L$ and width
$\bm{N}=(N_{1},N_{2},\ldots,N_{L})$ denoted as

$\tilde{f}_{L,\bm{N}}(\bm{x};\bm{\Theta}):\mathbb{R}^{n}\mapsto\mathbb{R},$

can be recursively defined as:
$\bm{\Theta}^{\ell}=(\bm{\Theta}^{\ell-1},\theta^{\ell})$,
$\tilde{f}_{\bm{\Theta}^{\ell}}=(\theta^{\ell}\circ\sigma\circ\tilde{f}_{\bm{\Theta}^{\ell-1}})$,
$\theta^{\ell}:\mathbb{R}^{N_{\ell}}\to\mathbb{R}^{N_{\ell+1}}$ with
$\theta^{\ell}(\bm{x})=\bm{W}^{\ell}\bm{x}+\bm{b}^{\ell}$, and
$\tilde{f}_{L,\bm{N}}:=\tilde{f}_{\bm{\Theta}^{L}}$.

Earlier results on the approximation property, i.e., universal approximation,
suggest that a wide class of functions can indeed be approximated by neural
networks with only one hidden layer, though the number of neurons, i.e. $N$,
may increase exponentially as we decrease $\varepsilon$ [58, 9, 41]. There are
benefits in increasing the depth $L$ of the neural network when approximating
a target function. For example, approximation with DNNs leads to an
exponential or polynomial reduction in the number of neurons while maintaining
the same level of approximation accuracy [91, 107, 54, 40]. [46, 143, 144]
presented concrete examples that there exist functions that can be more
efficiently represented with DNNs rather than shallow networks. In particular,
[144] showed that the DNNs with $\mathcal{O}(L^{3})$ layers and constant width
cannot be approximated by networks with $\mathcal{O}(L)$ depth and less than
$2^{L}$ width.

[104] investigated the efficiency of depth of ReLU activated DNNs from a
different angle by proving that there exist classes of wide neural networks
which cannot be realized by any narrow network whose depth is no more than a
polynomial bound. Comparing to the known result that there are classes of deep
networks which cannot be realized by any shallow network whose size is no more
than an exponential bound [40], results from [104] indicated that depth might
be more effective than width. Although depth is more important than width,
[69, 68] proved that there is a minimum width of ReLU activated DNNs to ensure
approximation of continuous functions. Their results indicated that a good DNN
cannot be too narrow, otherwise we cannot approximate continuous functions
even with infinite depth.

More recently, [160] analyzed the dependence of optimal approximation rate
with respect to depth for ReLU activated DNNs. When approximating a
multivariate polynomial, [126] proved that the total number of neurons in DNNs
should grow linearly with respect to the number of variables of the
polynomial. [135] provided intriguing analysis on ReLU activated DNNs via
nonlinear approximation of composite dictionaries. They demonstrated the
advantage of depth over width quantitatively by comparing the $N$-term
approximation order of DNNs v.s. one-hidden-layer neural networks. Other than
generic DNNs, theoretical analysis on the popular ResNet were also provided
[146, 92, 53].

In [70], the authors investigated the connection between linear finite element
functions and ReLU deep neural networks. Firstly, they proposed an efficient
ReLU activated DNN to represent any linear finite element functions and
theoretically established that at least $2$ hidden layers are needed in a ReLU
activated DNN to represent any linear finite element functions in
$\Omega\subseteq\mathbb{R}^{d}$ when $d\geq 2$. Then, using this relationship
they established a straightforward error estimate as
$\mathcal{O}(N^{-\frac{1}{d}})$ for a special kind of ReLU activated DNNs with
$\mathcal{O}(N)$ non-zero parameters by involving the h-adaptive linear finite
element approximation theory [115].

Different from the approximation viewpoint, [71] developed a unified model,
known as MgNet, that simultaneously recovers and extends some CNNs for image
classification and multigrid methods for solving discretized PDEs, by
combining multigrid and deep learning methodologies.

###  3.2 Differential Equation and Control Perspective

Given a DNN $\tilde{f}(\bm{x};\bm{\Theta}):\mathbb{R}^{n}\to\mathbb{R}^{m}$,
due to its composite structure as described in the previous subsection, we may
view $\tilde{f}(\bm{x};\bm{\Theta})$ as an iterative mapping between
$\mathbb{R}^{n}$ and $\mathbb{R}^{m}$. Then it is natural to view a generic
DNN as a certain dynamic system [30]. However, a dynamic system that
corresponds to a generic DNN is difficult to analyze since it does not have
much special structure to exploit. Fortunately, it has been proven empirically
that most of the effective DNNs have special structures in their architecture.
In fact, designing special structures of DNNs, i.e. the architecture design,
to make them easy to train and generalize better is one of the major research
directions in deep learning. Furthermore, the objective of the emerging
research topic neural architecture search (NAS), a subfield of automating
machine learning (AutoML), is to search for effective DNN architecture for
different data sets and tasks.

One of the most well-known DNNs with special structures is ResNet. Its
bypasses (or shortcuts) enable us to efficiently train ultra-deep networks and
achieve high accuracies in multiple tasks. The success of ResNet inspired the
design of numerous new neural architectures. However, most of the design were
based on empirical studies. Although we can deploy NAS to search for new
architectures, the current computation burden of NAS is still prohibitively
high for researchers without access to heavy computation resources, and NAS
cannot guarantee to find sufficiently new and interpretable neural structures.
Therefore, we direly need a way to interpret ResNet and their siblings
properly and to seek for guiding principles for the architecture design of
DNNs.

Recently, [151] made an inspiring observation that ResNet can be viewed as
forward Euler scheme solving for an ordinary differential equation (ODE), and
links training of DNNs with optimal control. [137] and [90] also regarded
ResNet as dynamic systems that are the characteristic lines of a transport
equation on the data distribution. Similar observations were also made by [33,
34]. A rigorous justification of the link between ResNet and ODEs was given by
[145], and that of the link between deep learning and optimal control was
given by [152]. The dynamics and control perspective enabled us to design more
efficient algorithms solving related deep learning problems [89, 36, 171,
166].

In [103], the authors suggested a general bridge between numerical ODEs and
deep neural architectures by observing that multiple state-of-the-art deep
network architectures, such as PolyNet [172], FractalNet [86] and RevNet [63],
can be viewed as different discretizations of ODEs. Furthermore, [103] proved
that ResNet with certain stochastic training strategies weakly approximates
stochastic differential equations, which granted stochastic control
perspective on randomized training of DNNs. More importantly, such new
perspectives enable us to systematically design deep neural architectures
through numerical (stochastic) differential equations, which is a rather
mature field in applied mathematics. In this section, we shall review some of
the findings of [103] and some other related works.

####  3.2.1 Numerical difference equations and architecture design

(a) ResNet

(b) PolyNet

(c) FractalNet

(d) RevNet

(e) LM-ResNet

Figure 4:  Schematics of different neural network architectures

We first show that how ResNet is related to forward Euler scheme in numerical
ODEs. Considering a building block of ResNet (2.1) as shown in Figure1, it can
be rewritten as

$\bm{u}_{k+1}=\bm{u}_{k}+\Delta t_{k}{\mathcal{F}}(\bm{u}_{k},t_{k}),$

or equivalently as

$\frac{\bm{u}_{k+1}-\bm{u}_{k}}{\Delta
t_{k}}={\mathcal{F}}(\bm{u}_{k},t_{k}),$

where $\Delta t_{k}$ is the step size and
${\mathcal{F}}(\bm{u}_{k},t_{k})=\frac{1}{\Delta
t_{k}}{\mathcal{F}}_{k}(u_{k})$. The above formula is the forward Euler scheme
solving the following ordinary differential equation (ODE)

$\frac{d\bm{u}}{dt}={\mathcal{F}}(\bm{u},t).$ (3.2)

Therefore, the ResNet can be viewed as the forward Euler discretization of the
ODE (3.2) with step size $\Delta t_{k}=1$ for every $k$. This was first
observed by [151]. More recently, [169] showed that there are benefits of
considering ResNet with $0<\Delta t_{k}<1$.

In [103], the authors further observed that many other DNNs with bypasses,
e.g., PolyNet [172] (Figure 4(b)), FractalNet [86] (Figure 4(c)) and RevNet
[63] (Figure 4(d)), can be interpreted as certain temporal discretizations of
ODEs. For example, the PolyInception module (Figure 4(b)) of PolyNet can be
written mathematically as

$(I+{\mathcal{F}}+{\mathcal{F}}^{2})(x)=x+{\mathcal{F}}(x)+{\mathcal{F}}({\mathcal{F}}(x)),$

where $I$ is the identity map, ${\mathcal{F}}$ is a nonlinear operator and $x$
is the input feature map. Note that the above polynomial of mapping
${\mathcal{F}}$ is an approximation of $(I-{\mathcal{F}})^{-1}$ using a
truncated Neumann series:

$(I-\varDelta t{\mathcal{F}})^{-1}\approx(I+\varDelta t{\mathcal{F}}+\varDelta
t^{2}{\mathcal{F}}^{2}).$

Therefore, PolyNet can be viewed as an approximation to the backward Euler
scheme solving the ODE (3.2). FractalNet (Figure 4(c)) can be viewed as
approximation of the ODE (3.2) with Runge-Kutta scheme. See [103] for more
examples and further details.

These examples suggest a potential link between numerical ODEs and deep neural
architecture. A remaining question is whether deep neural architecture design
can benefit from such perspective. The authors of [103] designed a new ResNet-
like module, called the linear multi-step structure (LM-structure) using the
linear multi-step schemes in numerical ODEs [5]. The LM-structure (linear two-
step structure to be more precise) can be written mathematically as

$\bm{u}_{k+1}=(1-\gamma_{k})\bm{u}_{k}+\gamma_{k}\bm{u}_{k-1}+{\mathcal{F}}(\bm{u}_{k},t_{k}),$
(3.3)

where $\gamma_{k}\in\mathbb{R}$ is a trainable parameter in each layer. Note
that when $\gamma_{k}=0$ for all $k$, the LM-structure reduces to ResNet.
Figure 4(e) shows the LM-structure. Empirical results of [103] showed that the
LM-structure boost classification accuracies of ResNet-like DNNs on CIFAR and
ImageNet. It can also reduce the depth (hence number of parameters) of ResNet-
like DNNs by 50–90% without hampering accuracies. Other than the LM-structure,
one can use the mid-point scheme or the leapfrog scheme to design new DNNs
[33], or using the Runge-Kutta method [176].

The performance gain of the LM-structure can be explained using the concept of
modified equations [150]. By Taylor’s expansion, the modified equation
associated with the LM-structure (3.3) is

$(1+\gamma_{k})\dot{\bm{u}}_{k}+\frac{1-\gamma_{k}}{2}\Delta
t\ddot{\bm{u}}_{k}={\mathcal{F}}(\bm{u}_{k},t).$ (3.4)

Comparing to ResNet, the LM-structure has the freedom to balance between
$\ddot{\bm{u}}_{k}$ of $\bm{u}_{k}$. Having bigger weights on
$\ddot{\bm{u}}_{k}$ can speed up the information propagation of the dynamics
as shown by various earlier work such as [138, 153, 49]. This is why LM-
structure can achieve comparable accuracies with a much smaller depth than
ResNet and its siblings.

####  3.2.2 Stochastic training and optimal control

Stochastic training, such as dropout and noise injections, is widely adopted
in training of DNNs. It helps with the generalization of the trained networks.
In [103], the authors showed that some stochastic training of ResNet, shake-
shake [60] and stochastic depth [80], can be viewed as stochastic control

$\displaystyle\min_{\bm{\Theta}}\mathbb{E}_{\bm{X}(0)\sim
data}\left[\mathbb{E}[L(\bm{X}(T))]+\int_{0}^{T}R(\bm{\Theta})\right]$
$\displaystyle s.t.\quad
d\bm{X}={\mathcal{F}}(\bm{X},\theta)dt+\mathcal{G}(\bm{X},\theta)d\bm{B}_{t},$
(3.5)

where the stochastic differential equation (3.2.2) is the weak limit of the
ResNet with shake-shake mechanism or stochastic depth. This suggests a
connection between stochastic training and stochastic control, and a
connection between DNNs with randomness and stochastic differential equations.
Later, [140] observed that the stochastic training of ResNet and its variants
is closely related to the optimal control of backward Kolmogorov’s equations,
and the popular dropout regularization essentially introduces viscosity to the
equations.

##  4 Deep Models in Medical Image Reconstruction

Classical medical image reconstruction methods, such as FBP and algebraic
reconstruction method (ART) for CT imaging, are highly efficient and widely
used in practice [110]. However, these methods are also prone to be sensitive
to noise and incompleteness of measured data. To obtain a high-quality image,
numerous regularization based models and algorithms have been developed [164,
132, 75] in the past three decades. In recent years, there has been a
continuous effort in the medical imaging community to further advance medical
image reconstruction by combining traditional image reconstruction methods
with deep learning. When combining traditional handcraft modeling with deep
modeling, two general approaches are often adopted: post-processing and raw-
to-image. The validity of these two approaches are generally supported by,
though still rather incomplete, the analysis on the approximation properties
of DNNs as described in Section 3.1, and by the dynamics perspective on the
DNNs with certain special structures as described in Section 3.2.

For post-processing, one needs to estimate the mapping between the initially
reconstructed low-quality image and its high-quality counterpart. This is
possible since DNNs can approximate generic functions or mappings as discussed
in Section 3.1. This approach is effective mostly when the initial
reconstruction and its high-quality counterpart are not drastically different.
However, due to limited measurements and the presence of noise, the initially
reconstructed image may contain heavy and complex artifacts which are
difficult to remove even by deep models. Furthermore, the information missing
from the initial reconstruction cannot be reliably recovered by any post-
processing. Thus, the post-processing approach has limited performance and is
more suitable to handle initial reconstructions that are of relatively high
quality.

For raw-to-image, one directly estimates the mapping between the raw data
(e.g. the projection data of CT and k-space data for MRI) and the
reconstruction image. The challenge, however, is that the data distribution in
the domain of raw data is often vastly different from that in the image
domain. Learning a direct mapping using a DNN without special structures
(e.g., a fully connected network or a vanilla CNN), though not impossible, may
require tons of training data, can be computationally expensive and heavily
relies on good initializations of the model parameters (e.g., the AUTOMAP
[175]). It is well-known in the literature of handcrafted modeling that the
mapping ought to have certain dynamic structures which can be represented by a
carefully designed (partial) differential equation or an optimization
algorithm solving a certain objective function(al). Thus, it is more plausible
to combine handcrafted dynamics with deep learning. The way of such
combination was depicted in Section 3.2 in a relatively general setting where
we did not discuss how $\mathcal{F}$ should be designed for a given image
restoration problem. Nonetheless, it is rather convincing that there are
connections between dynamic systems and DNNs and the benefits of recognizing
such connections.

Our rich history of handcraft modeling in image restoration provides us with
an abundant set of tools that we can select freely for the mapping
$\mathcal{F}$ via the general technique known as the unrolling dynamics [65,
139]. To be more precise, this approach first suggests us to unroll
optimization algorithms that are introduced to solve handcrafted models into
feed-forward networks. Then, we incorporate our domain knowledge of the
problem in-hand to determine which parameters are best to be learned from the
data in an end-to-end fashion. The advantage of designing deep models via
unrolling optimization algorithms is threefold: 1) the deep model through
unrolling dynamics is more interpretable than a regular deep model such as
U-Net; 2) the number of parameters are normally less than regular deep models
and thus more suitable for small sample learning; 3) it provides a general way
of combining domain knowledge with deep learning so that we can easily decide
on which component in the model need to be learned and which can be
handcrafted without losing much expressive power of the model.

As mentioned in the introduction that one of the major differences between
medical image reconstruction and image restoration in computer vision is the
quality metric of the reconstruction images. It has long been discussed in the
medical imaging community that such a quality metric is best, in many
scenarios, to be task-based rather than generic metrics such as PSNR and SSIM.
The importance of providing such a task-based metric for medical imaging was
recently discussed in the article [82]. The question is, however, how can we
realize such task-based image quality metric? Recently, the authors of [155]
proposes to realize task-based quality metric by “hooking” a image
reconstruction network from unrolling dynamics with a image analysis DNN, so
that the reconstructed images by the first network will be implicitly
evaluated by the second which effectively makes the quality metric task-based.
Similar idea appeared in computer vision for image denoising [94, 93]. On the
other than, these work also suggested a new “raw-to-task” modeling philosophy
with encouraging empirical results. Therefore, the entire pipeline of image
reconstruction, analysis, and decision making can be effectively integrated.

In the rest of this section, we provide more details on the aforementioned
models.

###  4.1 Post-Processing

(a) Ground truth

(b) Low dose CT

(c) Low tube current CT

Figure 5:  FBP reconstructed images

Post-processing is a procedure to enhance the quality of an initially
reconstructed image. In this subsection, we use CT as an example. Due to the
incompleteness of measured data in sparse view and limited angle CT, the FBP
reconstructed image is often degraded by streaking artifacts (Figure 5(b)).
Noise caused by low tube currents is another source of degradation of CT
images (Figure 5(c)). In [35, 174], a residual encoder-decoder CNN (Figure 2)
was used to approximate the map between the degraded image and the clean
image. This model is efficient in removing noise from the FBP reconstructed CT
images. To protect subtle structures in CT images while suppressing noise,
[158] adopted a generative adversarial network (GAN) with the loss function
defined by a combination of Wasserstein distance and the perceptual difference
between input degraded image and the corresponding clean image.

To reduce the radiation dose and acquisition time, one can decrease the number
of projections of X-ray CT, which is known as the sparse view or limited angel
CT. Such incompleteness of measurements leads to streaking artifacts with
global and yet relatively simple structures in the FBP reconstructed CT
images. In this case, a DNN with multi-scale architecture can be used to
capture the global patterns of streaking artifacts. With such observation,
[81, 67] utilized the U-Net (Figure 3) to reduce artifacts in FBP
reconstructed sparse view CT images. The repaired high-quality CT image is the
subtraction of the learned artifacts by the U-Net from the degraded input
image. In some sense, U-Net takes a role of residual learning [72].

###  4.2 Raw-to-Image

We describe how optimization algorithms can be unrolled and set up as a deep
feed-forward network for end-to-end training. We remark that, under some
specific conditions, the learning empowered optimization algorithms via
unrolling dynamics can have better provable convergence than the original
optimization algorithms [37, 96, 156]. This was in fact the original
motivation of [65] to use machine learning to improve optimization algorithms.
In this subsection, however, we shall focus on the “dual” aspect of unrolling
dynamics, i.e. how optimization algorithms inspire new and more effective deep
network architectures for medical image reconstruction or inverse problems in
general.

#### ADMM-Net

The work of ADMM-Net proposed by [139] was the first to suggest the potential
benefit of designing deep neural networks for inverse problems by unrolling
optimization algorithms.

In the iteration scheme (1.2.2) of ADMM algorithm (Subsection 1.2.2), the
tuning parameters such as $\mu,\lambda$ and $\beta_{k}$, the handcrafted
operator $\bm{W}$ and function $\Phi$ are difficult to determine adaptively
for a given data set. In [139], the authors proposed to unroll the ADMM
algorithm to design a new deep model, named ADMM-Net. By doing so, the tuning
parameters and the predefined linear operator $\bm{W}$ are now all learnable
from the training data. The proximal operator of the sparsity promoting
function $\Phi$ is parameterized by a piecewise linear function with learnable
parameters as well. As a result, the thresholding operator
$\mathcal{T}_{\lambda}(\cdot)$ in ADMM algorithm is also learned from the
training data. In a basic version of ADMM-Net [139], $\bm{d}^{k+1}$ is updated
by

$\bm{d}^{k+1}=\mathcal{T}_{\bm{\varTheta}_{1}}\left({\bm{\mathcal{W}}}_{\bm{\varTheta}_{2}}(\bm{u}^{k+1})+\bm{b}^{k}\right),$
(4.1)

where $\mathcal{T}_{\bm{\varTheta}_{1}}(\cdot)$ is a parameterized piecewise
linear function with parameters $\bm{\varTheta}_{1}$, and
${\bm{\mathcal{W}}}_{\bm{\varTheta}_{2}}$ is a parameterized convolution layer
with parameters $\bm{\varTheta}_{2}$. The ADMM-Net was later further improved
by [159] and the new model was called the Generic-ADMM-Net where different
variable splitting strategy was adopted in the derivation of the ADMM
algorithm. The Generic-ADMM-Net achieved state-of-the-art MR image
reconstruction results with a significant margin over the BM3D-based
algorithm.

#### Primal-Dual Networks (PD-Net)

In [2], the authors unrolled the iteration scheme (1.2.3) and (1.11) of the
PDHG algorithm to design new deep model for CT image reconstruction. This new
deep model was called the primal-dual network (PD-Net). The main idea is to
approximate each resolvent/proximal operator [118] in the subproblem of PDHG
by a neural network. Thus, it circumvents the difficulties in choosing optimal
forms $\Phi$ and $F$. One layer of PD-Net takes the form

$\displaystyle\bm{w}^{k+1}$ $\displaystyle=$
$\displaystyle{\bm{\mathcal{N}}}_{\bm{w}}\left([\bm{w}^{k},\bm{W}\bm{u}^{k}];\bm{\Theta}_{\bm{w}}^{k}\right),$
$\displaystyle\bm{u}^{k+1}$ $\displaystyle=$
$\displaystyle{\bm{\mathcal{N}}}_{\bm{u}}\left([\bm{u}^{k},\bm{W}^{\top}\bm{w}^{k+1},\bm{A}^{\top}\bm{f}];\bm{\Theta}_{w}^{k}\right),$
(4.2)

where $\bm{f}$ is the measured data, $\bm{A}$ is the imaging operator,
${\bm{\mathcal{N}}}_{\bm{w}}(\cdot;\bm{\Theta}_{\bm{w}}^{k})$ and
${\bm{\mathcal{N}}}_{\bm{u}}(\cdot;\bm{\Theta}_{\bm{u}}^{k})$ are neural
networks parameterized by $\bm{\Theta}_{\bm{w}}^{k}$ and
$\bm{\Theta}_{\bm{u}}^{k}$ respectively. The notation
$[\bm{v}_{1},\cdots,\bm{v}_{m}]$ denotes concatenation of the components
$\bm{v}_{1},\ldots,\bm{v}_{m}$ into a higher dimension tensor. The linear
operator $\bm{W}$ can be either fixed or learned from the data. PD-Net has a
significant performance boost compared with FBP and some handcrafted
reconstruction models [2, 1].

#### JSR-Net

To suppress the artifacts induced by incomplete data and noise, [50] proposed
a joint spatial-Radon domain reconstruction (JSR) model for sparse view CT
imaging as following

$\min_{\bm{u},\bm{f}}{\mathcal{F}}(\bm{u},\bm{f},\bm{Y})+\mathcal{R}(\bm{u},\bm{f}),$
(4.3)

where the data fidelity term ${\mathcal{F}}(\bm{u},\bm{f},\bm{Y})$ is defined
by

${\mathcal{F}}(\bm{u},\bm{f},\bm{Y})=\frac{1}{2}\|R_{\bm{\Gamma}^{c}}(\bm{f}-\bm{Y})\|^{2}+\frac{\alpha}{2}\|R_{\bm{\Gamma}}(\bm{A}\bm{u}-\bm{f})\|^{2}+\frac{\gamma}{2}\|R_{\bm{\Gamma}^{c}}(\bm{A}\bm{u}-\bm{Y})\|^{2},$

and the regularization term defined by

$\mathcal{R}(\bm{u},\bm{f})=\|\bm{\lambda}_{1}\cdot\bm{W}_{1}\bm{u}\|_{1,2}+\|\bm{\lambda}_{2}\cdot\bm{W}_{2}\bm{f}\|_{1,2}.$

The notation $R_{\bm{\Gamma}}$ is a restriction operator with respect to the
missing data region indexed by $\bm{\Gamma}$. $R_{\bm{\Gamma}}$ takes value
$1$ if the element’s index contained in $\bm{\Gamma}$ and $0$ elsewhere. Here,
$\bm{\Gamma}^{c}$ indicates the region of available measured data and is the
complement of $\bm{\Gamma}$. $\bm{A}$ is a discrete form of the Radon
transform, $\bm{Y}$ is the measured projection data. Note that, in JSR model
$\bm{u}$ and $\bm{f}$ are the underlying CT image and the restored high-
quality projection data respectively. $\bm{W}_{i},i=1,2$, are tight frame
transforms and $\bm{\lambda}_{i},i=1,2$, are the regularization parameters.

The handcrafted JSR model (4.3) enforces the data consistency in the Radon
domain and image domain simultaneously. Thus, it leads to improved quality of
the reconstructed image. Similar data fidelity design was adopted in [22] to
model the positron emission tomography. Later, [165] propose to improve the
JSR model by learning the tight frame transforms $\bm{W}_{i}$ from the data.
More recently, a re-weighting strategy was introduced in JSR model to reduce
the metal artifacts in multi-chromatic energy CT [167].

Existing work showed the potential of the JSR framework, and it is natural to
consider using unrolling to derive a deep model from algorithms solving the
JSR model. In [168], the authors designed the JSR-Net for sparse view and
limited angle CT image reconstruction. The JSR-Net is derived by unrolling an
alternative optimization algorithm with subproblems solved by ADMM. Similar to
the PD-Net, JSR-Net also adopted neural networks to approximate the proximal
operators. The advantage of JSR-Net is that it can efficiently utilize multi-
domain image features to improve the quality of the reconstructed image.

###  4.3 Raw-to-Task

(a) Image reconstruction

(b) Image analysis

(c) Image reconstruction and analysis

Figure 6:  CNN based workflows for medical image reconstruction and analysis.

The traditional workflow of medical image analysis has two separate stages: 1)
reconstruction of a high-quality image from raw data (see Figure 6(a)), and 2)
make a diagnosis based on the high-quality reconstructed image (see Figure
6(b)). The drawbacks of the two-stages’ approach and the potential benefit of
uniting the two stages were discussed earlier. Here, we shall describe how we
can join the two stages into one unified step (see Figure 6(c)).

As discussed in Section 4.2 that we can design feed-forward deep networks for
image reconstruction. Once we have an image, there are plenty of choices of
deep neural networks for various image analysis tasks. The most simple and
natural way of joining image reconstruction and image analysis is to connect
the two networks together and conduct end-to-end training (from scratch or by
fine-tuning). Such idea was first introduced by [155] in medical imaging and
by [94, 93] in computer vision for image denoising. By doing so, the second
network for image analysis can be regarded as a task-based image quality
metric that is learned from the data. As shown in [155], where the image
analysis task was lung nodule recognition, the learned image quality metric
automatically placed more emphasis within the lung areas and less emphasis
elsewhere. Such a quality metric is specific to the task of lung nodule
recognition since the image quality outside of the lung region is irrelevant
to the task.

##  5 Challenge and Opportunities

Although deep learning based models continue to dominant medical imaging,
there are still plenty of remaining challenges in deep modeling which limit
the application and implementation of these new methods in clinical practice.
These challenges also present themselves as new opportunities for researchers
working in related fields.

  * •

The everlasting hunger of labeled data. There are only limited labeled data
available to develop new deep models in medical imaging. Annotation of medical
images is time-consuming and requires expert knowledge from physicians. Can we
design effective learning models that can make good use of both the (very
limited) labeled data and the (relatively more abundant) unlabelled data?

  * •

The limited number of observations. Due to morbidity and privacy concerns, it
is generally difficult to gather very large medical data for a specific task.
Furthermore, the number of rare cases is (by definition) small but can be much
more valuable than common cases. Can we design learning models and data
augmentation techniques to effectively extract knowledge from these limited
samples and acknowledge such unequal importance among the samples?

  * •

Radiologists do not make the clinical decision only based on images. More
information from the patients and the knowledge of the doctors from their
years of training in medical school are also crucial in decision making. Thus,
incorporating data gathered from multiple diverse sources into deep modeling
is important in improving system performance.

  * •

Reasoning is just as important as, if not more important than, inferencing.
Currently, most deep models hide the reasoning procedure. There is a chance
that the model makes accurate predictions based on wrong reasoning. This makes
the model unreliable. Can we incorporate deep modeling with reasoning (such as
causal inference) or with medical knowledge graph? This may further reduce the
amount of annotated data we need to train deep models without hurting
performance.

## References

  * [1] Jonas Adler and Ozan Öktem.  Solving ill-posed inverse problems using iterative deep neural networks.  Inverse Probl., 33:124007 (24pp), 2017.
  * [2] Jonas Adler and Ozan Öktem.  Learned primal-dual reconstruction.  IEEE Trans. Med. Imaging, 37(6):1322–1332, 2018.
  * [3] Michal Aharon, Michael Elad, Alfred Bruckstein, et al.  K-svd: An algorithm for designing overcomplete dictionaries for sparse representation.  IEEE Trans. Signal Process., 54(11):4311, 2006.
  * [4] Luis Alvarez and Luis Mazorra.  Signal and image restoration using shock filters and anisotropic diffusion.  SIAM J. Numer. Anal., 31(2):590–605, 1994.
  * [5] Uri M Ascher and Linda R Petzold.  Computer methods for ordinary differential equations and differential-algebraic equations, volume 61.  Siam, 1998.
  * [6] Gilles Aubert and Pierre Kornprobst.  Mathematical problems in image processing: partial differential equations and the calculus of variations.  Springer, 2006.
  * [7] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE Trans. Pattern Anal. Mach. Intell., 39(12):2481–2495, 2017\.
  * [8] Chenglong Bao, Hui Ji, and Zuowei Shen.  Convergence analysis for iterative data-driven tight frame construction scheme.  Appl. Comput. Harmon. Anal., 38(3):510–523, 2015.
  * [9] Andrew R Barron.  Universal approximation bounds for superpositions of a sigmoidal function.  IEEE Trans. Inf. Theory, 39(3):930–945, 1993.
  * [10] Amir Beck and Marc Teboulle.  A fast iterative shrinkage-thresholding algorithm for linear inverse problems.  SIAM J. Imaging Sci., 2(1):183–202, 2009.
  * [11] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle.  Greedy layer-wise training of deep networks.  In NeurIPS, pages 153–160, 2007.
  * [12] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester.  Image inpainting.  In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 417–424. ACM Press/Addison-Wesley Publishing Co., 2000.
  * [13] Léon Bottou.  Large-scale machine learning with stochastic gradient descent.  In Proceedings of COMPSTAT, pages 177–186. Springer, 2010.
  * [14] Léon Bottou.  Stochastic gradient descent tricks.  In Neural networks: Tricks of the trade, pages 421–436. Springer, 2012.
  * [15] Léon Bottou, Frank E Curtis, and Jorge Nocedal.  Optimization methods for large-scale machine learning.  SIAM Rev., 60(2):223–311, 2018.
  * [16] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein.  Distributed optimization and statistical learning via the alternating direction method of multipliers.  Found. Trends Mach. Learn., 3(1):1–122, 2011.
  * [17] Robert W Brown, E Mark Haacke, Y-C Norman Cheng, Michael R Thompson, and Ramesh Venkatesan.  Magnetic resonance imaging: physical principles and sequence design.  John Wiley & Sons, 2014.
  * [18] Ronald E Bruck Jr.  On the weak convergence of an ergodic iteration for the solution of variational inequalities for monotone operators in hilbert space.  J. Math. Anal. Appl., 61(1):159–164, 1977.
  * [19] Antoni Buades, Bartomeu Coll, and J-M Morel.  A non-local algorithm for image denoising.  In CVPR, volume 2, pages 60–65, 2005.
  * [20] Antoni Buades, Bartomeu Coll, and Jean-Michel Morel.  A review of image denoising algorithms, with a new one.  Multiscale Model. Simul., 4(2):490–530, 2005.
  * [21] Antoni Buades, Bartomeu Coll, and Jean-Michel Morel.  Image denoising methods. a new nonlocal principle.  SIAM Rev., 52(1):113–147, 2010.
  * [22] Martin Burger, Jahn Müller, Evangelos Papoutsellis, and Carola-Bibiane Schönlieb.  Total variation regularization in measurement and image space for pet reconstruction.  Inverse Probl., 30(10):105003, 2014.
  * [23] Thorsten M Buzug.  Computed tomography: from photon statistics to modern cone-beam CT.  Springer Science & Business Media, 2008.
  * [24] Jian-Feng Cai, Emmanuel J Candès, and Zuowei Shen.  A singular value thresholding algorithm for matrix completion.  SIAM J. Optim., 20(4):1956–1982, 2010.
  * [25] Jian-Feng Cai, Bin Dong, and Zuowei Shen.  Image restoration: a wavelet frame based model for piecewise smooth functions and beyond.  Appl. Comput. Harmon. Anal., 41(1):94–138, 2016.
  * [26] Jian-Feng Cai, Hui Ji, Zuowei Shen, and Gui-Bo Ye.  Data-driven tight frame construction and image denoising.  Appl. Comput. Harmon. Anal., 37(1):89–105, 2014.
  * [27] Jian-Feng Cai, Xun Jia, Hao Gao, Steve B Jiang, Zuowei Shen, and Hongkai Zhao.  Cine cone beam ct reconstruction using low-rank matrix factorization: algorithm and a proof-of-principle study.  IEEE Trans. Med. Imaging, 33(8):1581–1591, 2014.
  * [28] Jian-Feng Cai, Stanley Osher, and Zuowei Shen.  Split bregman methods and frame based image restoration.  Multiscale Model. Simul., 8(2):337–369, 2009.
  * [29] Emmanuel J Candès and Benjamin Recht.  Exact matrix completion via convex optimization.  Found. Comput. Math., 9(6):717, 2009.
  * [30] Bruno Cessac.  A view of neural networks as dynamical systems.  Int. J. Bifurcation Chaos, 20(06):1585–1629, 2010.
  * [31] Antonin Chambolle and Thomas Pock.  A first-order primal-dual algorithm for convex problems with applications to imaging.  J. Math. Imaging Vis., 40(1):120–145, 2011.
  * [32] Tony F. Chan and Jianhong Shen.  Image processing and analysis: variational, PDE, wavelet, and stochastic methods.  SIAM, 2005.
  * [33] Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham.  Reversible architectures for arbitrarily deep residual neural networks.  In AAAI, 2018.
  * [34] Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert.  Multi-level residual networks from dynamical systems view.  In ICLR Poster, 2018.
  * [35] Hu Chen, Yi Zhang, Mannudeep K Kalra, Feng Lin, Yang Chen, Peixi Liao, Jiliu Zhou, and Ge Wang.  Low-dose ct with a residual encoder-decoder convolutional neural network.  IEEE Trans. Med. Imaging, 36(12):2524–2535, 2017.
  * [36] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  In NeurIPS, pages 6572–6583, 2018.
  * [37] Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin.  Theoretical linear convergence of unfolded ista and its practical weights and thresholds.  In NeurIPS, pages 9079–9089, 2018.
  * [38] Yunjin Chen, Wei Yu, and Thomas Pock.  On learning optimized reaction diffusion processes for effective image restoration.  In CVPR, pages 5261–5269, 2015.
  * [39] Jae Kyu Choi, Hyoung Suk Park, Shuai Wang, Yi Wang, and Jin Keun Seo.  Inverse problem in quantitative susceptibility mapping.  SIAM J. Imaging Sci., 7(3):1669–1689, 2014.
  * [40] Nadav Cohen, Or Sharir, and Amnon Shashua.  On the expressive power of deep learning: A tensor analysis.  In COLT, pages 698–728, 2016.
  * [41] George Cybenko.  Approximation by superpositions of a sigmoidal function.  Mathematics of control, signals and systems, 2(4):303–314, 1989\.
  * [42] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Image denoising by sparse 3-d transform-domain collaborative filtering.  IEEE Trans. Image Process., 16(8):2080–2095, 2007.
  * [43] Ingrid Daubechies.  Ten lectures on wavelets.  SIAM, 1992.
  * [44] Ingrid Daubechies, Michel Defrise, and Christine De Mol.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint.  Commun. Pure Appl. Math., 57(11):1413–1457, 2004.
  * [45] Ludovic de Rochefort, Tian Liu, Bryan Kressler, Jing Liu, Pascal Spincemaille, Vincent Lebon, Jianlin Wu, and Yi Wang.  Quantitative susceptibility map reconstruction from mr phase data using bayesian regularization: validation and application to brain imaging.  Magn. Reson. Med., 63(1):194–206, 2010.
  * [46] Olivier Delalleau and Yoshua Bengio.  Shallow vs. deep sum-product networks.  In NeurIPS, pages 666–674, 2011.
  * [47] R.A. DeVore and G.G. Lorentz.  Constructive approximation.  Springer, 1993.
  * [48] B. Dong and Z. Shen.  MRA-Based Wavelet Frames and Applications.  IAS Lecture Notes Series, Hong-Kai, ed. ”Mathematics in Image Processing”. Vol. 19, American Mathematical Society, 2013.
  * [49] Bin Dong, Qingtang Jiang, and Zuowei Shen.  Image restoration: wavelet frame shrinkage, nonlinear evolution pdes, and beyond.  Multiscale Model. Simul., 15(1):606–660, 2017.
  * [50] Bin Dong, Jia Li, and Zuowei Shen.  X-ray ct image reconstruction via wavelet frame based regularization and radon domain inpainting.  J. Sci. Comput., 54(2):333–349, 2013.
  * [51] Bin Dong and Zuowei Shen.  Image restoration: a data-driven perspective.  In Proceedings of the International Congress of Industrial and Applied Mathematics (ICIAM), pages 65–108, 2015.
  * [52] John Duchi, Elad Hazan, and Yoram Singer.  Adaptive subgradient methods for online learning and stochastic optimization.  J. Mach. Learn. Res., 12(Jul):2121–2159, 2011.
  * [53] Weinan E, Chao Ma, and Qingcan Wang.  A priori estimates of the population risk for residual networks.  arXiv preprint arXiv:1903.02154, 2019.
  * [54] Ronen Eldan and Ohad Shamir.  The power of depth for feedforward neural networks.  In COLT, pages 907–940, 2016.
  * [55] Kjersti Engan, Sven Ole Aase, and J Hakon Husoy.  Method of optimal directions for frame design.  In IEEE International Conference on Acoustics, Speech, and Signal Processing(ICASSP), volume 5, pages 2443–2446. IEEE, 1999.
  * [56] Ernie Esser, Xiaoqun Zhang, and Tony F Chan.  A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science.  SIAM J. Imaging Sci., 3(4):1015–1046, 2010.
  * [57] Thorsten Falk, Dominic Mai, Robert Bensch, Özgün Çiçek, Ahmed Abdulkadir, Yassine Marrakchi, Anton Böhm, Jan Deubner, Zoe Jäckel, Katharina Seiwald, et al.  U-net: deep learning for cell counting, detection, and morphometry.  Nat. Methods, 16:67–70, 2019.
  * [58] Ken-Ichi Funahashi.  On the approximate realization of continuous mappings by neural networks.  Neural Netw., 2(3):183–192, 1989.
  * [59] Daniel Gabay and Bertrand Mercier.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation.  Comput. Math. Appl., 2(1):17–40, 1976.
  * [60] Xavier Gastaldi.  Shake-shake regularization.  In ICLR workshop, 2017.
  * [61] Roland Glowinski and A Marroco.  Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de dirichlet non linéaires.  Revue française d’automatique, informatique, recherche opérationnelle. Analyse numérique, 9(R2):41–76, 1975.
  * [62] Tom Goldstein and Stanley Osher.  The split bregman method for $l_{1}$-regularized problems.  SIAM J. Imaging Sci., 2(2):323–343, 2009.
  * [63] Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse.  The reversible residual network: Backpropagation without storing activations.  In NeurIPS, pages 2214–2224, 2017.
  * [64] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, pages 2672–2680, 2014.
  * [65] Karol Gregor and Yann LeCun.  Learning fast approximations of sparse coding.  In ICML, pages 399–406. Omnipress, 2010.
  * [66] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng.  Weighted nuclear norm minimization with application to image denoising.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2862–2869, 2014.
  * [67] Yo Seob Han, Jaejun Yoo, and Jong Chul Ye.  Deep residual learning for compressed sensing ct reconstruction via persistent homology analysis.  arXiv preprint arXiv:1611.06391, 2016.
  * [68] Boris Hanin.  Universal function approximation by deep neural nets with bounded width and relu activations.  arXiv preprint arXiv:1708.02691, 2017.
  * [69] Boris Hanin and Mark Sellke.  Approximating continuous functions by relu nets of minimal width.  arXiv preprint arXiv:1710.11278, 2017.
  * [70] Juncai He, Lin Li, Jinchao Xu, and Chunyue Zheng.  Relu deep neural networks and linear finite elements.  arXiv preprint arXiv:1807.03973, 2018.
  * [71] Juncai He and Jinchao Xu.  Mgnet: A unified framework of multigrid and convolutional neural network.  arXiv preprint arXiv:1901.10415, 2019.
  * [72] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [73] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In ECCV, pages 630–645, 2016.
  * [74] Tobias Heimann and Hans-Peter Meinzer.  Statistical shape models for 3d medical image segmentation: a review.  Med. Image Anal., 13(4):543–563, 2009.
  * [75] Gabor T Herman.  Fundamentals of computerized tomography: image reconstruction from projections.  Springer Science & Business Media, 2009.
  * [76] Geoffrey Hinton.  Neural networks for machine learning.  Coursera, video lectures, 2012.
  * [77] Kurt Hornik.  Approximation capabilities of multilayer feedforward networks.  Neural Netw., 4(2):251–257, 1991.
  * [78] Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  Neural Netw., 2(5):359–366, 1989.
  * [79] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In CVPR, volume 1, pages 4700–4708, 2017.
  * [80] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger.  Deep networks with stochastic depth.  In ECCV, pages 646–661, 2016.
  * [81] Kyong Hwan Jin, Michael T. Mccann, Emmanuel Froustey, and Michael Unser.  Deep convolutional neural network for inverse problems in imaging.  IEEE Trans. Image Process., 26(9):4509–4522, 2017.
  * [82] Mannudeep Kalra, Ge Wang, and Colin G Orton.  Radiomics in lung cancer: Its time is here.  Med. Phys., 45(3):997–1000, 2018.
  * [83] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [84] Jakub Konečnỳ, Jie Liu, Peter Richtárik, and Martin Takáč.  Mini-batch semi-stochastic gradient descent in the proximal setting.  IEEE J. Sel. Top. Signal Process., 10(2):242–255, 2016.
  * [85] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS, pages 1097–1105, 2012.
  * [86] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich.  Fractalnet: Ultra-deep neural networks without residuals.  In ICLR, 2016.
  * [87] Huan Li, Yibo Yang, Dongmin Chen, and Zhouchen Lin.  Optimization algorithm inspired deep neural network structure design.  In Jun Zhu and Ichiro Takeuchi, editors, Proceedings of The 10th Asian Conference on Machine Learning, volume 95 of Proceedings of Machine Learning Research, pages 614–629. PMLR, 14–16 Nov 2018.
  * [88] Huan Li, Yibo Yang, Dongmin Chen, and Zhouchen Lin.  Optimization algorithm inspired deep neural network structure design.  arXiv preprint arXiv:1810.01638, 2018.
  * [89] Qianxiao Li, Long Chen, Cheng Tai, and E Weinan.  Maximum principle based algorithms for deep learning.  J. Mach. Learn. Res., 18(1):5998–6026, 2017.
  * [90] Zhen Li and Zuoqiang Shi.  Deep residual learning and pdes on manifold.  arXiv preprint arXiv:1708.05115, 2017.
  * [91] Shiyu Liang and R Srikant.  Why deep neural networks for function approximation?  In ICLR, 2017.
  * [92] Hongzhou Lin and Stefanie Jegelka.  Resnet with one-neuron hidden layers is a universal approximator.  In NeurIPS, pages 6172–6181, 2018.
  * [93] Ding Liu, Bihan Wen, Jianbo Jiao, Xianming Liu, Zhangyang Wang, and Thomas S Huang.  Connecting image denoising and high-level vision tasks via deep learning.  arXiv preprint arXiv:1809.01826, 2018.
  * [94] Ding Liu, Bihan Wen, Xianming Liu, Zhangyang Wang, and Thomas S Huang.  When image denoising meets high-level vision tasks: A deep learning approach.  arXiv preprint arXiv:1706.04284, 2017.
  * [95] Guangcan Liu, Zhouchen Lin, Shuicheng Yan, Ju Sun, Yong Yu, and Yi Ma.  Robust recovery of subspace structures by low-rank representation.  IEEE Trans. Pattern Anal. Mach. Intell., 35(1):171–184, 2013.
  * [96] Jialin Liu, Xiaohan Chen, Zhangyang Wang, and Wotao Yin.  Alista: Analytic weights are as good as learned weights in lista.  In ICLR, 2019.
  * [97] Risheng Liu, Shichao Cheng, Yi He, Xin Fan, Zhouchen Lin, and Zhongxuan Luo.  On the convergence of learning-based iterative methods for nonconvex inverse problems.  arXiv preprint arXiv:1808.05331, 2018.
  * [98] Risheng Liu, Zhouchen Lin, Wei Zhang, and Zhixun Su.  Learning pdes for image restoration via optimal control.  In ECCV, pages 115–128. Springer, 2010.
  * [99] Zichao Long, Yiping Lu, and Bin Dong.  Pde-net 2.0: Learning pdes from data with a numeric-symbolic hybrid deep network.  arXiv preprint arXiv:1812.04426, 2018.
  * [100] Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong.  Pde-net: Learning pdes from data.  In ICML Workshop, volume 80, pages 3208–3216, 2018.
  * [101] Yifei Lou, Xiaoqun Zhang, Stanley Osher, and Andrea Bertozzi.  Image recovery via nonlocal operators.  J. Sci. Comput., 42(2):185–197, 2010.
  * [102] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu.  Understanding and improving transformer from a multi-particle dynamic system point of view.  arXiv preprint arXiv:1906.02762, 2019.
  * [103] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong.  Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations.  In ICML, pages 3276–3285, 2018.
  * [104] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang.  The expressive power of neural networks: A view from the width.  In NeurIPS, pages 6231–6239, 2017.
  * [105] Stéphane Mallat.  A Wavelet Tour of Signal Processing, The Sparse Way.  Academic Press, Burlington,MA, third edition, 2009.
  * [106] Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang.  Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections.  In NeurIPS, pages 2802–2810, 2016.
  * [107] Hrushikesh Mhaskar, Qianli Liao, and Tomaso Poggio.  Learning functions: when is deep better than shallow.  arXiv preprint arXiv:1603.00988, 2016.
  * [108] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In International Conference on 3D Vision (3DV), pages 565–571. IEEE, 2016.
  * [109] David Mumford and Jayant Shah.  Optimal approximations by piecewise smooth functions and associated variational problems.  Commun. Pure Appl. Math., 42(5):577–685, 1989.
  * [110] Frank Natterer.  The mathematics of computerized tomography.  SIAM, 2001.
  * [111] Frank Natterer.  Image reconstruction in quantitative susceptibility mapping.  SIAM J. Imaging Sci., 9(3):1127–1131, 2016.
  * [112] Yurii E Nesterov.  A method for solving the convex programming problem with convergence rate $O(1/k^{2})$.  In Dokl. Akad. Nauk SSSR, volume 269, pages 543–547, 1983.
  * [113] Atsushi Nitanda.  Stochastic proximal gradient descent with acceleration techniques.  In NeurIPS, pages 1574–1582, 2014.
  * [114] Jorge Nocedal and Stephen J. Wright.  Numerical Optimization.  Springer, 2 edition, 2006.
  * [115] Ricardo H Nochetto and Andreas Veeser.  Primer of adaptive finite element methods.  In Multiscale and adaptivity: modeling, numerics and applications, pages 125–225. Springer, 2011.
  * [116] Stanley Osher, Yu Mao, Bin Dong, and Wotao Yin.  Fast linearized bregman iteration for compressive sensing and sparse denoising.  Commun. Math. Sci., 8(1):93–111, 2010.
  * [117] Stanley Osher and Leonid I Rudin.  Feature-oriented image enhancement using shock filters.  SIAM J. Numer. Anal., 27(4):919–940, 1990.
  * [118] Neal Parikh, Stephen Boyd, et al.  Proximal algorithms.  Foundations and Trends® in Optimization, 1(3):127–239, 2014.
  * [119] Gregory B Passty.  Ergodic convergence to a zero of the sum of monotone operators in hilbert space.  J. Math. Anal. Appl., 72:383–290, 1979.
  * [120] Gordana Pavlovic and A Murat Tekalp.  Maximum likelihood parametric blur identification based on a continuous spatial domain model.  IEEE Transactions on image processing, 1(4):496–504, 1992.
  * [121] Pietro Perona and Jitendra Malik.  Scale-space and edge detection using anisotropic diffusion.  IEEE Trans. Pattern Anal. Mach. Intell., 12(7):629–639, 1990.
  * [122] Pietro Perona, Takahiro Shiota, and Jitendra Malik.  Anisotropic diffusion.  In Geometry-driven diffusion in computer vision, pages 73–92. Springer, 1994.
  * [123] Allan Pinkus.  Approximation theory of the mlp model in neural networks.  Acta Numer., 8:143–195, 1999.
  * [124] Christopher Poultney, Sumit Chopra, Yann L Cun, et al.  Efficient learning of sparse representations with an energy-based model.  In NeurIPS, pages 1137–1144, 2007.
  * [125] Herbert Robbins and Sutton Monro.  A stochastic approximation method.  Ann. Math. Statist., 22(3):400–407, 1951.
  * [126] David Rolnick and Max Tegmark.  The power of deeper networks for expressing natural functions.  In ICLR, 2018.
  * [127] Amos Ron and Zuowei Shen.  Affine systems in $l_{2}(\mathbb{R}^{d})$: The analysis of the analysis operator.  J. Funct. Anal., 148(2):408–447, 1997.
  * [128] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention(MICCAI), pages 234–241, 2015.
  * [129] Leonid Rudin, Pierre-Luis Lions, and Stanley Osher.  Multiplicative denoising and deblurring: theory and algorithms.  In Geometric Level Set Methods in Imaging, Vision, and Graphics, pages 103–119. Springer, 2003.
  * [130] Leonid I Rudin, Stanley Osher, and Emad Fatemi.  Nonlinear total variation based noise removal algorithms.  Physica D, 60(1):259–268, 1992.
  * [131] Lars Ruthotto and Eldad Haber.  Deep neural networks motivated by partial differential equations.  arXiv preprint arXiv:1804.04272, 2018.
  * [132] Otmar Scherzer, editor.  Handbook of mathematical methods in imaging.  Springer New York, second edition, 2015.
  * [133] Zuowei Shen.  Wavelet frames and image restorations.  In Proceedings of the International congress of Mathematicians, volume 4, pages 2834–2863. World Scientific, 2010.
  * [134] Zuowei Shen, Kim Chuan Toh, and Sangwoon Yun.  An accelerated proximal gradient algorithm for frame-based image restoration via the balanced approach.  SIAM J. Imaging Sci., 4(2):573–596, 2011.
  * [135] Zuowei Shen, Haizhao Yang, and Shijun Zhang.  Nonlinear approximation via compositions.  arXiv preprint arXiv:1902.10170, 2019.
  * [136] Oren Solomon, Regev Cohen, Yi Zhang, Yi Yang, He Qiong, Jianwen Luo, Ruud JG van Sloun, and Yonina C Eldar.  Deep unfolded robust pca with application to clutter suppression in ultrasound.  arXiv preprint arXiv:1811.08252, 2018.
  * [137] Sho Sonoda and Noboru Murata.  Double continuum limit of deep neural networks.  In ICML Workshop on Principled Approaches to Deep Learning, 2017\.
  * [138] Weijie Su, Stephen Boyd, and Emmanuel Candès.  A differential equation for modeling nesterov’s accelerated gradient method: Theory and insights.  In NeurIPS, pages 2510–2518, 2014.
  * [139] Jian Sun, Huibin Li, Zongben Xu, et al.  Deep admm-net for compressive sensing mri.  In NeurIPS, pages 10–18, 2016.
  * [140] Qi Sun, Yunzhe Tao, and Qiang Du.  Stochastic training of residual networks: a differential equation viewpoint.  arXiv preprint arXiv:1812.00174, 2018.
  * [141] Cheng Tai and E Weinan.  Multiscale adaptive representation of signals: I. the basic framework.  J. Mach. Learn. Res., 17(1):4875–4912, 2016.
  * [142] Yunzhe Tao, Qi Sun, Qiang Du, and Wei Liu.  Nonlocal neural networks, nonlocal diffusion and nonlocal modeling.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, NeurIPS, pages 494–504. Curran Associates, Inc., 2018.
  * [143] Matus Telgarsky.  Representation benefits of deep feedforward networks.  arXiv preprint arXiv:1509.08101, 2015.
  * [144] Matus Telgarsky.  Benefits of depth in neural networks.  In COLT, volume 49, pages 1–23, 2016.
  * [145] Matthew Thorpe and Yves van Gennip.  Deep limits of residual neural networks.  arXiv preprint arXiv:1810.11741, 2018.
  * [146] Andreas Veit, Michael J Wilber, and Serge Belongie.  Residual networks behave like ensembles of relatively shallow networks.  In NeurIPS, pages 550–558, 2016.
  * [147] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre Antoine Manzagol.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  J. Mach. Learn. Res., 11(12):3371–3408, 2010.
  * [148] Bao Wang, Binjie Yuan, Zuoqiang Shi, and Stanley J Osher.  Enresnet: Resnet ensemble via the feynman-kac formalism.  arXiv preprint arXiv:1811.10745, 2018.
  * [149] Yi Wang and Tian Liu.  Quantitative susceptibility mapping (qsm): decoding mri data for a tissue magnetic biomarker.  Magn. Reson. Med., 73(1):82–101, 2015.
  * [150] RF Warming and BJ Hyett.  The modified equation approach to the stability and accuracy analysis of finite-difference methods.  J. Comput. Phys., 14(2):159–179, 1974.
  * [151] E Weinan.  A proposal on machine learning via dynamical systems.  Communications in Mathematics and Statistics, 5(1):1–11, 2017.
  * [152] E Weinan, Jiequn Han, and Qianxiao Li.  A mean-field optimal control formulation of deep learning.  Research in the Mathematical Sciences, 6(10):1–41, 2019.
  * [153] Ashia C Wilson, Benjamin Recht, and Michael I Jordan.  A lyapunov analysis of momentum methods in optimization.  arXiv preprint arXiv:1611.02635, 2016.
  * [154] John Wright, Arvind Ganesh, Shankar Rao, Yigang Peng, and Yi Ma.  Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization.  In NeurIPS, pages 2080–2088, 2009.
  * [155] Dufan Wu, Kyungsang Kim, Bin Dong, Georges El Fakhri, and Quanzheng Li.  End-to-end lung nodule detection in computed tomography.  arXiv preprint arXiv:1711.02074, 2017.
  * [156] Xingyu Xie, Jianlong Wu, Zhisheng Zhong, Guangcan Liu, and Zhouchen Lin.  Differentiable linearized admm.  In ICML, 2019.
  * [157] Jinchao Xu and Juncai He.  MgNet: A Unified Framework of Multigrid and Convolutional Neural Network.  arXiv preprint arXiv:1901.10415, 2019.
  * [158] Qingsong Yang, Pingkun Yan, Yanbo Zhang, Hengyong Yu, Yongyi Shi, Xuanqin Mou, Mannudeep K Kalra, Yi Zhang, Ling Sun, and Ge Wang.  Low dose ct image denoising using a generative adversarial network with wasserstein distance and perceptual loss.  37(6):1348 – 1357, 2018.
  * [159] Yan Yang, Jian Sun, Huibin Li, and Zongben Xu.  Admm-net: A deep learning approach for compressive sensing mri.  arXiv preprint arXiv:1705.06869, 2017.
  * [160] Dmitry Yarotsky.  Optimal approximation of continuous functions by very deep relu networks.  In COLT, 2018.
  * [161] Jong Chul Ye, Yoseob Han, and Eunju Cha.  Deep convolutional framelets: A general deep learning framework for inverse problems.  SIAM J. Imaging Sci., 11(2):991–1048, 2018.
  * [162] Rujie Yin, Tingran Gao, Yue M Lu, and Ingrid Daubechies.  A tale of two bases: Local-nonlocal regularization on image patches with convolution framelets.  SIAM J. Imaging Sci., 10(2):711–750, 2017.
  * [163] Wotao Yin, Stanley Osher, Donald Goldfarb, and Jerome Darbon.  Bregman iterative algorithms for $\ell_{1}$-minimization with applications to compressed sensing.  SIAM J. Imaging Sci., 1(1):143–168, 2008.
  * [164] Gengsheng Lawrence Zeng.  Medical image reconstruction: a conceptual tutorial. Springer, 2010.
  * [165] Ruohan Zhan and Bin Dong.  Ct image reconstruction by spatial-radon domain data-driven tight frame regularization.  SIAM J. Imaging Sci., 9(3):1063–1083, 2016.
  * [166] Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong.  You only propagate once: Accelerating adversarial training via maximal principle.  arXiv preprint arXiv:1905.00877, 2019.
  * [167] Haimiao Zhang, Bin Dong, and Baodong Liu.  A reweighted joint spatial-radon domain ct image reconstruction model for metal artifact reduction.  SIAM J. Imaging Sci., 11(1):707–733, 2018.
  * [168] Haimiao Zhang, Bin Dong, and Baodong Liu.  Jsr-net: A deep network for joint spatial-radon domain ct reconstruction from incomplete data.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)-2019, pages 3657–3661, May 2019.
  * [169] Jingfeng Zhang, Bo Han, Laura Wynter, Kian Hsiang Low, and Mohan Kankanhalli.  Towards robust resnet: A small step but a giant leap.  arXiv preprint arXiv:1902.10887, 2019.
  * [170] Tong Zhang.  Solving large scale linear prediction problems using stochastic gradient descent algorithms.  In ICML, pages 116–123. ACM, 2004.
  * [171] Xiaoshuai Zhang, Yiping Lu, Jiaying Liu, and Bin Dong.  Dynamically unfolding recurrent restorer: A moving endpoint control method for image restoration.  In ICLR, 2019.
  * [172] Xingcheng Zhang, Zhizhong Li, Chen Change Loy, and Dahua Lin.  Polynet: A pursuit of structural diversity in very deep networks.  In CVPR, pages 3900–3908, 2017.
  * [173] Yuchen Zhang and Lin Xiao.  Stochastic primal-dual coordinate method for regularized empirical risk minimization.  J. Mach. Learn. Res., 18(1):2939–2980, 2017.
  * [174] Zhicheng Zhang, Xiaokun Liang, Xu Dong, Yaoqin Xie, and Guohua Cao.  A sparse-view ct reconstruction method based on combination of densenet and deconvolution.  IEEE Trans. Med. Imaging, 37(6):1407–1417, 2018.
  * [175] Bo Zhu, Jeremiah Z Liu, Stephen F Cauley, Bruce R Rosen, and Matthew S Rosen.  Image reconstruction by domain-transform manifold learning.  Nature, 555(7697):487, 2018.
  * [176] Mai Zhu, Bo Chang, and Chong Fu.  Convolutional neural networks combined with runge-kutta methods.  arXiv preprint arXiv:1802.08831, 2018.
  * [177] Mingqiang Zhu and Tony Chan.  An efficient primal-dual hybrid gradient algorithm for total variation image restoration.  UCLA CAM Report, 34, 2008.