# DONet: Dual-Octave Network for Fast MR Image Reconstruction

Chun-Mei Feng, Zhanyuan Yang, Huazhu Fu, , Yong Xu, ,
Jian Yang, and Ling Shao C.-M. Feng and Y. Xu are with the Shenzhen Key
Laboratory of Visual Object Detection and Recognition, Harbin Institute of
Technology (Shenzhen), 518055, China. (Email<EMAIL_ADDRESS>yongxu@ymail.com). (Corresponding author: Yong Xu)Z. Yang is with the School
of Automation Engineering, University of Electronic Science and Technology of
China, 611731, China. (Email: hawkyang0826@gmail.com)H. Fu and L. Shao are
with the Inception Institute of Artificial Intelligence, Abu Dhabi, UAE.
(Email<EMAIL_ADDRESS>ling.shao@ieee.org)J. Yang is with the PCA Laboratory,
Key Laboratory of Intelligent Perception and Systems for High-Dimensional
Information of Ministry of Education, Nanjiang University of Science and
Technology, Nanjiang 210094, China, and also with the Jiangsu Key Laboratory
of Image and Video Understanding for Social Security, School of Computer
Science and Engineering, Nanjing University of Science and Technology, Nanjing
210094, China (e-mail: csjyang@njust.edu.cn).This work was done during the
internship of Chun-Mei Feng at Inception Institute of Artificial Intelligence.

###### Abstract

Magnetic resonance (MR) image acquisition is an inherently prolonged process,
whose acceleration has long been the subject of research. This is commonly
achieved by obtaining multiple undersampled images, simultaneously, through
parallel imaging. In this paper, we propose the Dual-Octave Network (DONet),
which is capable of learning multi-scale spatial-frequency features from both
the real and imaginary components of MR data, for fast parallel MR image
reconstruction. More specifically, our DONet consists of a series of Dual-
Octave convolutions (Dual-OctConv), which are connected in a dense manner for
better reuse of features. In each Dual-OctConv, the input feature maps and
convolutional kernels are first split into two components (i.e., real and
imaginary), and then divided into four groups according to their spatial
frequencies. Then, our Dual-OctConv conducts intra-group information updating
and inter-group information exchange to aggregate the contextual information
across different groups. Our framework provides three appealing benefits: (i)
It encourages information interaction and fusion between the real and
imaginary components at various spatial frequencies to achieve richer
representational capacity. (ii) The dense connections between the real and
imaginary groups in each Dual-OctConv make the propagation of features more
efficient by feature reuse. (iii) DONet enlarges the receptive field by
learning multiple spatial-frequency features of both the real and imaginary
components. Extensive experiments on two popular datasets (i.e., clinical knee
and fastMRI), under different undersampling patterns and acceleration factors,
demonstrate the superiority of our model in accelerated parallel MR image
reconstruction.

###### Index Terms:

MR imaging, feature fusion, image reconstruction, complex-valued data

##  I Introduction

Magnetic resonance (MR) imaging has become increasingly popular in radiology
and medicine over the past decade, thanks to its advantages in being non-
radiative, having a high spatial-resolution, and providing superior soft
tissue contrast [1]. However, a major limitation of MR imaging is that it
requires a much longer acquisition time than other imaging techniques, e.g.,
computed tomography (CT), X-Ray, and ultrasound [2]. In addition, it is
impossible to keep patients in the scanner for a long time and obtain clean
data without motion artifacts. Therefore, accelerating MR reconstruction has
become an urgent research problem, since it can in turn greatly accelerate MR
imaging. Recently, significant efforts have been devoted to this task, which
is typically achieved by reconstructing the desired full images from
undersampled measured data [3].

Figure 1: Motivation of our work. Previous methods utilize vanilla
convolutions (a) to process the real- and imaginary-valued parts of an MR
image independently, or complex convolutions (b) to jointly deal with the two
parts. In contrast, we propose Dual-OctConv, which is a generalization of
complex convolutions, to process complex-valued inputs in a multi-frequency
space for more effective feature representations.

Compressed Sensing (CS) has achieved significant progress in fast MR imaging,
because sub-Nyquist sampling can significantly reduce the acquisition time by
skipping some phase information. Specifically, CS-based methods overcome
aliasing artifacts caused by the violation of the Shannon-Nyquist sampling
theorem, by introducing additional prior knowledge of the images. CS-based
methods employ sparse coding in the transformed domain (e.g., undersampled
$k$-space data) to naturally compress the MR image through a discrete cosine
transform (DCT) [4, 5], discrete Fourier transform (DFT) [6, 7], discrete
wavelet transform (DWT) [8, 9, 10], or dictionary learning [11]. However,
these traditional methods only exploit prior information of the images to-be-
reconstructed, or involve very few reference images. For a large number of
medical images acquired in clinical practice, traditional CS-based methods do
not explore the potential regularities within them. Because CS-based methods
require iterative optimization to find the optimal value, even if the CS
involves only a small number of reference images, they still suffer from heavy
computational overhead.

Recently, parallel MR imaging has been considered as one of the most important
achievements in accelerated MR imaging [12, 13, 14, 15]. Multi-coil data for
parallel imaging is composed of multiple physical receivers simultaneously
recorded from different perspectives. Parallel MR imaging reconstructs the
data points using the coil sensitivity profiles across multiple channels, from
a small amount of $k$-space data. At present, parallel imaging is the default
option for many scanning protocols. Most studies (e.g., SENSE [16], GRAPPA
[17, 18], SPIRiT [19]) take advantage of spatial sensitivity and gradient
coding to reduce the amount of data required for reconstruction, thereby
shortening the imaging time. However, a higher acceleration rate may introduce
aliased artifacts and substantially reduce the signal-to-noise ratio (SNR) in
clinical practice.

More recently, with the renaissance of deep neural networks, deep learning
techniques, especially convolutional neural networks (CNNs), have been widely
used for parallel MR imaging [20, 21]. Since models are trained offline over
large-scale data, only a few extra online samples are required for
reconstruction. The model-based unrolling methods [22, 23] combine
mathematical structures (e.g., variational inference, compressed sensing) with
deep learning for fast MR image reconstruction. Moreover, extensive approaches
[24, 25, 26, 27, 2] introduce end-to-end learnable models to remove the
aliasing artifacts from images that are reconstructed from undersampled multi-
coil $k$-space data. The mapping between a zero-filled $k$-space and fully-
sampled MR image is automatically learned by CNNs, requiring no sub-problem
division.

Most of the above approaches directly borrow vanilla convolutions used in
standard CNNs for $k$-space data in MR image reconstruction. However, vanilla
convolutions are designed for real-valued natural images, and cannot deal with
complex-valued inputs. To solve this, early studies [28] simply discarded the
imaginary part or processed the real and imaginary parts independently for
real-valued convolutions (see Fig. 1(a)). To avoid information loss, the
complex convolution [29] was recently proposed to process complex-valued
inputs and encourage information exchange between real and imaginary values
(see Fig. 1(b)). Though impressive, existing complex convolution operations
ignore the intrinsic multi-frequency property of MR images, leading to limited
single-scale contextual information and high spatial redundancy in the final
representations.

To address these limitations, we take a further step towards exploring multi-
frequency representation learning in parallel MR image reconstruction (see
Fig. 1(c)). Specifically, we propose a novel Dual-Octave Network (DONet)
which contains a series of Dual-Octave Convolutions (Dual-OctConv), enabling
our model to learn multi-frequency representations of multi-coil MR images
[30]. The dense connections in the Dual-OctConv enable the feature propagation
of the high- and low-frequency components of both the real and imaginary data.
Unlike complex convolutions, our DONet processes the real (or imaginary) part
of MR image features by factorizing it into high- and low-frequency
components. The low-frequency component shares information across neighboring
locations, and can thus be efficiently processed in low-resolution to enlarge
the receptive field and reduce the spatial redundancy. Finally, we combine the
features of the real and imaginary parts for reconstruction. Benefiting from
the Dual-OctConv, our network has a more powerful capability in multi-scale
representation learning, and can thus better capture soft tissues (e.g.,blood
vessels, muscles) with varying sizes and shapes.

Our main contributions are three-fold: First, we propose a novel DONet
containing multiple Dual-OctConvs connected in a dense manner for multi-
frequency feature reuse, and demonstrate their ability to capture multi-scale
contextual information. Second, we employ the Dual-OctConv to deal with
complex-valued inputs in a multi-frequency representation space, and encourage
information exchange and fusion across various frequency domains. The Dual-
OctConv is a generalization of the standard complex convolution, and endows
our model several appealing characteristics (e.g.,a larger receptive field,
higher flexibility, and more computational efficiency). Third, our model shows
significant performance improvements against state-of-the-art algorithms on a
clinical knee dataset and the fastMRI dataset.

Figure 2: Architecture of our DONet for parallel MR image reconstruction. The
input is a set of zero-filled multi-coil $k$-space measurements, while the
output is the reconstructed multi-channel MR image. $\rm IFT$ represents the
2D inverse Fourier transform. See §III-B for details.

##  II Related Work

###  II-A Deep Learning in MR Image Reconstruction

Ever since the pioneering works introducing CNNs for computer vision tasks,
such as image classification and face recognition, researchers have made
substantial efforts to improve medical and clinical practice using deep
learning techniques. Wang et al. [28] proposed the first deep learning based
MR image reconstruction framework, which learns the mapping between fully-
sampled single-coil MR images and their counterpart data reconstructed from a
zero-filled undersampled $k$-space. A large number of networks have since been
developed for MR image reconstruction, especially non-parallel reconstruction
[31]. For example, [32] proposed a model-based unrolling method, which
formulates the algorithm within a deep neural network, and trained the network
with a small amount of data. As an end-to-end method, [33] employed U-Net to
model a domain adaptation structure that removes aliasing artifacts from
corrupted images. Zhu et al. [34] proposed the automated transform by manifold
approximation (AUTOMAP) framework, which they implemented using a deep neural
network to learn a mapping between the sensor and image domain. Han et al.
[33] proposed a deep learning approach with domain adaptation which uses a
large amount of X‐ray computed tomography (CT) or synthesized radial MR data
and is fine‐tuned with a small amount of real radial MR data. Dongwook et al.
[35] trained amplitude and phase, respectively, using a residual network to
reduce the computational cost of the reconstruction algorithm. With the
pioneering works on Generative Adversarial Networks (GANs) for natural image
synthesis, GAN-based methods have also made great progress in MR image
reconstruction [36, 37, 38]. For example, Tran et al. [36] designed an
adversarial model for CS-MR image reconstruction, accurately interpolating and
restoring data in an undersampled $k$-space data, and with a loss of data
consistency. Moreover, image refinement on specific network structures with
different training objectives, such as the adversarial loss [37] or perceptual
loss [36], has gained more attention,

In parallel imaging, one representative model is the variational network (VN-
Net) [22], which combines the mathematical structure of the variational model
with deep learning for fast multi-coil MR image reconstruction. Another model-
based deep framework [23] was designed with a split Bregman iterative
algorithm to achieve accurate reconstruction from multi-coil undersampled
$k$-space data. To obtain high-fidelity reconstructions, GrappaNet [27] was
proposed to combine traditional parallel imaging methods with deep neural
networks. Recently, complex-valued representations have demonstrated
superiority in processing complex-valued inputs [29]. For example, [2] applied
complex convolutions to jointly process real and imaginary values for
comprehensive feature representations. In contrast, our approach represents
complex-valued input features in a multi-frequency space. The Dual-OctConv,
proposed for processing such multi-frequency data, can capture richer
contextual knowledge, leading to significant improvement in performance.

$h$$w$$c$$h/2$$w/2$$ac_{in}$$(1-a)c_{in}$$\mathbf{X}$$\check{\mathbf{X}}$$\mathbf{X}_{\text{i}(0)}^{\text{L}}$$\mathbf{X}_{\text{r}(0)}^{\text{L}}$$\mathbf{X}_{\text{i}(0)}^{\text{H}}$$\mathbf{X}_{\text{r}(0)}^{\text{H}}$$\mathbf{X}_{\text{i}(1)}^{\text{L}}$$\mathbf{X}_{\text{r}(1)}^{\text{L}}$$\mathbf{X}_{\text{i}(1)}^{\text{H}}$$\mathbf{X}_{\text{r}(1)}^{\text{H}}$$\mathbf{X}_{\text{i}(2)}^{\text{L}}$$\mathbf{X}_{\text{r}(2)}^{\text{L}}$$\mathbf{X}_{\text{i}(2)}^{\text{H}}$$\mathbf{X}_{\text{r}(2)}^{\text{H}}$$\mathbf{X}_{\text{i}(3)}^{\text{L}}$$\mathbf{X}_{\text{r}(3)}^{\text{L}}$$\mathbf{X}_{\text{i}(3)}^{\text{H}}$$\mathbf{X}_{\text{r}(3)}^{\text{H}}$$\mathbf{X}_{\text{i}(K)}^{\text{L}}$$\mathbf{X}_{\text{r}(K)}^{\text{L}}$$\mathbf{X}_{\text{i}(K)}^{\text{H}}$$\mathbf{X}_{\text{r}(K)}^{\text{H}}$$\mathbf{X}_{i(2)}^{\text{L}}$$\mathbf{X}_{r(2)}^{\text{L}}$$\mathbf{X}_{i(2)}^{\text{H}}$$\mathbf{X}_{r(2)}^{\text{H}}$$\mathbf{X}_{i(3)}^{\text{L}}$$\mathbf{X}_{r(3)}^{\text{L}}$$\mathbf{X}_{i(3)}^{\text{H}}$$\mathbf{X}_{r(3)}^{\text{H}}$$f(\mathbf{X}_{\text{i(3)}}^{\text{L}};\mathbf{K}_{\text{r}}^{\text{L}\\!\rightarrow\\!\text{L}})\\!-\\!f(\mathbf{X}_{\text{i(3)}}^{\text{L}};\mathbf{K}_{\text{i}}^{\text{L}\\!\rightarrow\\!\text{L}})$$u(f(\mathbf{X}_{\text{i(3)}}^{\text{L}}\\!;\\!\mathbf{K}_{\text{r}}^{\text{L}\\!\rightarrow\\!\text{H}})\\!,\\!2)\\!-\\!u(f(\mathbf{X}_{\text{i(3)}}^{\text{L}}\\!;\\!\mathbf{K}_{\text{i}}^{\text{L}\\!\rightarrow\\!\text{H}})\\!,\\!2)$$f(\mathbf{X}_{\text{r(3)}}^{\text{L}};\mathbf{K}_{\text{r}}^{\text{L}\\!\rightarrow\\!\text{L}})\\!+\\!f(\mathbf{X}_{\text{r(3)}}^{\text{L}};\mathbf{K}_{\text{i}}^{\text{L}\\!\rightarrow\\!\text{L}})$$u(f(\mathbf{X}_{\text{r(3)}}^{\text{L}}\\!;\\!\mathbf{K}_{\text{r}}^{\text{L}\\!\rightarrow\\!\text{H}})\\!,\\!2)\\!+\\!u(f(\mathbf{X}_{\text{r(3)}}^{\text{L}}\\!;\\!\mathbf{K}_{\text{i}}^{\text{L}\\!\rightarrow\\!\text{H}})\\!,\\!2)$$f(p(\mathbf{X}_{\rm
i(3)\\!}^{\rm H},2);\mathbf{K}_{\rm r}^{{\rm H\\!}\\!\rightarrow\\!{\rm
L\\!}}))\\!-\\!f(p(\mathbf{X}_{\rm i(3)\\!}^{\rm H\\!},2);\mathbf{K}_{\rm
i\\!}^{{\rm H\\!}\\!\rightarrow\\!{\rm
L\\!}}))$$f(\mathbf{X}_{\text{i(3)}}^{\text{H}};\mathbf{K}_{\text{r}}^{\text{H}\\!\rightarrow\\!\text{H}})\\!-\\!f(\mathbf{X}_{\text{i(3)}}^{\text{H}};\mathbf{K}_{\text{i}}^{\text{H}\\!\rightarrow\\!\text{H}})$$f(p(\mathbf{X}_{\rm
r(3)\\!}^{\rm H},2);\mathbf{K}_{\rm r}^{{\rm H\\!}\\!\rightarrow\\!{\rm
L\\!}}))\\!+\\!f(p(\mathbf{X}_{\rm r(3)\\!}^{\rm H\\!},2);\mathbf{K}_{\rm
i}^{{\rm H\\!}\\!\rightarrow\\!{\rm
L\\!}}))$$f(\mathbf{X}_{\text{r(3)}}^{\text{H}};\mathbf{K}_{\text{r}}^{\text{H}\\!\rightarrow\\!\text{H}})\\!+\\!f(\mathbf{X}_{\text{r(3)}}^{\text{H}};\mathbf{K}_{\text{i}}^{\text{H}\\!\rightarrow\\!\text{H}})$

Figure 3: Detailed design of our Dual-OctConv block.
$\mathbf{X}\\!\in\\!\mathbb{C}^{c\\!\times\\!h\\!\times\\!w}$ represents the
input complex-valued feature maps, and
$\mathbf{Y}\\!\in\\!\mathbb{C}^{c\\!\times\\!h\\!\times\\!w}$ indicates the
corresponding output feature maps, modulated by the Dual-OctConv. $u$ and $p$
denote the upsampling and average pooling operations, respectively. Please see
Eq. (9) for more details.

###  II-B Multi-Scale Representation Learning

Multi-scale information has proven effective in various computer vision tasks
(e.g., image classification, object detection, semantic segmentation).
Especially in the era of deep learning, multi-scale representation has been
successfully applied in various fields due to its strong robustness and
generalization ability. Several strategies have been proposed for this,
yielding significant performance improvement in a number of tasks. For
example, [39] proposed a multi-grid network to propagate and integrate
information across multiple scales for image classification. Multi-scale
information has also been proven effective in restoring image details for
image enhancement [40, 41, 42]. Seungjun et al. [40] proposed a multi-scale
convolutional neural network together with a multi-scale loss function for
dynamic scene deblurring. In addition, various well-known techniques (e.g.,
FPN [43] and PSP [44]) have been proposed for learning multi-scale
representations in object detection and segmentation tasks. For example, Lin
et al. [43] proposed a top-down feature pyramid architecture with lateral
connections at all scales for object detection. Recently, the Octave
convolution [30] was proposed to learn multi-scale features based on the
spatial frequency model [45, 46], greatly improving performance in natural
image and video recognition.

In this work, we demonstrate the appealing properties of the Octave
convolution for accelerated parallel MR image reconstruction, which helps to
capture multi-scale information from features of multiple spatial frequencies.
Based on this, we propose a novel Dual-OctConv for accelerated parallel MR
image reconstruction, which enables our model to capture details of
vasculatures and tissues with varying sizes and shapes, yielding high-fidelity
reconstructions.

##  III Methodology

###  III-A Problem Formulation

MR scanners acquire $k$-space data through the receiver coils and then utilize
an inverse multidimensional Fourier transform to obtain the final MR images.
In parallel imaging, multiple receiver coils are used to simultaneously
acquire $k$-space data from the target under scanning.

Let $\mathbf{A}\\!=\\!\mathbf{M}\\!\mathbf{F}\\!\in\\!\mathbb{C}^{M\times N}$
denote the undersampled Fourier encoding matrix, where $\mathbf{F}$ is the
multidimensional Fourier transform, and $\mathbf{M}$ is an undersampled mask
operator. In parallel imaging, the same mask is used for all coils. The
undersampled $k$-space data from each coil can be expressed as:

$\mathbf{y}_{i}=\mathbf{A}(\mathbf{S}_{i}\mathbf{x}),$ (1)

where $i=1,2,...,c$, with $c$ denoting the number of coils,
$\mathbf{x}\\!\in\\!\mathbb{C}^{N\times 1}$ is the ground truth MR image,
$\mathbf{y}_{i}\\!\in\\!\mathbb{C}^{M\times 1}(M\\!<\\!<\\!N)$ is the
undersampled $k$-space data for the $i$-th coil, and $\mathbf{S}_{i}$ is a
complex-valued diagonal matrix encoding the sensitivity map of the $i$-th
coil. The coil sensitivities, which are measured by each coil, modulate the
$k$-space data. The coil configuration and interactions with the anatomical
structures under scanning can affect the coil sensitivities, so
$\mathbf{S}_{i}$ changes across different scans. In addition, the obtained
image will contain aliasing artifacts if the inverse Fourier transform is
directly applied to the undersampled $k$-space data.

We can reconstruct $\hat{\mathbf{x}}$ with prior knowledge of its properties,
which can be formulated as the following optimization problem:

$\hat{\mathbf{x}}=\mathop{\arg\min}_{\mathbf{x}}\sum_{i=1}^{c}\
\|{\mathbf{y}_{i}-\mathbf{A}(\mathbf{S}_{i}\mathbf{x})}\|^{2}_{2}+\
\lambda\Psi(\mathbf{x}),$ (2)

where $\Psi$ is a regularization function and $\lambda$ controls the trade-off
between the two terms.

The problem presented in Eq. (2) can be effectively resolved using CNNs, which
avoids time-consuming numerical optimization and the need of a coil
sensitivity map. During training, we update the network weights by minimizing
an $\ell_{1}$ loss function,

$\hat{\bm{\theta}}=\mathop{\arg\min}_{\bm{\theta}}\frac{1}{N}\sum_{n=1}^{N}\|{\mathbf{x^{\prime}}(n)-f_{\bm{\theta}}(\mathbf{y^{\prime}}(n))}\|_{1},$
(3)

where $\mathbf{y^{\prime}}(n)$ is the $n$-th multi-channel image obtained from
the zero-filled $k$-space data, $\mathbf{x^{\prime}}(n)$ is the $n$-th multi-
channel ground truth image, $N$ is the total number of training samples, and
$f_{\bm{\theta}}(\cdot)$ is an end-to-end mapping function parameterized by
$\bm{\theta}$, which contains a large number of adjustable network weights.
Training with Eq. (3) can reconstruct the expected MR images, but the original
information of the data acquired in the $k$-space cannot be well preserved. If
we incorporate the undersampled $k$-space data into the data fidelity at the
training stage, the network can yield improved reconstruction results. For
this purpose, we add the data fidelity units in our network, as in [2]. After
the network is trained, we obtain a set of optimal parameters
$\bm{\hat{\theta}}$ for the reconstruction of a multi-channel image, and
predict this image via
$\mathbf{\hat{x}^{\prime}}=f_{\hat{\bm{\theta}}}(\mathbf{y^{\prime}})$.
Finally, we use an adaptive coil combination method [2] to obtain the expected
MR image from $\mathbf{\hat{x}^{\prime}}$.

Figure 4: The Dual-OctConv kernels. Red and blue squares correspond to the
imaginary and real kernels, respectively.

###  III-B Framework Overview

Fig. 2 illustrates the architecture of our DONet, which simultaneously
considers both $k$-space data fidelity and image space proximity to achieve
high-fidelity reconstruction. Specifically, given an undersampled multi-coil
$k$-space measurement as input, we first convert it to aliased multi-channel
images via an inverse Fourier transform (IFT). The transformed image is then
fed into a cascade of $T$ Dual-OctConv blocks, with each block followed by a
data fidelity unit [2]. The data fidelity unit helps to preserve the original
$k$-space information during training. Formally, the $t$-th data fidelity unit
can be written as:

$\mathbf{X}^{t}=\mathbf{F}^{-1}((\mathbf{1}-\mathbf{M})\mathbf{F}(\check{\mathbf{X}}^{t})+\mathbf{M}\mathbf{y}),$
(4)

where $\check{\mathbf{X}}^{t}$ denotes the reconstructed output image of the
$t$-th Dual-OctConv block, while $\mathbf{X}^{t}$ represents the output after
applying the $t$-th data fidelity unit, which will be fed into the
$t\\!+\\!1$-th Dual-OctConv block as input. $\mathbf{F}^{-1}$ denotes the
inverse Fourier transform.

Next, we will describe the details of our Dual-OctConv block in §III-C and the
Dual-OctConv layer in §III-D.

###  III-C Dual-Octave Convolutional Block

We present the detailed architecture of our Dual-OctConv block, which is
employed to learn comprehensive, multi-scale feature representations for
reconstruction. In the following paragraphs, the superscript ‘$t$’ is omitted
for conciseness, unless necessary.

As shown in Fig. 3, each Dual-OctConv block consists of $K$ Dual-OctConv
layers $\\{\mathcal{D}_{k}\\}_{k=1}^{K}$. These layers accecpt $\mathbf{X}$ as
input and produce the reconstruction $\check{\mathbf{X}}$, in a sequential
manner. More specifically, to obtain rich multi-scale context information, we
first represent the multi-channel input with complex filters, and then
decompose it into low and high spatial frequency parts. Let
$\mathbf{X}\\!\in\\!\mathbb{C}^{c\times h\times w}=\mathbf{X}_{\rm
r(0)}+i\mathbf{X}_{\rm i(0)}$ be the input complex feature maps with $c$, $h$,
and $w$ denoting the number of channels, height, and width, respectively. The
$\mathbf{X}_{\rm r(0)}$ and $\mathbf{X}_{\rm i(0)}$ are the real and imaginary
parts of the complex data, respectively. As illustrated in Fig. 3, we
initially split the complex input feature maps into low- and high-frequency
groups: $\mathbf{X}_{\rm r(0)}\\!=\\!\\{\mathbf{X}_{\text{r}(0)}^{\rm
L},\mathbf{X}_{\text{r}(0)}^{\rm H}\\}$,
$\mathbf{X}_{\rm{i}(0)}\\!=\\!\\{{\mathbf{X}_{\text{i}(0)}^{\rm
L},\mathbf{X}_{\text{i}(0)}^{\rm H}}\\}$, where $\mathbf{X}^{\rm
H}\\!\in\\!\mathbb{C}^{(1-\alpha)c\times h\times w}$ captures the high-
frequency fine details of the data and $\mathbf{X}^{\rm
L}\\!\in\\!\mathbb{C}^{\alpha c\times 0.5h\times 0.5w}$ determines the low-
frequency image contrast. Here, $\alpha\\!\in\\![0,1]$ controls the ratio of
channels that are allocated to low-frequency and high-frequency feature maps.
Note that the Dual-OctConv becomes the standard complex convolution [29] when
$\alpha\\!=\\!0$. Then, for the $k$-th Dual-OctConv layer $\mathcal{D}_{k}$,
we have:

$\displaystyle\mathbf{X}_{\text{r}(k)}^{\text{L}},$
$\displaystyle\mathbf{X}_{\text{r}(k)}^{\text{H}},\mathbf{X}_{\text{i}(k)}^{\rm
L},\mathbf{X}_{\text{i}(k)}^{\rm H}$ (5)
$\displaystyle=\mathcal{D}_{k}(\mathbf{X}_{\text{r}(k-1)}^{\rm
L},\mathbf{X}_{\text{r}(k-1)}^{\rm H},\mathbf{X}_{\text{i}(k-1)}^{\rm
L},\mathbf{X}_{\text{i}(k-1)}^{\rm H}),$

where $\mathbf{X}_{\text{r}(k-1)}^{\rm L}$, $\mathbf{X}_{\text{r}(k-1)}^{\rm
H}$, $\mathbf{X}_{\text{i}(k-1)}^{\rm L}$, $\mathbf{X}_{\text{i}(k-1)}^{\rm
H}$ are the inputs, and $\mathbf{X}_{\text{r}(k)}^{\rm L}$,
$\mathbf{X}_{\text{r}(k)}^{\rm H}$, $\mathbf{X}_{\text{i}(k)}^{\rm L}$,
$\mathbf{X}_{\text{i}(k)}^{\rm H}$ are the outputs of $\mathcal{D}_{k}$. Then,
for each block, we obtain the final feature representations at the $K$-th
Dual-OctConv layer, which are combined to obtain the reconstruction:

$\check{\mathbf{X}}=u(c(\mathbf{X}_{\text{r}(K)}^{\rm
L},\mathbf{X}_{\text{i}(K)}^{\text{L}}),2)+c(\mathbf{X}_{\text{r}(K)}^{\text{H}},\mathbf{X}_{\text{i}(K)}^{\rm
H}),$ (6)

where $u(\cdot,z)$ denotes the upsampling operation with a factor of $z$.
Here, we use the nearest neighbor interpolation. $c(\cdot,\cdot)$ denotes the
concatenation operation. The output $\check{\mathbf{X}}$ is then fed into the
data fidelity layer Eq. (4) as well as the subsequent Dual-OctConv block for
cascade reconstruction.

In addition, we emphasize that the sequential pipeline mentioned above
conducts individual feature learning layer-by-layer, neglecting the feature
fusion among different layers, which often produces more comprehensive feature
representations [47, 48]. To this end, we improve the sequential Dual-OctConv
block by introducing dense connections in an interleaved manner. Formally, for
the output of the $k$-th ($k\\!\geq\\!1$) Dual-OctConv layer, its features are
modulated as follows:

$\begin{split}\mathbf{X}_{\text{r}(k)}^{\rm
L}&=\mathcal{C}_{k}([\mathbf{X}_{\text{r}(k)}^{\rm
L},\mathbf{X}_{\text{r}(k-1)}^{\rm L},\cdots,\mathbf{X}_{\text{r}(0)}^{\rm
L}]),\\\ \mathbf{X}_{\text{i}(k)}^{\rm
L}&=\mathcal{C}_{k}([\mathbf{X}_{\text{i}(k)}^{\rm
L},\mathbf{X}_{\text{i}(k-1)}^{\rm L},\cdots,\mathbf{X}_{\text{i}(0)}^{\rm
L}]),\\\ \mathbf{X}_{\text{r}(k)}^{\rm
H}&=\mathcal{C}_{k}([\mathbf{X}_{\text{r}(k)}^{\rm
H},\mathbf{X}_{\text{r}(k-1)}^{\rm H},\cdots,\mathbf{X}_{\text{r}(0)}^{\rm
H}]),\\\ \mathbf{X}_{\text{i}(k)}^{\rm
H}&=\mathcal{C}_{k}([\mathbf{X}_{\text{i}(k)}^{\rm
H},\mathbf{X}_{\text{i}(k-1)}^{\rm H},\cdots,\mathbf{X}_{\text{i}(0)}^{\rm
H}]).\\\ \end{split}$ (7)

Here, the feature maps at the $k$-th layer are firstly concatenated the
features at all previous layers, and then processed with a function
$\mathcal{C}_{k}$ to learn collective knowledge from all these feature maps.
Following [48], we implement $\mathcal{C}_{k}$ as: BN-ReLU-
Conv($1\\!\times\\!1$)-BN-ReLU-Conv($3\\!\times\\!3$). Such dense connections
enable our model to reuse features from previous layers with high
computational efficiency, since there is no need to relearn redundant feature
maps. They also help to alleviate the gradient vanishing problem when the
number of layers in our network increases. In our experiments, we demonstrate
that the reconstruction performance significantly improves after introducing
these dense connections.

###  III-D Dual-Octave Convolutional Layer

Now we present the proposed Dual-Octave convolutional layer, which fuses the
real and imaginary parts of a complex valued input. For the $k$-th layer
$\mathcal{D}_{k}$, we denote
$\mathbf{X}_{(k-1)}\\!=\\!\mathbf{X}_{\text{r}(k-1)}+i\mathbf{X}_{\text{i}(k-1)}$
as its input. We convolve $\mathbf{X}_{(k-1)}$ with a complex filter matrix
$\mathbf{K}_{(k)}=\mathbf{K}_{\text{r}(k)}+i\mathbf{K}_{\text{i}(k)}$ as
follows:

$\\!\\!\left[\\!\begin{array}[]{ccc}\Re(\mathbf{K}_{(k)}*\mathbf{X}_{(k-1)})\\\
\Im(\mathbf{K}_{(k)}*\mathbf{X}_{(k-1))}\end{array}\\!\right]\\!=\\!{\left[\\!\begin{array}[]{ccc}\mathbf{K}_{\text{r}(k)}\\!\\!&\\!\\!-\mathbf{K}_{\text{i}(k)}\\\
\mathbf{K}_{\text{i}(k)}\\!\\!&\\!\\!\mathbf{K}_{\text{r}(k)}\end{array}\\!\right]}\\!*\\!{\left[\\!\begin{array}[]{ccc}\mathbf{X}_{\text{r}(k-1)}\\\
\mathbf{X}_{\text{i}(k-1)}\end{array}\\!\right]},$ (8)

where the matrices $\mathbf{K}_{\text{r}(k)}$ and $\mathbf{K}_{\text{i}(k)}$
represent real and imaginary kernels, respectively. Note that all the kernels
and feature maps are expressed by real matrices since the complex arithmetics
are simulated by real-valued entities.

As shown in Fig. 4, the complex filter matrix is further expressed as
$\mathbf{K}_{\rm r}^{\rm H}=[\mathbf{K}_{\text{r}(k)}^{{\rm H}\rightarrow{\rm
L}},\mathbf{K}_{\text{r}(k)}^{{\rm H}\rightarrow{\rm H}}]$,
$\mathbf{K}_{\text{i}(k)}^{\rm H}\\!=\\![\mathbf{K}_{\text{i}(k)}^{{\rm
H}\rightarrow{\rm L}},\mathbf{K}_{\text{i}(k)}^{{\rm H}\rightarrow{\rm H}}]$,
$\mathbf{K}_{\text{r}(k)}^{\rm L}\\!=\\![\mathbf{K}_{\text{r}(k)}^{{\rm
L}\rightarrow{\rm H}},\mathbf{K}_{\text{r}(k)}^{{\rm L}\rightarrow{\rm L}}]$,
$\mathbf{K}_{\text{i}(k)}^{\rm L}\\!=\\![\mathbf{K}_{\text{i}(k)}^{{\rm
L}\rightarrow{\rm H}},\mathbf{K}_{\text{i}(k)}^{{\rm L}\rightarrow{\rm L}}]$
to convolve with $\mathbf{X}_{\text{r}(k-1)}^{\rm L}$,
$\mathbf{X}_{\text{i}(k-1)}^{\rm L}$, $\mathbf{X}_{\text{r}(k-1)}^{\rm H}$ and
$\mathbf{X}_{\text{i}(k-1)}^{\rm H}$. Then, $\mathcal{D}_{k}$ can be
formulated as:

$\begin{split}\\!\mathbf{X}_{\text{r}(k)}^{\rm
L}\\!=\\!&f(\mathbf{X}_{\text{r}(k-1)}^{\rm L};\mathbf{K}_{\text{r}(k)}^{{\rm
L}\rightarrow{\rm L}})+u(f(\mathbf{X}_{\text{r}(k-1)}^{\rm
L};\mathbf{K}_{\text{r}(k)}^{{\rm L}\rightarrow H}),2)\\\
&\\!\\!\\!\\!\\!+f(\mathbf{X}_{\text{r}(k-1)}^{\rm
L};\mathbf{K}_{\text{i}(k)}^{{\rm L}\rightarrow{\rm
L}})+u(f(\mathbf{X}_{\text{r}(k-1)}^{\rm L};\mathbf{K}_{\text{i}(k)}^{{\rm
L}\rightarrow{\rm H}}),2),\\\ \\!\mathbf{X}_{\text{i}(k)}^{\rm
L}\\!=\\!&f(\mathbf{X}_{\text{i}(k-1)}^{\rm L};\mathbf{K}_{\text{r}(k)}^{{\rm
L}\rightarrow{\rm L}})+u(f(\mathbf{X}_{\text{i}(k-1)}^{\rm
L};\mathbf{K}_{\text{r}(k)}^{{\rm L}\rightarrow{\rm H}}),2)\\\
&\\!\\!\\!\\!\\!-f(\mathbf{X}_{\text{i}(k-1)}^{\rm{\rm
L}};\mathbf{K}_{\text{i}(k)}^{{\rm L}\rightarrow{\rm
L}})-u(f(\mathbf{X}_{\text{i}(k-1)}^{\rm L};\mathbf{K}_{\text{i}(k)}^{{\rm
L}\rightarrow{\rm H}}),2),\\\ \\!\mathbf{X}_{\text{r}(k)}^{\rm
H}\\!=\\!&f(\mathbf{X}_{\text{r}(k-1)}^{\rm H};\mathbf{K}_{\text{r}(k)}^{{\rm
H}\rightarrow{\rm H}})+f(p(\mathbf{X}_{\text{r}(k-1)}^{\rm
H},2);\mathbf{K}_{\text{r}(k)}^{{\rm H}\rightarrow{\rm L}}))\\\
&\\!\\!\\!\\!\\!+f(\mathbf{X}_{\text{r}(k-1)}^{\rm
H};\mathbf{K}_{\text{i}(k-1)}^{{\rm H}\rightarrow{\rm
H}})+f(p(\mathbf{X}_{\text{r}(k-1)}^{\rm H},2);\mathbf{K}_{\text{i}(k)}^{{\rm
H}\rightarrow{\rm L}})),\\\ \\!\mathbf{X}_{\text{i}(k)}^{\rm
H}\\!=\\!&f(\mathbf{X}_{\text{i}(k-1)}^{\rm H};\mathbf{K}_{\text{r}(k)}^{{\rm
H}\rightarrow{\rm H}})+f(p(\mathbf{X}_{\text{i}(k-1)}^{\rm
H},2);\mathbf{K}_{\text{r}(k)}^{{\rm H}\rightarrow{\rm L}}))\\\
&\\!\\!\\!\\!\\!-f(\mathbf{X}_{\text{i}(k-1)}^{\rm
H};\mathbf{K}_{\text{i}(k)}^{{\rm H}\rightarrow{\rm
H}})-f(p(\mathbf{X}_{\text{i}(k-1)}^{\rm H},2);\mathbf{K}_{\text{i}(k)}^{{\rm
H}\rightarrow{\rm L}})),\\\ \end{split}$ (9)

where $f(\mathbf{X};\mathbf{K})$ indicates the convolution with kernel
$\mathbf{K}$, $u(\mathbf{X},k)$ denotes the upsampling operation with a factor
of $k$ via nearest interpolation, and $p(\mathbf{X},z)$ represents an average
pooling layer with kernel size $z\\!\times\\!z$. The real and imaginary parts
are fused with the operations $\left\\{{\rm L}\rightarrow{\rm L},{\rm
H}\rightarrow{\rm H}\right\\}$ and $\left\\{{\rm H}\rightarrow{\rm L},{\rm
L}\rightarrow{\rm H}\right\\}$, which correspond to the information updating
and exchanging between high- and low-frequency feature maps. Therefore, our
Dual-OctConv is able to enlarge the receptive fields of the low-frequency
feature maps both in the real and imaginary parts. To put this into
perspective, after convolving the low-frequency feature maps of the real and
imaginary parts ( $\mathbf{X}_{\text{r}(k-1)}^{\rm L}$,
$\mathbf{X}_{\text{i}(k-1)}^{\rm L}$) with $z\times z$ complex convolutional
kernels, the receptive fields of both achieve a 2$\times$ enlargement compared
to the vanilla convolution. Thus, our Dual-OctConv has a strong ability to
capture rich context information at different scales.

TABLE I: Quantitative comparison of state-of-the-art methods under different
undersampling patterns on the clinical knee dataset. Best and second best
results are marked in red and blue, respectively.

undersampling pattern | 1D Uniform | 1D Cartesian | 2D Random | 2D Radial
---|---|---|---|---
acceleration rate | 3x | 5x | 3x | 5x | 3x | 5x | 4x | 6x
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
Zero-filing | 24.406 | 0.676 | 23.579 | 0.655 | 25.922 | 0.726 | 24.550 | 0.685 | 30.540 | 0.827 | 27.078 | 0.750 | 31.026 | 0.826 | 28.107 | 0.766
SPIRiT | 29.385 | 0.700 | 28.300 | 0.676 | 32.310 | 0.801 | 31.222 | 0.782 | 32.179 | 0.786 | 32.258 | 0.812 | 30.308 | 0.720 | 29.061 | 0.702
L1-SPIRiT | 29.815 | 0.847 | 27.353 | 0.788 | 33.346 | 0.887 | 30.912 | 0.837 | 38.597 | 0.937 | 34.071 | 0.887 | 37.004 | 0.919 | 34.149 | 0.881
VN-Net | 35.436 | 0.907 | 32.730 | 0.858 | 36.364 | 0.912 | 33.236 | 0.866 | 38.409 | 0.956 | 35.734 | 0.923 | 37.956 | 0.930 | 34.609 | 0.907
ComplexMRI | 34.989 | 0.909 | 32.803 | 0.873 | 35.957 | 0.916 | 34.126 | 0.876 | 39.563 | 0.946 | 37.315 | 0.908 | 38.098 | 0.933 | 35.768 | 0.904
DONet† | 36.243 | 0.919 | 34.128 | 0.885 | 37.029 | 0.923 | 34.944 | 0.884 | 39.964 | 0.948 | 38.279 | 0.930 | 38.607 | 0.935 | 36.584 | 0.908
DONet | 36.338 | 0.918 | 34.526 | 0.892 | 37.074 | 0.931 | 35.400 | 0.902 | 40.652 | 0.953 | 38.428 | 0.930 | 38.387 | 0.935 | 36.281 | 0.909

##  IV Experiments

TABLE II: Quantitative comparison of state-of-the-art methods under different
undersampling patterns on fastMRI dataset. Best and second best results are
marked in red and blue, respectively.

undersampling pattern | 1D Uniform | 1D Cartesian | 2D Random | 2D Radial
---|---|---|---|---
acceleration rate | 3x | 5x | 3x | 5x | 3x | 5x | 3x | 5x
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
Zero-filing | 22.981 | 0.681 | 22.600 | 0.662 | 22.553 | 0.689 | 22.274 | 0.671 | 29.292 | 0.833 | 25.614 | 0.760 | 31.308 | 0.849 | 28.460 | 0.787
SPIRiT | 27.305 | 0.601 | 27.185 | 0.610 | 29.387 | 0.694 | 28.133 | 0.648 | 30.950 | 0.725 | 30.293 | 0.735 | 29.778 | 0.681 | 28.099 | 0.611
L1-SPIRiT | 29.597 | 0.827 | 26.501 | 0.774 | 28.757 | 0.835 | 28.000 | 0.801 | 37.275 | 0.911 | 33.107 | 0.852 | 36.896 | 0.907 | 34.420 | 0.868
VN-Net | 31.032 | 0.860 | 28.641 | 0.810 | 32.105 | 0.875 | 31.544 | 0.853 | 37.113 | 0.907 | 34.726 | 0.813 | 36.516 | 0.894 | 33.899 | 0.825
ComplexMRI | 31.548 | 0.867 | 29.922 | 0.829 | 32.773 | 0.888 | 31.590 | 0.852 | 37.751 | 0.926 | 35.227 | 0.891 | 37.238 | 0.922 | 34.753 | 0.882
DONet† | 32.415 | 0.874 | 29.895 | 0.829 | 33.204 | 0.891 | 32.576 | 0.862 | 38.058 | 0.927 | 35.911 | 0.894 | 37.523 | 0.930 | 35.112 | 0.884
DONet | 33.890 | 0.888 | 32.780 | 0.856 | 35.094 | 0.904 | 33.644 | 0.868 | 38.316 | 0.928 | 36.385 | 0.896 | 37.905 | 0.925 | 35.413 | 0.886

DONet†

Figure 5: Comparison of different methods in terms of reconstruction accuracy,
with 1D undersampling patterns and a 3$\times$ acceleration rate.
Reconstruction results and error maps are presented with corresponding
quantitative measurements in PSNR/SSIM.

DONet†

Figure 6: Comparison of different methods in terms of reconstruction accuracy,
with 2D undersampling patterns and 4$\times$ and 5$\times$ acceleration rates.
Reconstruction results and error maps are presented with corresponding
quantitative measurements in PSNR/SSIM. Figure 7: Quantitative comparison of
Dual-OctConv and the baseline convolution ($\alpha\\!=\\!0$) in terms of PSNR
and SSIM. Note that ratio $\alpha\\!=\\!0$ is equivalent to the complexMRI
model. Figure 8: Analysis of spatial frequency ratio ($\alpha$) in terms of
PSNR and SSIM. Figure 9: Performance comparison of our network with respect to
the number of Dual-OctConv blocks. Figure 10: FLOPs analysis with respect to
spatial frequency ratios ($\alpha$). The number below each point is the value
of $\alpha$. We see that, under various settings ($0\\!<\\!\alpha\\!<\\!1$),
our DONet is always more efficient and accurate than the baseline model
($\alpha\\!=\\!0$).

###  IV-A Datasets

We introduce a clinical knee [22] dataset and the fastMRI [49] dataset under
different undersampling patterns and acceleration factors to evaluate our
method. The clinical multi-coil fully-sampled MR knee dataset is acquired
using a clinical 3T Siemens Magnetom Skyra scanner with a sequence called
“Coronal Spin Density Weighted without Fat Suppression”. The imaging protocol
is detailed as follows: 15-channel knee coil, matrix size
320$\times$320$\times$20, TR=2750 ms, TE=27 ms, and in-plane resolution =
0.49$\times$0.44 mm2. There are 20 subjects in total with the following
information: 5 female/15 male, age 15-76, and BMI 20.46-32.94. We randomly
select fourteen patients for training, three for validation, and three for
testing. For the fastMRI dataset, we randomly select 400 patients from the
multi-coil knee dataset using a Proton Density (PD) weighted sequence for
training, 40 for validation, and 60 for testing. In our experiments, we
exclude the first few slices of each volume since the frontal slices are much
noisier than the other slices, making the distribution of frontal slices
different from the rest. More details of the fastMRI dataset can be obtained
from [49].

The pre-defined undersampling masks are used to obtain the undersampled
measurements. In our experiments, we adopt four different $k$-space
undersampling patterns, including 1D uniform, 1D Cartesian, 2D random, and 2D
radial. Examples of the undersampling patterns are illustrated in Fig. 5 and
Fig. 6. For 1D uniform, 1D Cartesian, and 2D random masks, the acceleration
rate is set to 3$\times$ and 5$\times$. For the 2D radial mask, 4$\times$ and
6$\times$ accelerations are adopted.

###  IV-B Implementation Details

We implement our model using Tensorflow 1.14 and perform experiments using an
NVIDIA 1080Ti GPU with 11GB of memory. Following [2], we initialize the
magnitude and phase of the complex parameters using Rayleigh and uniform
distributions, respectively. The network is trained using the Adam optimizer
[2] with initial learning rate 0.001 and weight decay 0.95. The batch size is
set to four and the convolutional kernel size is set to $3\times 3$. Each
complex convolutional layer has 64 feature maps, except for the last layer,
which is determined by the concatenated real and imaginary channels of the
data. The spatial frequency ratio $\alpha$ is set to $0.125$ by default.

To demonstrate their effectiveness, we compare our DONet and DONet† (without
dense connections) with several state-of-the-art parallel MR imaging
approaches, including traditional methods (SPIRiT [19] and L1-SPIRiT [50]) as
well as CNN-based methods (VN-Net [22] and ComplexMRI [2]). All these methods
are trained on the same dataset with their default settings. For CNN-based
methods, we re-train them according to the specifications with TensorFlow,
using their default parameter settings.

###  IV-C Quantitative Evaluation

We use peak signal-to-noise ratio (PSNR) and structural similarity index
measure (SSIM) [2] for quantitative evaluation. Table I reports the average
PSNR and SSIM results with respect to different undersampling patterns and
acceleration factors on the clinical dataset. As can be seen, DONet† obtains
consistent performance improvements against all baseline methods,
demonstrating the superiority of our Dual-OctConv. Moreover, by incorporating
dense connections, our full model (i.e., DONet) further improve the
performance significantly across various settings.

Additionally, we observe that the undersampling patterns greatly affect the
quality of reconstruction. For instance, the 2D sampling masks generally
outperform the 1D masks. Another important observation is that the
reconstruction becomes more difficult when the acceleration rate increases.

In particular, our model significantly outperforms previous methods under
extremely challenging settings (e.g., 2D masks with 5$\times$ and 6$\times$
acceleration). This can be attributed to the powerful capability of our DONet
in aggregating rich contextual information of real and imaginary data.
Moreover, the overall results in Table I show the strong robustness of our
model under various undersampling patterns and acceleration rates. For
example, our method restores more information and with minimum artifacts, as
demonstrated in Fig. 5 and Fig. 6 with error maps.

In Table II, we provide reconstruction evaluations of our model on fastMRI,
which is currently the largest MR image dataset. As we can see, DONet obtains
the best reconstruction results among different undersampling patterns and
acceleration rates. In particular, under 1D uniform undersampling patterns
with 5$\times$ acceleration, our DONet improves the reconstruction PSNR from
29.992 dB to 32.780 dB and SSIM from 0.829 to 0.856, compared to the best
previous state-of-the-art fusion method, ComplexMRI. Similarly, we observe
that our DONet yields outstanding results on both the 1D uniform and 1D
Cartesian undersampling patterns with 3$\times$ and 5$\times$ acceleration.
More importantly, even without the dense connection, our DONet† achieves
compelling reconstruction results compared to the state-of-the-art methods.
Under the 2D radial pattern with 3$\times$ acceleration, DONet† achieves SSIM
= 0.930. All the results demonstrate that our DONet preserves powerful and
rich contextual information for real and imaginary data.

###  IV-D Qualitative Evaluation

For qualitative analysis, we first show the reconstructed images and
corresponding error maps for 1D uniform and 1D Cartesian sampling with a
3$\times$ acceleration rate in Fig. 5. In general, 1D Cartesian masks provide
better results in comparison to 1D uniform masks. Our method provides the
best-quality reconstructed images and significantly reduces prediction errors.
In contrast, the baseline methods yield large prediction errors and show
unsatisfactory performance.

We next examine the results for 2D radial masks with a 4$\times$ acceleration
rate and 2D random masks with a 5$\times$ acceleration rate. As shown in Fig.
6, CNN-based methods significantly improve the results with less errors and
clearer structures, in comparison with SPIRiT and L1-SPIRiT. In particular,
our DONet produces higher-quality images with clear details and minimum
artifacts. The superior performance is owed to the fact that our method can
effectively aggregate the information of various spatial frequencies present
in the real and imaginary parts of an MR image.

###  IV-E Ablation Studies

The crucial part of our DONet is the Dual-OctConv. It is thus important to
prove the effectiveness of the proposed convolution. First, we study the
effects of Dual-OctConv. For comparison, we build a baseline convolution by
setting $\alpha\\!=\\!0$, which turns our method into a standard complex
convolution. We conduct experiments on the test set of $in$ $vivo$ dataset
with 60 complex-valued images under the uniform undersampling mask with a
3$\times$ acceleration rate. As illustrated in Fig. 7, our Dual-OctConv
significantly outperforms the baseline model, especially in terms of SSIM.
This reveals the superiority of the proposed method in improving the
reconstruction.

Secondly, we investigate the influence of the spatial frequency ratio $\alpha$
for reconstruction. The ratio determines the receptive fields in both the real
and imaginary parts, and also influences the fusion of these parts at multiple
spatial frequencies. As shown in Fig. 8, our model achieves the best PSNR and
SSIM scores at $\alpha\\!=\\!0.125$, which means that 12.5% of the channels in
the real and imaginary parts are reduced to a low spatial frequency. When
$\alpha$ becomes larger, the performance quickly degrades due to severe
information loss induced by over-large ratios.

The number of network parameters increases as the number of blocks ($bn$)
increases. Therefore, it is necessary to choose an appropriate number of
blocks to ensure that the network structure reaches the highest reconstruction
accuracy without inducing higher computational and memory requirements.
Herein, we carry out various experiments using different numbers of blocks.
The results are presented in Fig. 9. As can be seen from the curves, our
model can successfully reconstruct the MR images at $bn\\!=\\!4$, and the
reconstruction accuracy reaches the highest at $bn\\!=\\!10$.

Finally, we study the FLOPs of DONet with respect to different $\alpha$ in
Fig. 10. The number below each point is the value of $\alpha$, and
$\alpha\\!=\\!0$ refers to the baseline model. As can be observed, a small
$\alpha$ leads to improved performance with a higher FLOPs. Moreover, compared
with the baseline model (e.g., $\alpha\\!=\\!0$), our model consistently shows
better performance with much lower FLOPs.

##  V Conclusion

In this work, we focus on spatial frequency feature expression in complex-
valued data for parallel MR image reconstruction. For this purpose, we propose
a DONet with a series of novel Dual-OctConv operations to deal with the real
and imaginary components of the data at multiple spatial frequencies. By
convolving the feature maps of both the real and imaginary components under
different spatial resolutions, the proposed Dual-OctConv facilitates our DONet
to learn more comprehensive feature representations, yielding higher-quality
reconstructed images with significantly reduced artifacts. We conduct
extensive experiments on the $in~{}vivo$ knee and fastMRI datasets under
different settings of undersampling patterns and acceleration rates. The
results demonstrate the advantages of our model against state-of-the-art
methods in accelerated MR image reconstruction.

## References

  * [1] L. Sun, Z. Fan, Y. Huang, X. Ding, and J. Paisley, “Compressed sensing mri using a recursive dilated network,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [2] S. Wang, H. Cheng, L. Ying, T. Xiao, Z. Ke, H. Zheng, and D. Liang, “Deepcomplexmri: Exploiting deep residual network for fast parallel mr imaging with complex convolution,” _Magnetic Resonance Imaging_ , vol. 68, pp. 136–147, 2020.
  * [3] H. K. Aggarwal, M. P. Mani, and M. Jacob, “Modl: Model-based deep learning architecture for inverse problems,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 2, pp. 394–405, 2018.
  * [4] S. G. Lingala and M. Jacob, “Blind compressive sensing dynamic mri,” _IEEE transactions on medical imaging_ , vol. 32, no. 6, pp. 1132–1145, 2013\.
  * [5] Y. Wang and L. Ying, “Undersampled dynamic magnetic resonance imaging using kernel principal component analysis,” in _2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society_. IEEE, 2014, pp. 1533–1536.
  * [6] E. Hot and P. Sekulić, “Compressed sensing mri using masked dct and dft measurements,” in _2015 4th Mediterranean Conference on Embedded Computing (MECO)_. IEEE, 2015, pp. 323–326.
  * [7] A. C. Gilbert, P. Indyk, M. Iwen, and L. Schmidt, “Recent developments in the sparse fourier transform: A compressed fourier transform for big data,” _IEEE Signal Processing Magazine_ , vol. 31, no. 5, pp. 91–100, 2014.
  * [8] Z. Lai, X. Qu, Y. Liu, D. Guo, J. Ye, Z. Zhan, and Z. Chen, “Image reconstruction of compressed sensing mri using graph-based redundant wavelet transform,” _Medical image analysis_ , vol. 27, pp. 93–104, 2016.
  * [9] X. Qu, D. Guo, B. Ning, Y. Hou, Y. Lin, S. Cai, and Z. Chen, “Undersampled mri reconstruction with patch-based directional wavelets,” _Magnetic resonance imaging_ , vol. 30, no. 7, pp. 964–977, 2012.
  * [10] L. Chaâri, J.-C. Pesquet, A. Benazza-Benyahia, and P. Ciuciu, “A wavelet-based regularized reconstruction algorithm for sense parallel mri with applications to neuroimaging,” _Medical image analysis_ , vol. 15, no. 2, pp. 185–201, 2011.
  * [11] Q. Liu, Q. Yang, H. Cheng, S. Wang, M. Zhang, and D. Liang, “Highly undersampled magnetic resonance imaging reconstruction using autoencoding priors,” _Magnetic resonance in medicine_ , vol. 83, no. 1, pp. 322–336, 2020.
  * [12] F. Knoll, K. Hammernik, C. Zhang, S. Moeller, T. Pock, D. K. Sodickson, and M. Akcakaya, “Deep learning methods for parallel magnetic resonance image reconstruction,” _arXiv preprint arXiv:1904.01112_ , 2019.
  * [13] S. Wang, S. Tan, Y. Gao, Q. Liu, L. Ying, T. Xiao, Y. Liu, X. Liu, H. Zheng, and D. Liang, “Learning joint-sparse codes for calibration-free parallel mr imaging,” _IEEE Transactions on Medical Imaging_ , vol. 37, no. 1, pp. 251–261, 2017.
  * [14] Y. Chang, D. Liang, and L. Ying, “A kernel approach to parallel mri reconstruction,” in _2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro_ , 2011, pp. 389–392.
  * [15] C. Chen, Y. Li, and J. Huang, “Calibrationless parallel mri with joint total variation regularization,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2013, pp. 106–114.
  * [16] K. P. Pruessmann, M. Weiger, M. B. Scheidegger, and P. Boesiger, “Sense: sensitivity encoding for fast mri,” _Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine_ , vol. 42, no. 5, pp. 952–962, 1999.
  * [17] M. A. Griswold, P. M. Jakob, R. M. Heidemann, M. Nittka, V. Jellus, J. Wang, B. Kiefer, and A. Haase, “Generalized autocalibrating partially parallel acquisitions (grappa),” _Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine_ , vol. 47, no. 6, pp. 1202–1210, 2002.
  * [18] D. S. Weller, J. R. Polimeni, L. Grady, L. L. Wald, E. Adalsteinsson, and V. K. Goyal, “Sparsity-promoting calibration for grappa accelerated parallel mri reconstruction,” _IEEE Transactions on Medical Imaging_ , vol. 32, no. 7, pp. 1325–1335, 2013.
  * [19] M. Lustig and J. M. Pauly, “Spirit: iterative self-consistent parallel imaging reconstruction from arbitrary k-space,” _Magnetic resonance in medicine_ , vol. 64, no. 2, pp. 457–471, 2010.
  * [20] S. Ramani and J. A. Fessler, “Parallel mr image reconstruction using augmented lagrangian methods,” _IEEE Transactions on Medical Imaging_ , vol. 30, no. 3, pp. 694–706, 2010.
  * [21] J. P. Haldar and J. Zhuo, “P-loraks: low-rank modeling of local k-space neighborhoods with parallel imaging data,” _Magnetic resonance in medicine_ , vol. 75, no. 4, pp. 1499–1514, 2016.
  * [22] K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson, T. Pock, and F. Knoll, “Learning a variational network for reconstruction of accelerated mri data,” _Magnetic resonance in medicine_ , vol. 79, no. 6, pp. 3055–3071, 2018.
  * [23] Y. Chen, T. Xiao, C. Li, Q. Liu, and S. Wang, “Model-based convolutional de-aliasing network learning for parallel mr imaging,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2019, pp. 30–38.
  * [24] K. Kwon, D. Kim, and H. Park, “A parallel mr imaging method using multilayer perceptron,” _Medical physics_ , vol. 44, no. 12, pp. 6209–6224, 2017.
  * [25] J. Schlemper, C. Qin, J. Duan, R. M. Summers, and K. Hammernik, “$sigma$-net: Ensembled iterative deep neural networks for accelerated parallel mr image reconstruction,” _arXiv preprint arXiv:1912.05480_ , 2019.
  * [26] J. Schlemper, J. Duan, C. Ouyang, C. Qin, J. Caballero, J. V. Hajnal, and D. Rueckert, “Data consistency networks for (calibration-less) accelerated parallel mr image reconstruction,” _arXiv preprint arXiv:1909.11795_ , 2019\.
  * [27] A. Sriram, J. Zbontar, T. Murrell, C. L. Zitnick, A. Defazio, and D. K. Sodickson, “Grappanet: Combining parallel imaging with deep learning for multi-coil mri reconstruction,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 315–14 322.
  * [28] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, and D. Liang, “Accelerating magnetic resonance imaging via deep learning,” in _IEEE International Symposium on Biomedical Imaging_ , 2016, pp. 514–517.
  * [29] C. Trabelsi, O. Bilaniuk, D. Serdyuk, S. Subramanian, J. F. Santos, S. Mehri, N. Rostamzadeh, Y. Bengio, and C. Pal, “Deep complex networks,” _ArXiv_ , vol. abs/1705.09792, 2018.
  * [30] Y. Chen, H. Fan, B. Xu, Z. Yan, Y. Kalantidis, M. Rohrbach, S. Yan, and J. Feng, “Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3435–3444.
  * [31] L. Sun, Z. Fan, X. Fu, Y. Huang, X. Ding, and J. Paisley, “A deep information sharing network for multi-contrast compressed sensing mri reconstruction,” _IEEE Transactions on Image Processing_ , vol. 28, no. 12, pp. 6141–6153, 2019.
  * [32] Y. Yang, J. Sun, H. Li, and Z. Xu, “Admm-csnet: A deep learning approach for image compressive sensing,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 3, pp. 521–538, 2020.
  * [33] Y. Han, J. Yoo, H. H. Kim, H. J. Shin, K. Sung, and J. C. Ye, “Deep learning with domain adaptation for accelerated projection-reconstruction mr,” _Magnetic resonance in medicine_ , vol. 80, no. 3, pp. 1189–1205, 2018.
  * [34] B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, and M. S. Rosen, “Image reconstruction by domain-transform manifold learning,” _Nature_ , vol. 555, no. 7697, pp. 487–492, 2018.
  * [35] D. Lee, J. Yoo, S. Tak, and J. C. Ye, “Deep residual learning for accelerated mri using magnitude and phase networks,” _IEEE Transactions on Biomedical Engineering_ , vol. 65, no. 9, pp. 1985–1995, 2018.
  * [36] T. M. Quan, T. Nguyen-Duc, and W.-K. Jeong, “Compressed sensing mri reconstruction using a generative adversarial network with a cyclic loss,” _IEEE Transactions on Medical Imaging_ , vol. 37, no. 6, pp. 1488–1497, 2018\.
  * [37] G. Yang, S. Yu, H. Dong, G. Slabaugh, P. L. Dragotti, X. Ye, F. Liu, S. Arridge, J. Keegan, Y. Guo _et al._ , “Dagan: Deep de-aliasing generative adversarial networks for fast compressed sensing mri reconstruction,” _IEEE Transactions on Medical Imaging_ , vol. 37, no. 6, pp. 1310–1321, 2017.
  * [38] M. Mardani, E. Gong, J. Y. Cheng, S. S. Vasanawala, G. Zaharchuk, L. Xing, and J. M. Pauly, “Deep generative adversarial neural networks for compressive sensing mri,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 1, pp. 167–179, 2018.
  * [39] T.-W. Ke, M. Maire, and S. X. Yu, “Multigrid neural architectures,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6665–6673.
  * [40] S. Nah, T. Hyun Kim, and K. Mu Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3883–3891.
  * [41] W. Ren, S. Liu, H. Zhang, J. Pan, X. Cao, and M.-H. Yang, “Single image dehazing via multi-scale convolutional neural networks,” in _Proceedings of the European Conference on Computer Vision_ , 2016, pp. 154–169.
  * [42] J. Li, F. Fang, K. Mei, and G. Zhang, “Multi-scale residual network for image super-resolution,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 517–532.
  * [43] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2117–2125.
  * [44] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2881–2890.
  * [45] F. W. Campbell and J. G. Robson, “Application of fourier analysis to the visibility of gratings,” _The Journal of physiology_ , vol. 197, no. 3, p. 551, 1968.
  * [46] R. L. DeValois and K. K. DeValois, _Spatial vision_. Oxford university press, 1990, vol. 14.
  * [47] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [48] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [49] J. Zbontar _et al._ , “fastMRI: An open dataset and benchmarks for accelerated MRI,” _arXiv preprint arXiv:1811.08839_ , 2018.
  * [50] M. Murphy, K. Keutzer, S. Vasanawala, and M. Lustig, “Clinically feasible reconstruction time for l1-spirit parallel imaging and compressed sensing mri,” in _Proceedings of the ISMRM Scientific Meeting & Exhibition_, 2010, p. 4854.

|  Chun-Mei Feng received her M.S. degree at QuFu Normal University, China, in
2018. She is currently a Ph.D. student of the school of Information Science
and Technology, Harbin Institute of Technology Shenzhen, China. Her research
interests include medical imaging and bioinformatics.
---|---
|  Zhanyuan Yang received his M.S. degree at University of Electronic Science
and Technology of China (UESTC), China, in 2020. His research interests are
computer vision based on deep learning, including image restoration and image
enhancement.
---|---
|  Huazhu Fu (SM’18) is a Senior Scientist at Inception Institute of
Artificial Intelligence, Abu Dhabi, United Arab Emirates. He received his
Ph.D. from Tianjin University in 2013, and was a Research Fellow at Nanyang
Technological University for two years. From 2015 to 2018, he was a Research
Scientist in Institute for Infocomm Research at Agency for Science, Technology
and Research. His research interests include computer vision, machine
learning, and medical image analysis. He currently serves as an Associate
Editor of IEEE Transactions on Medical Imaging, IEEE Journal of Biomedical and
Health Informatics, and IEEE Access. He also serves as a co-chair of OMIA
workshop and co-organizer of ocular image series challenge (i-Challenge).
---|---
|  Yong Xu (Senior Member, IEEE) received his B.S. and M.S. degrees at Air
Force Institute of Meteorology (China) in 1994 and1997, respectively. He then
received his Ph.D. degree in pattern recognition and intelligence system at
the Nanjing University of Science and Technology in 2005. From May 2005 to
April 2007, he worked at Harbin Institute of Technology Shenzhen as a post-
doctoral research fellow. Now he is a professor at HIT Shenzhen. His current
interests include pattern recognition, machine learning, and bioinformatics.
---|---
|  Jian Yang (Member, IEEE) received the PhD degree from Nanjing University of
Science and Technology (NUST), on the subject of pattern recognition and
intelligence systems in 2002. In 2003, he was a Postdoctoral researcher at the
University of Zaragoza. From 2004 to 2006, he was a Postdoctoral Fellow at
Biometrics Centre of Hong Kong Polytechnic University. From 2006 to 2007, he
was a Postdoctoral Fellow at Department of Computer Science of New Jersey
Institute of Technology. Now, he is a Chang-Jiang professor in the School of
Computer Science and Technology of NUST. He is the author of more than 200
scientific papers in pattern recognition and computer vision. His papers have
been cited more than 6000 times in the Web of Science, and 18000 times in the
Scholar Google. His research interests include pattern recognition, computer
vision and machine learning. Currently, he is/was an associate editor of
Pattern Recognition, Pattern Recognition Letters, IEEE Trans. Neural Networks
and Learning Systems, and Neurocomputing. He is a Fellow of IAPR.
---|---
|  Ling Shao (Fellow, IEEE) is the CEO and Chief Scientist of the Inception
Institute of Artificial Intelligence, Abu Dhabi, United Arab Emirates. He is
an Associate Editor of the IEEE TRANSACTIONS ON IMAGE PROCESSING, the IEEE
TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, the IEEE TRANSACTIONS ON
CIRCUITS ANDSYSTEM FOR VIDEO TECHNOLOGY, and several other journals. His
research interests include Computer Vision, Machine Learning and Medical
Imaging. He is a Fellow of the IEEE, the IAPR, the IET and the BCS.
---|---