# An Enhanced SCMA Detector Enabled by Deep Neural Network

Chao Lu, Wei Xu, Hong Shen, Hua Zhang, and Xiaohu You Email: {220170709, wxu,
shhseu, huazhang<EMAIL_ADDRESS>National Mobile Communications Research
Laboratory, Southeast University, Nanjing 210096, China

###### Abstract

In this paper, we propose a learning approach for sparse code multiple access
(SCMA) signal detection by using a deep neural network via unfolding the
procedure of message passing algorithm (MPA). The MPA can be converted to a
sparsely connected neural network if we treat the weights as the parameters of
a neural network. The neural network can be trained off-line and then deployed
for online detection. By further refining the network weights corresponding to
the edges of a factor graph, the proposed method achieves a better
performance. Moreover, the deep neural network based detection is a
computationally efficient since highly paralleled computations in the network
are enabled in emerging Artificial Intelligence (AI) chips.

###### Index Terms:

deep learning, neural network, SCMA, MPA.

##  I Introduction

The 5th generation (5G) wireless communication network aims at realizing
expectations including extremely heavy connectivity, considerable high
spectral efficiency, and ultra-low latency. Sparse code multiple access (SCMA)
is a typical non-orthogonal multiple access (NOMA) strategy for 5G. It has
been seen as a potential solution to address some of the critical requirements
of 5G. Even though SCMA can deal with the troublesome issues in 5G, it faces
two major challenges in terms of low-complexity detection [1] and efficient
codebook design [2].

Message passing algorithm (MPA) is the most popular approach to implement
multiuser detection with reduced complexity. Theoretically, it has been proved
that if there is no loop in the factor graph of a, e.g., SCMA schema, the MPA
gets exactly the marginal probabilities needed for detection [3], thus
achieving the maximum likelihood (ML) boundary. While there inevitably exists
loops, this algorithm is inherently suboptimal.

Recently, deep learning methods have been evidenced amazing progress in the
fields of computer vision, speech recognition and natural language processing.
Dramatic effects of deep learning have attracted a lot of attention. In
traditional research fields of wireless communication, deep learning has been
showing the promising ability to solve some specific problems [4, 5, 6, 7, 8,
9]. Some researches showed that deep learning methods are promising in
performance enhancement of decoding [4, 5]. Other researches treated a
communication system as an end-to-end encoder and decoder network [6]. In
addition, deep learning was proved successful in channel estimation by
treating the channel matrix as a 2D image [7].

Besides the above studies, the investigation of applying deep learning
approaches in wireless communication is still in its infancy. In this paper we
consider the multiuser detection problem in the scenario of SCMA. By unfolding
MPA and assigning weights to the edges of the factor graph, we construct a
sparsely connected neural network. After training the neural network offline,
we can achieve a better performance and the network can be deployed for online
detection.

Note that a parallel work was presented in [8] recently on the joint
optimization of constellation mapping and detection for SCMA network. [8]
achieves great performance gain compared with the conventional method since
the constellation mapping is also trained in the network which is crucial to
the system performance. However if the constellation mapping is given, its
neural network schema cannot outperform MPA. Or other, there will still be
some distance between the neural network method proposed in [8] and MPA if a
much more better constellation mapping is given. Different from [8], we only
study the detection problem, which means we fix the constellation mapping.
From the results, our network can outperform the MPA at high signal noise
ratio (SNR).

The remainder of the paper is organized as follows. In Section II, we describe
the system model of SCMA structure. Then in Section III, we present the neural
network based detection algorithm by transforming an MPA based detection
procedure. In Section IV, the symbol error rate (SER) of the MPA and the
neural network method is compared. Section V draws a conclusion of this work.

##  II SCMA Architecture

We consider a SCMA network where there are _J_ independent users and _K_
orthogonal resource blocks. For a NOMA system, we generally have _K_ $<$ _J_.
Each user has an _M_ -ary symbol set in which the symbols are assumed to be
independent with equal probability. The ${\log_{2}}M$ symbols are one-to-one
mapped to _K_ complex vectors by referring to a pre-designed table or codebook
in other words. The _K_ vectors all have _p_ non-zero elements and (_K-p_)
zero elements. Fig. 1 exemplifies a system architecture of 6 users and 4
resources. The 4 resources are defined by the 4 square boxes. The blank box
implies that the user has no symbol transmission assigned on this resource
element. Mathematically, this process can be expressed in the following
formula

Figure 1: 6-user-4-resource SCAM Architecture.

$\mathbf{y}=\sum\limits_{j=1}^{J}{{\mathop{\rm
diag}\nolimits}({\mathbf{h}_{j}}){\mathbf{x}_{j}}}+\mathbf{n},$ (1)

where $\mathbf{y}={[{y_{1}},{y_{2}},...,{y_{K}}]^{T}}$ represents the received
signal while
${\mathbf{x}_{j}}={[{x_{(j,1)}},{x_{(j,2)}},...,{x_{(j,K)}}]^{T}}$ represents
the _K_ -ary complex constellation signals of the _j_ -th user at the
transmitter. The channel of the _j_ -th user is denoted as
${\mathbf{h}_{j}}={[{h_{(j,1)}},{h_{(j,2)}},...,{h_{(j,K)}},]^{T}}$ and
$h_{(j,k)},k=1,2,...,K$ defines the channel of the _j_ -th user on the _k_ -th
resource. As we can see from (1), the signals of all _J_ users are added
together on every resource block after passing the channel. Here, $\mathbf{n}$
represents the additive white Gaussian noise (AWGN) caused by the thermal
noise of the amplifiers. The noise is a _k_ -ary vector and each element is
subjected to a Gaussian distribution with mean of 0 and variance of
${\sigma^{2}}$.

For efficient elaboration, we consider a typical SCMA network which has 6
users and 4 resources. As depicted in Fig. 2, the relationship between users
and resources can be expressed by a factor graph. The edges between user nodes
and resource nodes mean that these users have signals transmitted on these
resource blocks. In Fig. 2, every resource has ${d_{c}}=3$ conflicting user
signals. This means that the points of the constellation on every resource are
sparse, which makes it possible to ensure the Euclidean distance of every two
points large enough for reliable detection.

Figure 2: Typical factor graph.

Alternatively, the factor graph can be represented by an indicator matrix
$\mathbf{F}$. The factor graph in Fig. 2 equivalently corresponds to the
indicator matrix:

${\bf{F}}=\left[{\begin{array}[]{*{20}{c}}{\begin{array}[]{*{20}{c}}1&1&{\begin{array}[]{*{20}{c}}1&0&0\end{array}}&0\end{array}}\\\
{\begin{array}[]{*{20}{c}}1&0&{\begin{array}[]{*{20}{c}}0&1&1\end{array}}&0\end{array}}\\\
{\begin{array}[]{*{20}{c}}0&1&{\begin{array}[]{*{20}{c}}0&1&0\end{array}}&1\end{array}}\\\
{\begin{array}[]{*{20}{c}}0&0&{\begin{array}[]{*{20}{c}}1&0&1\end{array}}&1\end{array}}\end{array}}\right].$
(2)

Using $\mathbf{F}$, we define two sets,

$V(k){\rm{=\\{}}j|{\left[{\bf{F}}\right]_{k,j}}=1{\rm{\\}}},{\rm{}}k=1,{\rm{}}2,{\rm{}}\cdots,{\rm{}}K$
(3)

and

$C(j){\rm{=}}\left\\{{k|{{\left[{\bf{F}}\right]}_{k,j}}=1}\right\\},{\rm{}}j=1,{\rm{}}2,{\rm{}}\cdots,{\rm{}}J,$
(4)

where _V(k)_ represents the set of users that reuse the _k_ -th resource block
while _C(j)_ represents the set of the resource blocks that the _j_ -th user
occupies. _V(k)_ and _C(j)_ will be used for illustration in the following
part.

##  III From MPA to Neural Network

MPA calculates the marginal probabilities through iterations. In this work, we
replace the iteration steps by neural network layers. Next, we will explain
how MPA and our neural network work.

###  III-A Message Passing Detection Algorithm

The detection of SCMA is conducted by maximizing the posterior probability

${\hat{\bf{X}}}=\arg\mathop{\max}\limits_{{\bf{X}}\in{{\bf{X}}^{J,K}}}p({\bf{X}}|{\bf{y}}),$
(5)

where ${\bf{X}}=[{{\bf{x}}_{1}},{{\bf{x}}_{2}},...,{{\bf{x}}_{J}}]$ and
${{\bf{x}}_{j}}\in{{\bf{X}}_{j}},j=1,2,...,J$. ${{\bf{X}}_{j}}$ represents the
set of the _j_ -th user’s constellation signals, while ${{{\bf{X}}^{J,K}}}$ is
all the combinations of possible constellation signals from different users.
By calculating the marginal probability in (5), the decision formula for each
user can be written as

${{\hat{\bf{x}}}_{j}}=\arg\mathop{\max}\limits_{{{\bf{x}}_{j}}\in{{\bf{X}}_{j}}}\sum\limits_{{\bf{X}}\in{{\bf{X}}^{J,K}},{{\bf{x}}_{j}}=\left[{\bf{X}}\right]_{j}}{p({\bf{X}}|{\bf{y}})},$
(6)

where ${{\hat{\bf{x}}}_{j}}$ is the estimation obtained by using, e.g., MPA
for solving (6).

Figure 3: Structure of neural network, every blue square represents a block
and the block in red circle is named as pooling-concat block.

In MPA, the posterior probability is calculated iteratively. Two types of
messages are calculated and then exchanged in the factor graph. The message
transmitted from the _k_ -th resource to the _j_ -th user is defined by
$I_{{R_{k}}\to{u_{j}}}^{t}$. The message transmitted from the _j_ -th user to
the _k_ -th resource is defined by $Q_{{u_{j}}\to{R_{k}}}^{t}$. The
superscript _t_ represents the iteration index. The two types of messages are
updated according to the following equations.

$\displaystyle I_{{R_{k}}\to{u_{j}}}^{t}({{\bf{x}}_{j}})$
$\displaystyle=\sum\limits_{{{\bf{X}}^{V(k)\backslash
j}}}{p({y_{k}}{\rm{|}}{{\bf{x}}_{{j_{1}}}},{j_{1}}\in V(k))}$ (7)
$\displaystyle\quad\times\mathop{\prod}\limits_{{j_{2}}\in V(k)\backslash
j}Q_{{u_{{j_{2}}}}\to{R_{k}}}^{t-1}({{\bf{x}}_{{j_{2}}}})$
$\displaystyle=\sum\limits_{{{\bf{X}}^{V(k)\backslash
j}}}{\frac{1}{{\sqrt{2\pi{\sigma^{2}}}}}}\exp\left({-\frac{1}{{2{\sigma^{2}}}}{A_{k}}}\right)$
$\displaystyle\quad\times\mathop{\prod}\limits_{{j_{2}}\in V(k)\backslash
j}Q_{{u_{{j_{2}}}}\to{R_{k}}}^{t-1}({{\bf{x}}_{{j_{2}}}}),$

${A_{k}}{\rm{=}}{\left\|{{y_{k}}-\sum\limits_{{j_{1}}\in
V(k)}{{h_{(k,{j_{1}})}}{x_{(k,{j_{1}})}}}}\right\|^{2}},$ (8)

$Q_{{u_{j}}\to{R_{k}}}^{t}({{\bf{x}}_{j}})=p({{\bf{x}}_{j}})\mathop{\prod}\limits_{k^{\prime}\in
C(j)\backslash k}I_{{R_{k^{\prime}}}\to{u_{j}}}^{t-1}({{\bf{x}}_{j}}),$ (9)

where $C(j)\backslash k$ means the set of $C(j)$ after removing component _k_
and $V(k)\backslash j$ means the set of $V(k)$ after removing component _j_.
${{\bf{X}}^{V(k)\backslash j}}$ represents all possible combinations of
symbols sent by all other users in set $V(k)$ expect for the _j_ -th user.

Multiplication can be replaced by addition in logarithmic domain. Usually,
equations (7) and (9) can be rewritten as

$\displaystyle LI_{{R_{k}}\to{u_{j}}}^{t}({{\bf{x}}_{j}})$
$\displaystyle=\ln\biggl{(}\sum\limits_{{{\bf{X}}^{V(k)\backslash
j}}}\exp\Bigl{(}-\frac{1}{{2{\sigma^{2}}}}{A_{k}}+$ (10)
$\displaystyle\quad\sum\limits_{{j_{2}}\in V(k)\backslash
j}{LQ_{{u_{{j_{2}}}}\to{R_{k}}}^{t-1}({{\bf{x}}_{{j_{2}}}})}\Bigr{)}\biggr{)}+\beta$
$\displaystyle\approx\mathop{max}\limits_{{x_{j}}}\biggl{(}{-\frac{1}{{2{\sigma^{2}}}}{A_{k}}}+$
$\displaystyle\quad{\sum\limits_{{j_{2}}\in V(k)\backslash
j}{LQ_{{u_{{j_{2}}}}\to{R_{k}}}^{t-1}({{\bf{x}}_{{j_{2}}}})}}\biggr{)}+\beta,$

$LQ_{{u_{j}}\to{R_{k}}}^{t}({{\bf{x}}_{j}})=\ln(p({{\bf{x}}_{j}}))+\sum\limits_{k^{\prime}\in
C(j)\backslash k}{LI_{{R_{k^{\prime}}}\to{u_{j}}}^{t-1}({{\bf{x}}_{j}})},$
(11)

where $\beta=\ln(1/\sqrt{2\pi{\sigma^{2}}})$ is a constant value.

Figure 4: Pooling-concat block, small bold color lines are the inputs of the
network.

###  III-B Neural Network Architecture

Regarding for computations of MPA, it is possible to replace manipulations
with the propagation of a neural network. We consider a deep neural network of
4 blocks and each block consists of two layers divided by imaginary lines, as
depicted in Fig. 3. Let us focus on the major computation steps in (10) and
(11) of MPA. Each iterative computations of (10) and (11) can be unfolded as
the propagation between two layers in a block. The pooling-concat block shown
in Fig. 4. is calculated according to

$\displaystyle LI_{{R_{k}}\to{u_{j}}}^{2(l+1)}({{\bf{x}}_{j}})$
$\displaystyle=\mathop{\max}\limits_{{{\bf{X}}^{V(k)\backslash
j}}}\biggl{(}-\frac{{{c_{2(l+1),{j_{1}},j,k}}}}{{2{\sigma^{2}}}}{A_{k}}+$ (12)
$\displaystyle\quad\sum\limits_{{j_{2}}\in V(k)\backslash
j}{{w_{2(l+1),{j_{2}},j,k}}LQ_{{u_{{j_{2}}}}\to{R_{k}}}^{2l+1}({{\bf{x}}_{{j_{2}}}})}\biggr{)}$
$\displaystyle\quad+{a_{j,k}}\beta.$

In Fig. 2, ${d_{c}}$ is equal to 3, so $V(k)\backslash j$ has 2 elements. As
we can see in Fig. 4, the pooling-concat block has 2 inputs which denote
$LI_{{R_{k}}\to{u_{j}}}^{2l+1}$. When _M_ is set to 4, each user will have 4
symbols, thus each input of the pooling-concat block has 4 nodes. The small
bold color lines linked to the neurons before max-pooling operation are the
inputs of calculated ${A_{k}}$. As for $\beta$ in (12), it exists as an offset
in a neuron. The neurons assign different weights to different inputs and
finally output their summation. The outputs of the neurons are followed by a
maximum operation and the results will be merged to a 4-nodes chunk since
${{\bf{x}}_{j}}$ has 4 values.

The first layer in the block is calculated according to

$\displaystyle LQ_{{u_{j}}\to{R_{k}}}^{2l{\rm{+}}1}({{\bf{x}}_{j}})$
$\displaystyle={b_{2l{\rm{+}}1,j,k}}\ln\left({p({{\bf{x}}_{j}})}\right)+$ (13)
$\displaystyle\quad\sum\limits_{k^{\prime}\in C(j)\backslash
k}{{w_{2l{\rm{+}}1,k^{\prime},j,k}}LI_{{R_{k^{\prime}}}\to{u_{j}}}^{2l}({{\bf{x}}_{j}})}.$

The front 4 chunks serve as placeholders to store the inputs from the previous
block, usually $LQ_{{u_{j}}\to{R_{k}}}^{(2l+1)}$. The outputs of the
placeholders are all set to $\ln\left(1/M\right)$ if there are no inputs such
as the first block. Similar to the pooling-concat block, the neurons of the
back 6 chunks also output the weighted summation of their inputs.

In (12) and (13), it can be seen that we do not apply any activation functions
which are often used to ensure the non-linear of the neuron output. To ensure
the equivalence with MPA, we just let the linear combination of the inputs of
neurons become the outputs.

As for the output layer, we follow the equation (14) to calculate the output
logits of the network, which is also applied in MPA.

$L{Q_{{u_{j}}}}({\bf{x}}_{j})=\ln\left({p({{\bf{x}}_{j}})}\right)+\sum\limits_{k\in
C(j)}{LI_{{R_{k}}\to{u_{j}}}^{L}({{\bf{x}}_{j}})}$ (14)

Finally, we can decide the estimate symbol by

${\bf{x}}{{}^{\prime}_{j}}=\arg\mathop{\max}\limits_{{{\bf{x}}_{j}}}L{Q_{{u_{j}}}}({{\bf{x}}_{j}}){\rm{,}}\quad
j=1,{\rm{}}2,\cdots,{\rm{}}J.$ (15)

After designing the structure of the neural network, we construct the
following loss function as our optimization objective for network training:

$Loss={\rm{E}}\left\\{\sum\limits_{j}{{{I}}({{\bf{x}}_{j}},{\bf{x}}_{j}^{\prime})}\log(\frac{{\exp(L{Q_{{u_{j}}}}({{\bf{x}}_{j}}))}}{{\sum\limits_{{{\bf{x}}_{j^{\prime}}}}{\exp(L{Q_{{u_{j}}}}({{\bf{x}}_{j^{\prime}}}))}}})\right\\},$
(16)

where the function ${{I}}({{\bf{x}}_{j}},{\bf{x}}_{j}^{\prime})$ is an
indicator function and it has the following form

${I}({{\mathbf{x}}_{j}},{\mathbf{x}}_{j}^{\prime})=\left\\{{\begin{array}[]{*{20}{c}}{1,\quad{{\mathbf{x}}_{j}}={\mathbf{x}}_{j}^{\prime}}\\\
{0,\quad{{\mathbf{x}}_{j}}\neq{\mathbf{x}}_{j}^{\prime}}\end{array}}\right.,$
(17)

and ${\rm{E}}\\{\cdot\\}$ represents mathematical expectation.

This loss function can be regarded as a softmax cross entropy. More
specifically, the 4 neurons’ outputs are normalized by a softmax function
separately. Then, the normalized results are used to calculate the cross
entropy. Since it is impossible to calculate the mathematical expectation
during training, we can replace the function ${\rm{E}}\\{\cdot\\}$ by batch
average mean function.

##  IV Numerical Experiments

In this section, we describe all the key details of our experimental tests.
All of our the simulation uses the codebook proposed in [10] and its
constellation graph is given in Fig. 5. In our experiment, the SCAM network
has 6 users and every user has a set of 4 symbols. This corresponds to the
output layer with 24 neurons.

Figure 5:  Constellations on 4 resources.

###  IV-A Neural Network Training

We build our neural network by the TensorFlow [11] framework. TensorFlow is an
open source machine learning library provided by Google. Because of its
encapsulation, ease of use, and free for academic research, we choose this
framework to implement our neural network.

In order to reduce the training time and achieve a better result, it is
important to give the neural network a good initialization. We initialize all
the parameters with the same non-zero value such as all-in-one. We set the
initial learning rate value to 0.001, and then select the Adam [12] optimizer
to optimize all network parameters. It is much more suitable to use such an
advanced gradient optimizer because this optimizer can adjust learning rate to
learning steps while a raw stochastic gradient optimizer cannot. We trained
two neural networks with 2 blocks and 4 blocks, respectively. Both the
networks are trained under the AWGN channel at a predetermined SNR. But after
training, these networks effectively work at an arbitrary SNR.

###  IV-B Dataset

Here, we consider the SCMA setup where _J_ is 6, _K_ is 4 and _M_ is 4. The 6
users have 4096 signal combinations in total. We generate the training data
dynamically which means the noises are all different at each step. The mini
batch feed to the network is fixed to 4096 (the total combinations) and the
training data is generated at a fixed SNR (16dB). We generate the test data in
the same way, but at 3dB intervals.

Figure 6: SER of MPA with different iterations at AWGN.

Figure 7: SER of MPA and neural network at AWGN.

Figure 8: SER of MPA and neural network at AWGN.

###  IV-C Results

In Fig. 6, we compare the SER performance of traditional MPA with various
iteration numbers. As is shown in Fig. 6, with the growth of iteration
numbers, the performance gets better. On the other hand, the performance gain
becomes less with the increase of iteration numbers. When the iteration
numbers come to 4, the algorithm performance almost converges.

For comparing, we give the SER performance of MPA and neural network method in
Fig. 7 and Fig. 8. The neural network blocks are set to 2(4) when the MPA
iterations are set to 2(4) for a fair comparison. Obviously, the neural
network method show the potential to achieve a better performance at high SNR.
From Fig. 7 we can see that the curves of the two method intersect together at
15 dB. After that point, the curve of the neural network method shows some
improvement. Similarly, this phenomenon also appears in Fig. 8. In other
words, the neural network can find a better solution of this problem after
training.

##  V Conclusion

In this article, we propose a neural network architecture for SCMA detection.
The result of this method converges faslty when the number of iterations is up
to four. We show how to unfold the MPA to a neural network in detail. By
adding different weights to the edges of the factor graph, the MPA is
represented as a sparsely connected neural network. This neural network
outperforms the traditional MPA especially at high SNR. Moreover, the neural
network based method can be accelerated by GPU or AI processor, which also
makes sense.

## Acknowledgment

This work was support by the National Natural Science Foundation of China
under Grants 61471114, 61521061, 61501110, 61601115, U1534208, Six talent
peaks project in Jiangsu Province under GDZB-005, the Natural Science
Foundation of Jiangsu Province under Grant BK20150635, and the Fundamental
Research Funds for the Central Universities under Grant 2242014K40037.

## References

  * [1] F. Wei and W. Chen, “A low complexity SCMA decoder based on list sphere decoding,” in _Proc. IEEE GLOBECOM_ , Washington, DC, USA, Dec. 2016 , pp. 1–6.
  * [2] M. Alam and Q. Zhang, “Performance study of SCMA codebook design,” in _Proc. IEEE WCNC_ , San Francisco, CA, Mar. 2017, pp. 1–5.
  * [3] F. R. Kschischang, B. J. Frey, and H. A. Loeliger, “Factor graphs and the sum-product algorithm,” _IEEE Trans. Inf. Theory_ , vol. 47, no. 2, pp. 498–519, Feb. 2001.
  * [4] E. Nachmani, Y. Be’ery, and D. Burshtein, “Learning to decode linear codes using deep learning,” in _Proc. 54th Annual Allerton Conference on Communication, Control, and Computing_ , Monticello, IL, Sept. 2016, pp. 341–346.
  * [5] T. Gruber, S. Cammerer, J. Hoydis, and S. t. Brink, “On deep learning-based channel decoding,” in _Proc. 51st Annual Conference on Information Sciences and Systems_ , Baltimore, MD, Mar. 2017, pp. 1–6.
  * [6] T. J. O’Sheam and J. Hoydis, “An introduction to machine learning communication systems,” _arXiv preprint arXiv:1702.00832_ , 2017.
  * [7] C. K. Wen, W. T. Shih, and S. Jin, “Deep learning for massive MIMO CSI feedback,” _IEEE Wireless Commun. Lett._ , Mar. 2018, early access.
  * [8] M. Kim, N. I. Kim, W. Lee, and D. H. Cho, “Deep learning-aided SCMA,” _IEEE Commun. Lett._ , vol. 22, no. 4, pp. 720–723, Apr. 2018.
  * [9] H. Ye, G. Y. Li, and B. H. Juang,“Power of deep learning for channel estimation and signal detection in OFDM systems,” _IEEE Wireless Commun. Lett._ , vol. 7, no. 1, pp. 114–117, Sept. 2018.
  * [10] Qualcomm Inc. “RSMA and SCMA comparison,” _3GPP R1–164689_ , May 23–27, 2016.
  * [11] M. Abadi et al., “Tensorflow: Large-scale machine learning on heterogeneous systems,” _arXiv preprint arXiv:1603.04467_ , 2016.
  * [12] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv perprint arXiv: 1412.6980_ , 2014.


11institutetext: Institute of Medical Informatics, Universität zu Lübeck,
Lübeck, Germany
11email<EMAIL_ADDRESS>
https://github.com/MDL-UzL/MICCAI24_IMMoCo.git

# IM-MoCo: Self-supervised MRI Motion Correction using Motion-Guided Implicit
Neural Representations

Ziad Al-Haj Hemidi  Christian Weihsbach  Mattias P. Heinrich

###### Abstract

Motion artifacts in Magnetic Resonance Imaging (MRI) arise due to relatively
long acquisition times and can compromise the clinical utility of acquired
images. Traditional motion correction methods often fail to address severe
motion, leading to distorted and unreliable results. Deep Learning (DL)
alleviated such pitfalls through generalization with the cost of vanishing
structures and hallucinations, making it challenging to apply in the medical
field where hallucinated structures can tremendously impact the diagnostic
outcome. In this work, we present an instance-wise motion correction pipeline
that leverages motion-guided Implicit Neural Representations (INRs) to
mitigate the impact of motion artifacts while retaining anatomical structure.
Our method is evaluated using the NYU _fastMRI_ dataset with different degrees
of simulated motion severity. For the correction alone, we can improve over
state-of-the-art image reconstruction methods by $+5\%$ SSIM, $+5\>db$ PSNR,
and $+14\%$ HaarPSI. Clinical relevance is demonstrated by a subsequent
experiment, where our method improves classification outcomes by at least
$+1.5$ accuracy percentage points compared to motion-corrupted images.

###### Keywords:

Motion Correction Reconstruction Implicit Neural Representations Magnetic
Resonance Imaging

##  1 Introduction

Magnetic Resonance Imaging (MRI) is a non-invasive imaging technique providing
detailed images of soft-tissue structures. However, common artifacts,
particularly motion-related ones, degrade image quality, impacting clinical
diagnosis and leading to high clinical costs [23].

Various retrospective strategies for motion correction (MoCo) have been
developed [4, 24]. Retrospective correction handles motion artifacts post-
acquisition, offering flexibility without requiring scanner modifications.
This approach benefits from prior knowledge such as from navigators [7], and
from the imaged object itself in so-called autofocus [3] or in data-
consistency (DC) [9] methods. Unfortunately, these methods face challenges in
solving poorly conditioned, non-convex optimization problems with high
computational complexity. Image-based deep learning (DL) models, like
convolutional neural networks (CNNs) [1, 14] and generative adversarial
networks [2], have been employed for direct motion-refined image estimation to
overcome the challenges mentioned earlier. While promising, image-based DL-
based methods can be unstable, struggle with pathology preservation, and
introduce undesired alterations or hallucinations [4], raising concerns about
their suitability for clinical applications. Specific approaches break down
the problem into intermediate steps to address instability, maintain DC, and
alleviate network hallucinations. For instance, CNNs are employed to identify
subsets of k-space lines affected by motion, which can either be excluded [21]
or down-weighted [5] in the context of a DC-based reconstruction. However,
these methods do not explicitly model motion (transformations) during
reconstruction, potentially compromising quality with many motion-corrupted
lines. Efforts to improve efficiency integrate image-based MoCo CNNs into DC-
based [8] or autofocus [15] iterative algorithms. Alternatively, they replace
the optimization-based motion estimation with a DL network and incorporate
this information into the reconstruction process [10]. However, they rely on
supervised training and require large paired datasets, which are difficult to
obtain in the medical domain. Recently, Implicit neural representations (INRs)
were used to model images as a function, spurring significant interest in
instance-optimization-based reconstruction [19]. INRs have proven successful
in self-supervised MRI reconstruction, addressing issues present in prior
methods [6, 11]. Despite this, the potential of INRs for MoCo remains
untapped.

In this work, we introduce a novel MoCo pipeline utilizing INRs to address
motion artifacts effectively while preserving anatomical structures in MRI
scans. Our contributions are 1) Developing a motion-guided INR-based method
called IM-MoCo, combining a k-space line motion detection network ($k$lD-Net),
an Image INR, and a motion-INR. 2) Demonstrating its efficacy on the NYU
_fastMRI_ dataset [13, 25] through significant reductions in simulated motion-
induced distortions compared to competing methods. 3) Highlighting its
potential by improving pathology classification of _fastMRI+_ annotations
[26]. These findings position IM-MoCo as a promising advancement for enhancing
MRI quality for scans with light and heavy motion corruption.

Figure 1: IM-MoCo pipeline. The pre-trained $k$lD-Net takes a motion-corrupted
$k$-space and outputs a motion mask, which is post-processed to yield a list
of movement groups indicated by the different colors in $S$ for better
visualization. The number of movements and corresponding lines guide the
Motion INR and the Image INR. The Image INR predicts the motion-free image and
the Motion INR is used for guidance by optimizing the forward motion yielding
a motion-corrupted $k$-space. The discrepancy between the motion-corrupted and
the measured $k$-space is minimized using the data-consistency (DC) loss. The
gradient entropy is a denoiser on the Image-INR output to impose crisp image
priors. The final motion-corrected image is the output of the Image INR.

##  2 Materials & Methods

An overview of the proposed IM-MoCo pipeline is depicted in Fig. 1. Our
pipeline comprises three steps: 1) Motion Detection with $k$LD-Net, 2)
Generation of motion-free intensity priors with an Image INR, and 3) Forward
application of predicted motion from a Motion INR.

###  2.1 Physical Model & Motion Simulation

The physical forward model, $K=\mathbf{F}I+\epsilon$, of a 2D rigid motion-
induced MRI acquisition $K\in\mathbb{C}^{N_{x}\times N_{y}\times N_{c}}$ with
$N_{x}$ and $N_{y}$ being the number of points in the frequency and phase
encoding directions, respectively, and $N_{c}$ being the number of coils, can
be described as follows:

$K=\displaystyle\sum_{t=1}^{T}S_{t}\odot\mathcal{F}\mathbf{C}\mathbf{M}_{t}I,$
(1)

where $I\in\mathbb{C}^{N_{x}\times N_{y}\times N_{c}}$ is the desired motion
free image, $\mathbf{M}_{t}$ is the motion transform at time point $t$,
$\mathbf{C}\in\mathbb{C}^{N_{x}\times N_{y}\times N_{c}}$ are the coil
sensitivity maps, $\mathcal{F}$ is the Fourier transform, $\odot$ a sampling
operator, and $S_{t}$ is the sampling mask at time point $t$. The motion
transform $\mathbf{M}_{t}$ combines rotation and planar translations in $x$
and $y$ directions. The sampling mask $S_{t}$ is a binary mask indicating the
$k$-space lines acquired at time $t$. Equation (1) describes motion in the
image domain. However, the forward model can also be formulated in $k$-space
as it is mathematically equivalent [18]. Irrespective of the space in which
motion is described, for the simulation of motion, it is crucial to
synchronize the sampling of motion $S_{t}$ with the MR $k$-space filling
scheme to simulate realistic artifacts accurately. This work follows
previously developed methods [16] and relies on the simulation of rigid-body
motion in 2D single-coil cartesian sampled $k$-spaces filled sequentially
left-to-right.

###  2.2 IM-MoCo

####  2.2.1 Motion Detection.

In the first step of our pipeline, we present a $k$-space Line Detection
Network [5, 21] ($k$LD-Net) to detect motion-corrupted lines in raw $k$-space
data following a left-to-right sequential cartesian sampling scheme. The
$k$LD-Net, built on a U-Net [22], takes a complex-valued motion-corrupted
$k$-space as input, where the real and imaginary parts are concatenated in the
channel dimension, and outputs a binary mask predicting the motion-corrupted
lines denoted as $S$ in Eq. (1). The four layers of the U-Net consist of
convolutions with a kernel size of $3\times 3$, batch normalization, ReLU
activation, and an average pooling layer with kernel size $2\times 2$. The
number of channels (starting from $16$) is doubled after each down-sampling
operation and halved after each up-sampling operation. A final $1\times 1$
convolution outputs the predicted mask $\hat{S}$. The network is trained with
the binary cross entropy loss with logits between the predicted and ground
truth masks. During inference, the raw prediction of the network is activated
with $sigmoid$ and thresholded ($>0.5$) to yield a binary prediction mask. We
consider a line corrupted if $20\%$ of its frequencies are classified as so,
and we generate an index list, e.g., $[1,1,0,0,0,1,1,1]$, where $1$ indicates
a motion-corrupted line and $0$ a motion-free one. We then post-process it to
yield a list of movement groups, where each group is a list of line indices
for one movement, e.g., $[[1,1,0,0,0,0,0,0],[0,0,0,0,0,1,1,1]]$ for the
previously mentioned example. The following steps use the number of movements
and corresponding lines to guide the motion correction process.

Motion Correction. We use $\hat{S}$ from the $k$LD-Net (step 1) in a MoCo
model, which is a combination of an Image INR (step 2) that acts as an
implicit image prior and a Motion INR (step 3) to search for motion that
satisfies the forward model (see Sec. 2.1). We can rewrite the forward model
$\mathbf{F}$ from Eq. (1), for single-coil scans, as
$\mathbf{F}_{\theta,\Psi}$ as follows:

$\hat{K}=\displaystyle\sum_{t=1}^{T}\hat{S}_{t}\odot\mathcal{F}\;\text{INR}_{\theta}(h_{\mathbf{M}}(n_{\mathbf{M}},\mathbf{x}))_{t}\;\text{INR}_{\Psi}(h_{I}(\mathbf{x})),$
(2)

where $\text{INR}_{\Psi}$ is the image INR with optimizable parameters $\Psi$,
$h_{I}(\mathbf{x})$ is an encoding of the 2D coordinate $\mathbf{x}$,
$\text{INR}_{\theta}$ is the motion INR with optimizable parameters $\theta$,
$h_{\mathbf{M}}(n_{\mathbf{M}},\mathbf{x})$ is an encoding of the 2D
coordinate $\mathbf{x}$ and $n_{\mathbf{M}}$, which is a sequence of linearly
spaced numbers from $-1$ to $1$ representing the number of movement groups.
$\hat{S}_{t}$ is the predicted motion mask at time $t$. Each INR is built from
MPLs taking encodings as input. To encode our data, we use hash grid encoding
[20] for $h_{I}s(\mathbf{x})$ and $h_{\mathbf{M}}(n_{\mathbf{M}},\mathbf{x})$.
This encoding introduces learnable feature grids that enable a speed-up by
replacing large MLPs with a look-up table of feature vectors and a much
smaller MLP. The Image INR consists of three layers with $256$ channels and
ReLU activations, while the Motion INR has three layers with $64$ channels and
tanh activations. The $\text{INR}_{\Psi}$ predicts a 2-channel complex-valued
intensity image, while $\text{INR}_{\theta}$ predicts $n$ transformation grids
for $n$ movements.

In each iteration of the pipeline, we optimize the $\text{INR}_{\theta}$ and
$\text{INR}_{\Psi}$ in an end-to-end fashion by feeding the encoding and the
mask $\hat{S}$ through the forward model $\mathbf{F}_{\theta,\Psi}$ which in
term predicts a motion-free intensity image $\hat{I}$, applies the motion to
the intensity image resulting in a motion corrupted image that is transferred
to the $k$-space domain by the Fourier transform yielding $\hat{K}$. Using the
DC-loss in k-space, we measure the discrepancy to the acquired $k$-space
$K_{acq}$. We use the gradient entropy on $\hat{I}$ for regularization in
image-space. The motion correction loss is then defined as follows:

$\mathcal{L}=\underbrace{\frac{1}{N}\sum_{i=1}^{N}||K_{acq,i}-\hat{K}_{i}||_{2}^{2}}_{\text{DC-
loss
in}\;k\text{-space}}+\underbrace{\lambda(-\sum_{i=1}^{N}\nabla\hat{I}_{i}\cdot\log(\nabla\hat{I}_{i}))}_{\text{Regularization
in image-space}},$ (3)

where $N$ is the number of data points, $\nabla$ is the gradient operator, and
$\lambda$ is the regularization weight. The regularization is used to
introduce image priors to the Image INR as a denoiser, forcing the INR to
learn an artifact-free image. To ensure better motion estimates, high-
frequencies are suppressed at the beginning of the optimization by setting a
large weight $\lambda$ and halving it every $s$ step after $i$ iterations
[17]. The model is optimized until convergence, and the final motion-corrected
image is the output of $\text{INR}_{\Psi}$.

##  3 Experimental Results

###  3.1 Datasets & Motion Simulation

To evaluate our method, we use the MRI $T_{2}$-weighted brain $k$-space data
from the open-source NYU _fastMRI_ database [13, 25] in our first experiment.
We employed $300$ 2D slices from different patients and split them into
$200$/$50$/$50$ for training/validation/testing. We cropped the images to
$320\times 320$ and combined the coils. The second dataset contains
$T_{1}$-weighted and FLAIR brain _fastMRI_ scans, which are part of the
_fastMRI+_ [26] annotations and were used for the downstream classification
task. We classify two pathologies, _Nonspecific White Matter Lesion_ and
_Craniotomy_ , resulting in $1116$ slices of $60$ subjects with a total of
$2851$ annotations. We split the slices into $889$ ($1460$ annotations) for
training, $174$ ($467$ annotations) for validation, and $50$ ($141$) for
testing. For all experiments, two motion scenarios were simulated [16] with
random rotations and translations between $-10$ and $+10$ in $\degree$ and
$mm$, respectively. In the Light Motion scenario between $6$ and $10$
movements were introduced during simulation, whereas the Heavy Motion scenario
contained between $16$ and $20$ movements.

###  3.2 Experimental Setup

Experiment I: Motion Correction in Simulated Data. Here the image correction
quality is assessed. We implemented the following methods: $1)$ AF [3], $2)$
U-Net [1], $3)$ AF+ [15], $4)$ IM-MoCo (ours). AF is an instance-wise
autofocusing approach and optimizes motion parameters using Adam for $100$
iterations. The U-Net is a population-trained model, mapping motion-corrupted
images to motion-free ones, for $200$ epochs using Adam with $3e^{-4}$
learning rate and an L1-SSIM loss. AF+ combines AF and the U-Net for deep
image priors, trained with default settings using the available codebase [15].
IM-MoCo is optimized for $200$ iterations using Adam with learning rates of
$1e^{-2}$ for the INRs and an initial $\lambda=1e^{-2}$, halved every $10th$
step after $100$ iterations. All methods were evaluated on the
$T_{2}$-weighted $50$ corrupted test scans for the two motion scenarios.
Results were evaluated using the structural similarity index (SSIM), peak-
signal-to-noise-ratio (PSNR) and the Haar Perceptual Similarity Index
(HaarPSI) [12].
Experiment II: Downstream Classification Task. In this experiment, we
demonstrate clinical relevance with a classification of pathologies in motion
and motion-corrected images. We extract patches of size $124\times 124$ around
bounding boxes of the _fastMRI+_ dataset and normalized between $[0,1]$. We
used a pre-trained _ResNet18_ (PyTorch
Hub222https://pytorch.org/hub/pytorch˙vision˙resnet/) backbone for feature
extraction and added two trainable linear layers as the classification head.
Using the SGD optimizer, we trained the model for $100$ epochs with a batch
size of $20$ and a learning rate of $1e^{-3}$. The trained model is then
applied to the test set. We report the SSIM, PSNR, and HaarPSI for extracted
patches around the bounding boxes and accuracy for the classification.
Implementation Details. The $k$LD-Net was population-trained with Adam
(learning rate of $1e^{-4}$) for $4200$ epochs and a batch size of $4$. We
used PyTorch for implementations and an NVIDIA GeForce RTX 2080 TI with 12 GB
of VRAM. The code is publicly available.

###  3.3 Results

Experiment I: Motion Correction in Simulated Data. Table 1 shows quantitative
results from the motion correction experiment. For light and heavy motion,
corrupted images yield SSIM values of $87\%$ and $74\%$, PSNR values of
$28\>db$ and $24\>db$, and HaarPSI values of $70\%$ and $56\%$, respectively.
AF and U-Net improve SSIM by $7\%$ and $10\%$, PSNR by $5\>db$ and $3\>db$,
and HaarPSI by $18\%$ and $16\%$, respectively. However, AF+ worsens results
by ca. $2\%$ across all metrics. Compared to the best baseline, AF, our method
improves SSIM/HaarPSI by $>4\%$ for SSIM, $>9\%$ for HaarPSI, and $>10\>db$
for PSNR. A qualitative comparison is shown in Fig. 2.

Figure 2: The visualization shows the median results of motion-corrected
images of our IM-MoCo pipeline besides motion-corrupted, ground truth, and
comparison methods. The first and third rows show the light and heavy
correction results, respectively. The second and fourth rows show the residual
error images. Table 1:  Quantitative results of experiment I: motion
correction for all methods and motion scenarios. We report the results as mean
$\pm$ standard deviation over the test set. The arrows indicate the direction
of improvement.

Scenario | Method | SSIM$\uparrow$ | PSNR (db)$\uparrow$ | HaarPSI$\uparrow$
---|---|---|---|---
Light | Motion corrupted | $87.26\pm 4.42$ | $28.34\pm 2.97$ | $70.48\pm 8.69$
|  AF [3] | $94.47\pm 2.06$ | $33.91\pm 2.37$ | $88.49\pm 4.11$
|  U-Net [1] | $91.39\pm 2.14$ | $30.58\pm 2.33$ | $81.58\pm 4.49$
|  AF+ [15] | $85.18\pm 4.75$ | $27.93\pm 2.79$ | $70.82\pm 8.83$
| IM-MoCo (ours) | $\mathbf{98.25\pm 1.25}$ | $\mathbf{40.06\pm 3.33}$ | $\mathbf{97.20\pm 4.05}$
Heavy | Motion corrupted | $74.06\pm 6.36$ | $24.28\pm 2.50$ | $56.56\pm 8.68$
|  AF [3] | $87.19\pm 3.51$ | $29.84\pm 2.32$ | $78.91\pm 6.21$
|  U-Net [1] | $84.55\pm 3.63$ | $27.40\pm 2.34$ | $72.72\pm 5.69$
|  AF+ [15] | $70.61\pm 8.30$ | $24.12\pm 2.67$ | $56.91\pm 10.35$
| IM-MoCo (ours) | $\mathbf{92.77\pm 3.59}$ | $\mathbf{33.06\pm 3.59}$ | $\mathbf{87.29\pm 9.38}$

Experiment II: Downstream Classification Task.

The results are summarized in Table 2. Motion-free images achieve an accuracy
of $97\%$. For corrupted images, the accuracy is $96\%$ for light and $94\%$
for heavy motion, which is due to image quality loss as reflected in SSIMs of
$89\%$ and $77\%$, respectively. The U-Net’s accuracy is $90\%$ for light and
$88\%$ for heavy while its SSIMs are $87\%$ and $79\%$. IM-MoCo achieves an
accuracy of $97\%$ for light and $96\%$ for heavy motion, with the highest
SSIMs of $99\%$ and $95\%$, respectively.

Table 2:  Quantitative results of experiment II: image quality and
classification accuracy improvements in patches of the _fastmri+_ annotations.
The arrows indicate the direction of improvement.

Scenario | Method | SSIM$\uparrow$ | PSNR (db)$\uparrow$ | HaarPSI$\uparrow$ | Accuracy$\uparrow$
---|---|---|---|---|---
Motion-free | n.A. | n.A. | n.A. | n.A. | $97.16$
Light | Motion corrupted | $89.93\pm 4.67$ | $28.29\pm 4.07$ | $76.12\pm 9.95$ | $96.32$
|  U-Net [1] | $87.37\pm 4.31$ | $25.80\pm 2.61$ | $70.87\pm 7.74$ | $90.44$
| IM-MoCo (ours) | $\mathbf{99.00\pm 1.82}$ | $\mathbf{44.82\pm 6.44}$ | $\mathbf{97.33\pm 5.56}$ | $\mathbf{97.06}$
Heavy | Motion corrupted | $77.03\pm 5.74$ | $23.56\pm 2.18$ | $58.87\pm 6.20$ | $94.12$
|  U-Net [1] | $79.45\pm 4.50$ | $23.70\pm 2.18$ | $59.82\pm 5.88$ | $88.24$
| IM-MoCo (ours) | $\mathbf{95.26\pm 3.31}$ | $\mathbf{34.56\pm 5.61}$ | $\mathbf{88.48\pm 7.80}$ | $\mathbf{96.32}$

##  4 Discussion & Outlook

Experiment I demonstrated that our method effectively enhanced the quality of
motion-corrupted images, surpassing comparison methods, which is mainly
contributed by the implicit image priors and DC with the acquired $k$-space in
the forward motion model. Notably, our approach exhibited robustness in
addressing heavier motion scenarios as evidenced in Fig. 2 and Table 1.
However, the efficacy of our method hinges on the accurate detection of the
$k$lD-Net, which is contingent upon the acquisition sequence and motion
simulation pattern. In Experiment II, we showcased the performance enhancement
of a classification task compared to both corrupted and U-Net-corrected images
as indicated in Table 2. This improvement in classification accuracy can be
attributed to the prevention of overfitting on "healthy" features in instance-
optimization methods in contrast to population training ones like the U-Net.
Although our work primarily revolved around 2D rigid motion, we are confident
in the method’s potential for extension to more complex motion and multi-coil
data. This can be achieved by incorporating coil sensitivity map estimation
following the third step of the pipeline. Future investigations will explore
the utilization of different sampling patterns, such as radial sampling, and
the adaptation of the motion detection network to these patterns or pattern-
independent methods. While real motion cases were not tested in this study, we
are optimistic regarding the adaptability of our method through adjustments to
the motion detection network.

##  5 Conclusion

We have introduced IM-MoCo, a pipeline that utilizes motion-guided INRs to
mitigate the impact of motion artifacts in MRI scans. We tested our method on
the _fastMRI_ dataset and observed an improvement in image quality and
pathology classification performance. Our results suggest that IM-MoCo is a
promising solution for enhancing the quality of MRI scans in the presence of
challenging motion artifacts. In the future, we plan to investigate the
applicability of our method to real motion to validate its potential for
clinical applications.

## References

  * [1] Al-Masni, M.A., Lee, S., Yi, J., Kim, S., Gho, S.M., Choi, Y.H., Kim, D.H.: Stacked U-Nets with self-assisted priors towards robust correction of rigid motion artifact in brain MRI. NeuroImage 259, 119411 (2022)
  * [2] Armanious, K., Jiang, C., Fischer, M., Küstner, T., Hepp, T., Nikolaou, K., Gatidis, S., Yang, B.: MedGAN: Medical image translation using GANs. Computerized medical imaging and graphics 79, 101684 (2020)
  * [3] Atkinson, D., Hill, D.L., Stoyle, P.N., Summers, P.E., Clare, S., Bowtell, R., Keevil, S.F.: Automatic compensation of motion artifacts in MRI. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 41(1), 163–170 (1999)
  * [4] Chen, Z., Pawar, K., Ekanayake, M., Pain, C., Zhong, S., Egan, G.F.: Deep learning for image enhancement and correction in magnetic resonance imaging—state-of-the-art and challenges. Journal of Digital Imaging 36(1), 204–230 (2023)
  * [5] Eichhorn, H., Hammernik, K., Spieker, V., Epp, S.M., Rueckert, D., Preibisch, C., Schnabel, J.A.: Physics-Aware Motion Simulation For T2*-Weighted Brain MRI. In: International Workshop on Simulation and Synthesis in Medical Imaging. pp. 42–52. Springer (2023)
  * [6] Feng, R., Wu, Q., Feng, J., She, H., Liu, C., Zhang, Y., Wei, H.: IMJENSE: Scan-specific Implicit Representation for Joint Coil Sensitivity and Image Estimation in Parallel MRI. IEEE Transactions on Medical Imaging (2023). https://doi.org/10.1109/TMI.2023.3342156
  * [7] Gallichan, D., Marques, J.P., Gruetter, R.: Retrospective correction of involuntary microscopic head movement using highly accelerated fat image navigators (3D FatNavs) at 7T. Magnetic resonance in medicine 75(3), 1030–1039 (2016)
  * [8] Haskell, M.W., Cauley, S.F., Bilgic, B., Hossbach, J., Splitthoff, D.N., Pfeuffer, J., Setsompop, K., Wald, L.L.: Network accelerated motion estimation and reduction (NAMER): convolutional neural network guided retrospective motion correction using a separable motion model. Magnetic resonance in medicine 82(4), 1452–1461 (2019)
  * [9] Haskell, M.W., Cauley, S.F., Wald, L.L.: TArgeted Motion Estimation and Reduction (TAMER): data consistency based motion mitigation for MRI using a reduced model joint optimization. IEEE transactions on medical imaging 37(5), 1253–1265 (2018)
  * [10] Hossbach, J., Splitthoff, D.N., Cauley, S., Clifford, B., Polak, D., Lo, W.C., Meyer, H., Maier, A.: Deep learning-based motion quantification from k-space for fast model-based magnetic resonance imaging motion correction. Medical Physics 50(4), 2148–2161 (2023). https://doi.org/10.1002/mp.16119
  * [11] Huang, W., Li, H.B., Pan, J., Cruz, G., Rueckert, D., Hammernik, K.: Neural Implicit k-Space for Binning-Free Non-Cartesian Cardiac MR Imaging. In: International Conference on Information Processing in Medical Imaging. pp. 548–560. Springer (2023)
  * [12] Kastryulin, S., Zakirov, J., Pezzotti, N., Dylov, D.V.: Image Quality Assessment for Magnetic Resonance Imaging. IEEE Access 11, 14154–14168 (2023). https://doi.org/10.1109/access.2023.3243466
  * [13] Knoll, F., Zbontar, J., Sriram, A., Muckley, M.J., Bruno, M., Defazio, A., Parente, M., Geras, K.J., Katsnelson, J., Chandarana, H., et al.: fastmri: A publicly available raw k-space and dicom dataset of knee images for accelerated mr image reconstruction using machine learning. Radiology: Artificial Intelligence 2(1), e190007 (2020)
  * [14] Küstner, T., Armanious, K., Yang, J., Yang, B., Schick, F., Gatidis, S.: Retrospective correction of motion-affected MR images using deep learning frameworks. Magnetic resonance in medicine 82(4), 1527–1540 (2019)
  * [15] Kuzmina, E., Razumov, A., Rogov, O.Y., Adalsteinsson, E., White, J., Dylov, D.V.: Autofocusing+: Noise-resilient motion correction in magnetic resonance imaging. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 365–375. Springer (2022)
  * [16] Lee, S.B., Jung, S., Jung, K.J., Kim, D.H.: Deep Learning in MR Motion Correction: a Brief Review and a New Motion Simulation Tool (view2Dmotion). Investigative Magnetic Resonance Imaging 24, 196–206 (2020)
  * [17] Lin, C.H., Ma, W.C., Torralba, A., Lucey, S.: Barf: Bundle-adjusting neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5741–5751 (2021)
  * [18] Loktyushin, A., Nickisch, H., Pohmann, R., Schölkopf, B.: Blind retrospective motion correction of MR images. Magnetic resonance in medicine 70(6), 1608–1618 (2013)
  * [19] Molaei, A., Aminimehr, A., Tavakoli, A., Kazerouni, A., Azad, B., Azad, R., Merhof, D.: Implicit Neural Representation in Medical Imaging: A Comparative Survey. In: 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). pp. 2373–2383 (2023). https://doi.org/10.1109/ICCVW60793.2023.00252
  * [20] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 41(4), 1–15 (2022)
  * [21] Oksuz, I., Clough, J., Ruijsink, B., Puyol-Antón, E., Bustin, A., Cruz, G., Prieto, C., Rueckert, D., King, A.P., Schnabel, J.A.: Detection and correction of cardiac MRI motion artifacts during reconstruction from k-space. In: International conference on medical image computing and computer-assisted intervention. pp. 695–703. Springer (2019)
  * [22] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. pp. 234–241. Springer (2015)
  * [23] Slipsager, J.M., Glimberg, S.L., Søgaard, J., Paulsen, R.R., Johannesen, H.H., Martens, P.C., Seth, A., Marner, L., Henriksen, O.M., Olesen, O.V., et al.: Quantifying the financial savings of motion correction in brain mri: a model-based estimate of the costs arising from patient head motion and potential savings from implementation of motion correction. Journal of Magnetic Resonance Imaging 52(3), 731–738 (2020)
  * [24] Zaitsev, M., Maclaren, J., Herbst, M.: Motion artifacts in MRI: A complex problem with many partial solutions. Journal of Magnetic Resonance Imaging 42(4), 887–901 (2015). https://doi.org/10.1002/jmri.24850
  * [25] Zbontar, J., Knoll, F., Sriram, A., Murrell, T., Huang, Z., Muckley, M.J., Defazio, A., Stern, R., Johnson, P., Bruno, M., et al.: fastMRI: An open dataset and benchmarks for accelerated MRI. arXiv preprint arXiv:1811.08839 (2018)
  * [26] Zhao, R., Yaman, B., Zhang, Y., Stewart, R., Dixon, A., Knoll, F., Huang, Z., Lui, Y.W., Hansen, M.S., Lungren, M.P.: fastMRI+, Clinical pathology annotations for knee and brain fully sampled magnetic resonance imaging data. Scientific Data 9(1), 152 (2022)

##  Appendix 0.A Supplementary Material

Figure 3: $k$lD-Net detection results. The first and second show the detection
performance of the $k$LD-Net for light and heavy motion, respectively. Figure
4: The visualization shows the worst results of motion-corrected images of our
IM-MoCo pipeline besides motion-corrupted, ground truth, and comparison
methods. The first and third rows show the light and heavy correction results,
respectively. The second and fourth rows show the residual error images.
Figure 5: The visualization shows the best results of motion-corrected images
of our IM-MoCo pipeline besides motion-corrupted, ground truth, and comparison
methods. The first and third rows show the light and heavy correction results,
respectively. The second and fourth rows show the residual error images.
Figure 6: A visualization of a _Non-specific white matter lesion_ as an
example. The first and second rows show the full image and the extracted
patches, respectively.