* (37) T. Blum, P.A. Boyle, T. Izubuchi, L. Jin, A. Jüttner, C. Lehner et al., _Calculation of the hadronic vacuum polarization disconnected contribution to the muon anomalous magnetic moment_ , _Phys. Rev. Lett._ 116 (2016) 232002 [1512.09054].
  * (38) P. Boyle et al., _Lattice QCD and the Computational Frontier_ , in _2022 Snowmass Summer Study_ , 3, 2022 [2204.00039].
  * (39) L. Giusti and M. Saccardi, _Four-dimensional factorization of the fermion determinant in lattice QCD_ , _Phys. Lett. B_ 829 (2022) 137103 [2203.02247].
  * (40) UKQCD collaboration, _Quark mass dependence of hadron masses from lattice QCD_ , _Phys. Rev. D_ 59 (1999) 074503 [hep-lat/9810021].
  * (41) ETM collaboration, _Dynamical twisted mass fermions with light quarks: Simulation and analysis details_ , _Comput. Phys. Commun._ 179 (2008) 695 [0803.0224].
  * (42) ALPHA collaboration, _Monte Carlo errors with less errors_ , _Comput. Phys. Commun._ 156 (2004) 143 [hep-lat/0306017].
  * (43) M. Lüscher, _Properties and uses of the Wilson flow in lattice QCD_ , _JHEP_ 08 (2010) 071 [1006.4518].
  * (44) RBC, UKQCD collaboration, _Update of Euclidean windows of the hadronic vacuum polarization_ , _Phys. Rev. D_ 108 (2023) 054507 [2301.08696].
  * (45) H.B. Meyer, _Lorentz-covariant coordinate-space representation of the leading hadronic contribution to the anomalous magnetic moment of the muon_ , _Eur. Phys. J. C_ 77 (2017) 616 [1706.01139].
  * (46) M. Cè, M. Bruno, J. Bulava, A. Francis, P. Fritzsch, J.R. Green et al., _Hadronic observables from master-field simulations_ , _PoS_ LATTICE2022 (2023) 052 [2301.05156].
  * (47) T. Aoyama et al., _The anomalous magnetic moment of the muon in the Standard Model_ , _Phys. Rept._ 887 (2020) 1 [2006.04822].
  * (48) M. Cè, A. Gérardin, G. von Hippel, H.B. Meyer, K. Miura, K. Ottnad et al., _The hadronic running of the electromagnetic coupling and the electroweak mixing angle from lattice QCD_ , _JHEP_ 08 (2022) 220 [2203.08676].
  * (49) M. Cè, A. Gérardin, K. Ottnad and H.B. Meyer, _The leading hadronic contribution to the running of the Weinberg angle using covariant coordinate-space methods_ , _PoS_ LATTICE2018 (2018) 137 [1811.08669].
  * (50) M. Luscher and S. Schaefer, _Lattice QCD with open boundary conditions and twisted-mass reweighting_ , _Comput. Phys. Commun._ 184 (2013) 519 [1206.2809].
  * (51) M. Bruno, _pyobs_ , _Zenodo_ (2023) v1.4.1.
  * (52) G. Parisi, _The strategy for computing the hadronic mass spectrum_ , _Phys. Rept._ 103 (1984) 203.
  * (53) G.P. Lepage, _The analysis of algorithms for lattice field theory_ , in _From Actions to Answers: Proceedings of the 1989 Theoretical Advanced Study Institute in Elementary Particle Physics_ , T. DeGrand and D. Toussaint, eds., pp. 97–120, World Scientific, 1989, http://inspirehep.net/record/287173.
  * (54) K.-F. Liu, J. Liang and Y.-B. Yang, _Variance reduction and cluster decomposition_ , _Phys. Rev. D_ 97 (2018) 034507 [1705.06358].
  * (55) N. Madras and A.D. Sokal, _The Pivot algorithm: a highly efficient Monte Carlo method for selfavoiding walk_ , _J. Statist. Phys._ 50 (1988) 109.
  * (56) ALPHA collaboration, _Topological susceptibility and the sampling of field space in N f = 2 lattice QCD simulations_, _JHEP_ 08 (2014) 150 [1406.5363].


# Multi-task MR Imaging with Iterative Teacher Forcing and Re-weighted Deep
Learning

Kehan Qi  Yu Gong  Xinfeng Liu  Xin Liu  Hairong Zheng  Shanshan Wang,Senior
Member, IEEE This research was partly supported by Scientific and Technical
Innovation 2030-"New Generation Artificial Intelligence" Project
(2020AAA0104100, 2020AAA0104105) the National Natural Science Foundation of
China (61871371, 81830056), Key-Area Research and Development Program of
Guangdong Province (2018B010109009), the Basic Research Program of Shenzhen
(JCYJ20180507182400762), Youth Innovation Promotion Association Program of
Chinese Academy of Sciences (2019351). (Kehan Qi and Yu Gong are co-first
authors. Corresponding author: Shanshan Wang.)K. Qi, X. Liu, H. Zheng and S.
Wang are with Paul C. Lauterbur Research Center for Biomedical Imaging,
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences,
Shenzhen 518055, China; S. Wang is also with Pengcheng Laboratory, Shenzhen,
Guangdong (e-mail<EMAIL_ADDRESS>ss.wang@siat.ac.cn).Y. Gong is with the
College of Medicine and Biological Information Engineering, Northeastern
University, Shenyang 110169, China and also with Paul C. Lauterbur Research
Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology,
Chinese Academy of Sciences, Shenzhen 518055, China
(e-mail:yu.90n9@gmail.com). X. Liu is with Guizhou Provincial People’s
hospital, China

###### Abstract

Noises, artifacts, and loss of information caused by the magnetic resonance
(MR) reconstruction may compromise the final performance of the downstream
applications. In this paper, we develop a re-weighted multi-task deep learning
method to learn prior knowledge from the existing big dataset and then utilize
them to assist simultaneous MR reconstruction and segmentation from the under-
sampled k-space data. The multi-task deep learning framework is equipped with
two network sub-modules, which are integrated and trained by our designed
iterative teacher forcing scheme (ITFS) under the dynamic re-weighted loss
constraint (DRLC). The ITFS is designed to avoid error accumulation by
injecting the fully-sampled data into the training process. The DRLC is
proposed to dynamically balance the contributions from the reconstruction and
segmentation sub-modules so as to co-prompt the multi-task accuracy. The
proposed method has been evaluated on two open datasets and one in vivo in-
house dataset and compared to six state-of-the-art methods. Results show that
the proposed method possesses encouraging capabilities for simultaneous and
accurate MR reconstruction and segmentation.

Keywords: Deep learning, MR image reconstruction and segmentation, multi-task
network, re-weighted loss, task-driven imaging, iterative teacher forcing

##  1 Introduction

Magnetic resonance (MR) imaging is of great value in clinical applications
such as medical diagnosis [1], disease staging [2] and clinical research [3]
since it can produce highly detailed images with excellent soft tissue
contrast. The raw measurements of MR images are Fourier transform coefficients
in "k-space" and MR images can be visualized after an inverse Fourier
transform of the fully-sampled k-space data. However, the acquisition time of
fully-sampled k-space data is long. It is one of the main limitations for the
MR extensive applications. Therefore, numerous techniques have been developed
for accelerating MR imaging over the last decades, including physics-based
fast imaging sequences [4], hardware-based parallel imaging [5], signal
processing-based image reconstruction from under-sampled k-space data [6, 7]
and deep learning-based techniques [8, 9, 10, 11].

In some clinical application scenarios, images are not an end in themselves,
but rather a means of access to clinically relevant parameters which are
obtained as post-processing steps (e.g., segmentation). However, most of the
existing segmentation methods are seen as a post-processing task that is
independent of reconstruction. It may be an issue that a lot of information
discarded during the reconstruction process may influence the final
segmentation performance. The reason is that most of the existing
reconstruction methods are designed to take optimal visual quality as the
first priority, rather than the specific-task quality. Therefore, some groups
tend to develop task-driven imaging methods for segmentation [12, 13, 14, 15,
16, 17, 18]. For instance, Caballero _et al._ [19] proposed an unsupervised
brain segmentation method from k-space based on the patch-based sparse
modeling and Gaussian mixture modeling. It is a pioneering work for task-
driven MRI. But its efficiency can still be improved. Recently, many deep
learning methods based on convolutional neural networks have been introduced
for MR segmentation from k-space data [20, 21, 22, 23, 24]. Schlemper _et al._
[20] proposed using two neural networks, namely end-to-end synthesis network
(SynNet) and latent feature interpolation network (LI-Net) to directly predict
segmentation results from under-sampled k-space data. The SynNet uses U-Net
[21] as the backbone to map zero-filling images (inverse Fourier transform of
under-sampled k-space) to segmentation maps directly. The LI-Net constructs a
latent feature interpolation network with improved results. Huang _et al._
[22] proposed a Joint-FR-Net that consists of a reconstruction module derived
from the fast iterative shrinkage-thresholding algorithm (FISTA) and a
segmentation module based on U-Net [21]. Huang _et al._ [23] proposed a unique
task-driven attention module that utilizes intermediate segmentation
estimation to facilitate image-domain feature extraction from the raw data to
bridge the gap between reconstruction and segmentation. Sun _et al._ [24]
proposed a unified deep neural network called SegNetMRI with two modules (the
reconstruction module and segmentation module). The two modules are pre-
trained and fine-tuned with shared reconstruction encoders. The outputs from
all segmentation modules are merged into the final segmentation result. All
these works [20, 22, 23, 24] have made contributions to prompt the development
of task-driven MR imaging. However, existing task-driven methods tend to
attribute too much importance to the task (segmentation), which may introduce
incorrect reconstruction, such as aliasing or loss of information.
Consequently, the aliasing in reconstruction could mislead the network from
identifying the correct boundaries. Therefore, it is in need to develop a
method for a proper balance and co-promotion of both reconstruction and
segmentation accuracy.

To this end, this paper develops a multi-task MR imaging approach with
iterative teacher forcing and re-weighted deep learning. The multi-task deep
learning framework is equipped with reconstruction and segmentation sub-
network modules, which are integrated and trained by our designed iterative
teacher forcing scheme (ITFS) under the dynamic re-weighted loss constraint
(DRLC). Teacher forcing [25] scheme is a training technique widely used in the
recurrent neural network (RNN) to reduce the training difficulty of the dual
module network. We developed the ITFS to replace intermediate reconstruction
outputs with the fully-sampled data. It is designed to avoid the bias
explosion and over-correction problem caused by the original teacher forcing
scheme. To the best of our knowledge, this is the first time teacher forcing
concept has been used outside of RNN architecture. The DRLC is proposed to
dynamically balance the contributions from the reconstruction and segmentation
sub-modules so as to co-prompt the multi-task accuracy. The contributions of
this paper are summarized in three aspects as follows:

  1. 1)

A multi-task MR imaging framework is proposed to derive high-quality image
reconstructions and segmented masks simultaneously from undersampled k-space
data with iterative teacher forcing and re-weightd deep learning.

  2. 2)

A dynamic re-weighted loss constraint is designed to help the proposed network
to co-prompt the reconstruction and segmentation accuracy.

  3. 3)

An iterative teacher forcing scheme is devised to stabilize the training
process and reduce the training difficulty.

  4. 4)

The proposed method has been evaluated on two open datasets and one in vivo
in-house dataset and compared to six state-of-the-art methods. Results show
that the proposed method possesses encouraging capabilities for simultaneous
and accurate MR reconstruction and segmentation.

The reminder of this paper is organized as follows: Section 2 describes our
proposed method, Section 3 presents the experimental results and relevant
analysis, and Section 4 gives the discussion and conclusion.

Figure 1: The overall workflow of the proposed multi-task MR imaging method.
It consists of reconstruction and segmentation modules, which are D5C5 and
U-Net respectively. Based on it, dynamic re-weighted loss constraint is
designed to keep the contribution balance between reconstruction and
segmentation tasks. Besides, an improved iterative teacher forcing scheme is
devised to stabilize the training process and reduce the training difficulty
of the dual-task network.

##  2 Method

###  2.1 Overview

The overall workflow of the proposed multi-task MR imaging framework is shown
in Fig. 1. It consists of reconstruction and segmentation modules. We selected
D5C5 [26] as the reconstruction module and U-Net [21] as the segmentation
module respectively. D5C5 [26] is a convolutional neural network (CNN) with a
deep cascade structure. It integrated data consistency layers with
convolutional layers in an end-to-end training manner for reconstruction.
U-Net [21] with encoder-decoder structure based fully convolutional network
(FCN) is a popular choice for medical image segmentation. The encoder and
decoder were used to capture high-level semantic information and recover the
spatial information respectively. In addition, multiple skip-connections were
introduced between the encoder layer and the corresponding decoder layer to
recover the information loss caused by down-sampling. The reason for selecting
the published networks is that we focus on exploiting the multi-task imaging
framework rather than proposing a reconstruction or segmentation network
structure. The trained multi-task MR imaging method takes the under-sampled
k-space data as input and outputs both the reconstructed images and the
segmented masks simultaneously.

###  2.2 Dynamic Re-weighted Loss Constraint

The proposed multi-task MR imaging framework was designed to produce high-
quality reconstructed images and the segmented masks simultaneously. For the
multi-task MR imaging method, the quality of the output of the downstream
module was influenced by the upstream module greatly. The reason for it is
that the input of the segmentation module is the output of the reconstruction
module. Combining the segmentation module with the reconstruction module as a
dual-task network could bridge the gap between two relational tasks. Multi-
task imaging framework has the ability to make the reconstruction module keep
the feature that is critical for segmentation instead of only the feature that
is critical for visual performance. Commonly, this kind of network can be
trained in an end-to-end manner with a loss function that is a fixed weighted
summation of each task-specific loss function as shown in Eq. (1):

$\displaystyle L=\sum_{i=1}^{N}\alpha_{i}L_{i}$ (1)

where $L_{i}$ denotes the loss function of the $i$-th task and $\alpha_{i}$
denotes the corresponding weight. It makes the reconstruction module under the
guidance of the loss function to keep the feature that can improve the quality
of segmentation results. Such loss function has advantages including simple
structure, easy implementation and low calculation amount. However, it can’t
guarantee the contribution balance between the two sub-tasks. Besides, this
kind of loss function keeps the same weight for the training stage. In other
words, the training process of the network will keep constant whether the
particular sub-task is sufficiently optimized. It may be insufficient for
producing high-quality reconstructed images and segmented masks simultaneously
as the results shown in our experiments. Therefore, we hope that the weight of
the task can be dynamically changed with the training process. It means that
the center of the training process can be automatically transformed.
Specifically, the center of the training process should be more inclined to
optimize the reconstruction module instead of the segmentation module in the
early training stage. In the later training stage, it is advisable to spend
more efforts in optimizing the segmentation module when the loss of the
reconstruction module is basically stable. It helps the network to keep the
contribution balance of each task to ensure the quality of the final result.
In summary, we proposed the following dynamic re-weighted loss constraint for
the proposed network as shown in Eq. (2):

$\displaystyle L=\alpha(t)L_{recon}+\beta(t)L_{seg}$ (2)

where $L_{recon}$ and $L_{seg}$ denote the loss function of the reconstruction
and segmentation tasks respectively. $\alpha(t)$ and $\beta(t)$ denote the
corresponding weights, and $t$ denotes the corresponding training epoch.
$\alpha(t)$ and $\beta(t)$ are the monotonic functions of $t$. In this paper,
we selected $L_{2}$-norm as the loss function for reconstruction task and
Cross-Entropy as the loss function for the segmentation task. The loss
function of the reconstruction task is defined as Eq. (3):

$\displaystyle
L_{recon}=\|C(\boldsymbol{I}_{\mathrm{FS}})-\boldsymbol{I}_{\mathrm{US}}\|_{2}^{2}$
(3)

where $\boldsymbol{I}_{\mathrm{FS}}$ and $\boldsymbol{I}_{\mathrm{US}}$ denote
the images reconstructed from fully-sampled and under-sampled k-space data
respectively, $C$ denotes the proposed network and $\|\cdot\|_{2}$ represents
the $L_{2}$-norm. The loss function of the segmentation task is defined as Eq.
4:

$\displaystyle
L_{seg}=1.0-\frac{2\times|\boldsymbol{y}.*\boldsymbol{\hat{y}}|}{|\boldsymbol{y}|+|\boldsymbol{\hat{y}}|+0.1}$
(4)

where $\boldsymbol{y}$ and $\boldsymbol{\hat{y}}$ denote the segmentation and
annotation mask respectively, $.*$ denotes the pixel-wise multiplication of
the two matrices and $|\cdot|$ indicates the element-wise sum operation.

As shown in Fig. 2, $\alpha(t)$ increases while $\beta(t)$ decreases with the
increase of $t$. In our proposed method, we set $\alpha(t)$ as the maximum
value of the difference between the exponential transformation of $-t$ and
$0.2$, and $0.05$ as shown in Eq. (6). $\beta(t)$ was set as the difference
between $1$ and $\alpha(t)$ as shown in Eq. (8). It is designed to let the
neural networks automatically adapt to the new weights according to their
training stage. It was set to avoid over-large or over-small value of
$\alpha(t)$ to make sure each task-specific loss is effective. The use of DRLC
guarantees the proposed method can take into account both the reconstruction
task and the segmentation task to produce high-quality reconstructed images
and segmented masks simultaneously.

Figure 2: The change trend of $\alpha(t)$ and $\beta(t)$ with $t$.

$\displaystyle\begin{array}[]{c}\alpha(t)=\max\\{\exp(-t)-0.2,0.05\\}\end{array}$
(6)

$\displaystyle\begin{array}[]{c}\beta(t)=1-\alpha(t)\end{array}$ (8)

###  2.3 Iterative Teacher Forcing Scheme

In the training phase, supposing there is a serialization task with $T$ steps.
Let $x_{t}$, $y_{t}$ and $\hat{y}_{t}$ denote the input, ground truth and
output of the step $t$ respectively. For common auto-regressive mode, $x_{t}$
is given with $\hat{y}_{t-1}$. The limitation of it is that the error of low-
level models will be accumulated gradually even to make the network divergent.
For teacher forcing mode, $y_{t-1}$ is taken as the input of the step $t$
instead of $\hat{y}_{t-1}$. It is equivalent to take the ground truth as the
teacher to correct the output of low-level models in time. The benefit
provided by the use of the teacher forcing scheme is making the training
process stable and quick to reduce the training difficulty of the network.
However, the teacher forcing scheme may lead to bias exposure. It refers to
the performance gap between the training and testing processes. Namely, the
use of the teacher forcing scheme makes the inputs of the step $t$ of the
network in the training process and the testing process are inferred from
different distributions. It may lead to discrepancy between the decoding
actions of the network in the training and testing phases. Meanwhile, the
constraint provided by the teacher forcing scheme may over-correct the output
to reduce the representation ability of the network.

Figure 3: The difference between the original teacher forcing scheme and the
proposed iterative teacher forcing scheme is that we take the ground truth
denoted by the blue line and the actual output denoted by the orange line of
step $t-1$ as the input of step $t$ alternately instead of the fixed ground
truth of step $t-1$. It is beneficial for improving the training process while
avoiding exposure bias and over-correction issue.

To overcome the aforementioned limitations of the original teacher forcing
scheme, we propose an iterative teacher forcing scheme. As shown in Fig. 3, we
use the blue line to denote the actual output of step $t-1$ and the orange
line to denote the ground truth of step $t-1$ respectively. The difference
between the original teacher forcing scheme and the proposed iterative teacher
forcing scheme is that taking the ground truth and the actual output of step
$t-1$ as the input of step $t$ alternately rather than the fixed ground truth
of step $t-1$. The proposed iterative teacher forcing scheme not only can
correct the prediction of the low-level model to a certain extent but also can
avoid the performance gap between the training and testing processes.

Specifically, we treated multi-task MR imaging as a two-step serialization
task that begins from the reconstruction step to the segmentation step for
applying the iterative teacher forcing scheme. As far as we know, it is the
first time that the teacher forcing scheme has been used in neural networks
outside RNN. In the training phase of the proposed network, we took the under-
sampled k-space data as the input of the first step $x_{1}$. The output
$\hat{y}_{1}$ and the ground truth $y_{1}$ of the first step were selected as
the input of the second step $x_{2}$ iteratively. It prevents the error
accumulation of the reconstruction step from propagating to the segmentation
step. In addition, the use of teacher forcing scheme is beneficial for
stabilizing the training process, reducing the training difficulty and
protecting the presentation ability of the proposed network.

##  3 Experiments and Results

###  3.1 Experimental Setting

####  3.1.1 Data

We instantiated our method on two competitive MR image datasets including
ATLAS [27] and realistic simulated k-space data generated from Brainweb [28].
The ATLAS dataset consists of 229 intensity normalized subjects on T1 modality
in standard space (normalized to the MNI-152 template) collected from 11
cohorts worldwide, with an in-plane resolution of $1mm^{3}$, in which 120
subjects were used for training, 40 for validation and 69 for testing. 20
cases were employed from the publicly available simulated brain dataset,
Brainweb, where 12 brain tissue types were included with known distributions,
in which 17 cases were used for training and 3 for testing.

We also conducted our approach on an in house in-vivo dataset. It consists of
40 cases and 485 slices from FLAIR sequence obtained with a Siemens 1.5T
scanner. The FOV phase amplitude is $70\%-87.5\%$. The slice thickness ranges
from 6.0 mm to 7.0 mm and the slice interval ranges from 7.8 mm to 9.1 mm.
Lesions were annotated by experienced clinicians using ITK snap. 20 cases (247
slices) are used for training with the rest 20 cases (238 slices) for testing.

For k-space data sub-sampling, we performed masks as shown in Fig. 4 on the
fully-sampled acquisition retrospectively. For a volume, all slices were
applied to the same under-sampling mask. The overall acceleration factor was
set as four. To achieve the desired acceleration factor, $8\%$ of all k-space
lines from the central region were kept and the remaining k-space lines were
kept with a set probability uniformly at random. It was set to meet the
general conditions for compressed sensing [29, 30].

Figure 4: Examples of under-sampled k-space trajectories. Table 1: Ablation analysis of iterative teacher forcing scheme and dynamic re-weighted loss constraint on ATLAS dataset. For each metric, we mark the best in bold. ITFS | DRLC | Dice | Precision | Recall | PSNR | SSIM
---|---|---|---|---|---|---
|  | $0.494\pm 0.071$ | $0.636\pm 0.135$ | $0.462\pm 0.067$ | $28.78\pm 0.732$ | $0.957\pm 0.005$
✓ |  | $0.499\pm 0.096$ | $0.682\pm 0.109$ | $0.467\pm 0.024$ | $27.20\pm 0.533$ | $0.943\pm 0.004$
| ✓ | $0.510\pm 0.068$ | $0.686\pm 0.126$ | $0.472\pm 0.060$ | $28.84\pm 0.619$ | $0.957\pm 0.005$
✓ | ✓ | $\textbf{0.518}\pm 0.064$ | $\textbf{0.688}\pm 0.076$ | $\textbf{0.479}\pm 0.020$ | $\textbf{28.87}\pm 0.409$ | $\textbf{0.958}\pm 0.00005$
Table 2: Ablation analysis of different dynamic re-weighted loss constraint. For each metric, we mark the best in bold. Re-weighted Loss Strategy | Dice | Precision | Recall | PSNR | SSIM
---|---|---|---|---|---
| Fixed Weight
---
($\alpha=0.5$ and $\beta=0.5$)
$0.494\pm 0.083$ | $0.673\pm 0.104$ | $0.459\pm 0.045$ | $27.23\pm 0.620$ | $0.933\pm 0.009$
| Fixed Weight
---
($\alpha=0.2$ and $\beta=0.8$)
$0.485\pm 0.077$ | $0.662\pm 0.080$ | $0.444\pm 0.042$ | $28.70\pm 0.519$ | $0.955\pm 0.007$
| Dynamic Weight
---
(Linear)
$0.494\pm 0.073$ | $0.664\pm 0.087$ | $0.452\pm 0.031$ | $28.87\pm 0.516$ | $0.956\pm 0.0006$
| Dynamic Weight
---
(Exponent)
$\textbf{0.518}\pm 0.064$ | $\textbf{0.688}\pm 0.076$ | $\textbf{0.479}\pm 0.020$ | $\textbf{28.87}\pm 0.409$ | $\textbf{0.958}\pm 0.00005$

####  3.1.2 Implementation Details

This subsection presented some of the details regarding the sub-sampling
procedure. The model was implemented using Pytorch deep learning framework.
D5C5 [26] was selected as the reconstruction module and U-Net [21] as the
segmentation module. The model was trained for 50 epochs using Adam [31]
optimization algorithm on a single NVIDIA Titan Xp GPU.

For ATLAS dataset, 189 slices were collected for every subject. Each slice was
cropped to $224\times 192$ for the convenience of deep learning model
inference. The batch size was set to 16, and the initial learning rate for
both sub-modules was set to $1e-6$ with a decreasing rate of 0.2 for every 10
epochs. 229 subjects were randomly split into 3 sub-datasets for training,
validation and testing, respectively. Each of these three sub-datasets
contained 120, 40 and 69 subjects, respectively. Segmented brain lesions were
employed to calculate the segmentation metrics. The model was trained for 50
epochs using Adam optimizer.

For simulated Brainweb data, 20 cases were employed with known tissue
annotation, 17 for training and 3 for testing. 12 tissue types were given, in
which 3 of them (CSF, WM, and GM) were used to calculate the segmentation
quantitative metrics. All the 3D images were resized and cropped to the size
of $362\times 416\times 352$. The batch size was set to 4, and the initial
learning rate was set to $1e-4$. Random flip, rotation and affine
transformation are used as data augmentation methods. The model was trained
for 3000 epochs using Adam optimizer.

For in vivo in-house dataset, the MR data were collected by Guizhou Provincial
People’s Hospital. Data from 40 cases were obtained with FLAIR sequence and
the stroke lesions were annotated by experienced clinicians using ITK snap,
with 20 cases as the training set and the rest 20 cases as the test set. All
images were obtained with a Siemens 1.5T scanner. The FOV phase amplitude is
$70\%-87.5\%$. The slice thickness ranges from 6.0 mm to 7.0 mm and the slice
interval ranges from 7.8 mm to 9.1 mm. All the data were resized to $224\times
224$ with intensity normalized. The batch size was set to 16, and the initial
learning rate was set to $1e-4$. The model was trained for 50 epochs using
Adam optimizer.

###  3.2 Quantitative Evaluation Metrics

For the ATLAS dataset and in vivo in-house dataset, five metrics are used to
measure the performance of the segmentation and reconstruction results,
including the Dice score, precision, recall, peak signal-to-noise ratio (PSNR)
and structural similarity index matrix (SSIM). The former three metrics were
employed to evaluate the segmentation performance and the latter two were used
to evaluate the reconstruction performance. For the Brainweb dataset, five
metrics are used to measure the performance of the segmentation and
reconstruction results, including the Dice score on CSF, GM, WM, PSNR and
SSIM. All the metrics referred to the 3D predicting results. Average metric
scores on the testing dataset were listed for the comparison between different
methods.

###  3.3 Ablation Study

####  3.3.1 Effectiveness of ITFS and DRLC

In this section, we focus on evaluating the effectiveness of the proposed
modules of ITFS and DRLC. First, we trained a joint framework of
reconstruction and segmentation modules from scratch without the iterative
teacher forcing scheme and dynamic re-weighted loss constraint. Second, we
alternatively introduced the iterative teacher forcing scheme and dynamic re-
weighted loss constraint to the joint framework. The related quantitative
comparison results are shown in Table 1. We could see both the iterative
teacher forcing scheme and dynamic re-weighted loss constraint improved the
performance of the proposed network. On the basis of it, we leveraged the
iterative teacher forcing scheme and dynamic re-weighted loss constraint
simultaneously. It is clearly observed that the network equipped with both the
iterative teacher-forcing scheme and the dynamic re-weighted loss constraint
achieved better performance than other networks. They were beneficial for
producing high-quality reconstruction and segmentation results simultaneously.
Furthermore, the results shown in Table 1 demonstrated that the dynamic re-
weighted loss constraint contributed more to the performance gain than the
iterative teacher-forcing does. However, ITFS has a more important capability
of stabilizing the training process. As shown in Fig. 5, the introduction of
ITFS improved the convergence of the proposed network. It stabilized the
training process of the dual-task network to reduce the training difficulty.

Figure 5: The loss values with or without ITFS. The colored area covers $95\%$
of the loss values. It is clear that using ITFS could make the training
process more stable.

####  3.3.2 Ablation Analysis of Different Kinds of DRLC

As the results showed above, the dynamic re-weighted loss constraint helped
the network produce promising results. In this section, we compared the
performance of different re-weighted loss constraints to validate the
effectiveness of our proposed exponential-like dynamic re-weighed loss
constraint. As shown in Table 2, we have evaluated three kinds of weight
setting schemes, namely the fixed weight, dynamic weight with linear settings
and our proposed scheme. Compared to the fixed weights of $\alpha$ and
$\beta$, dynamic re-weighted loss constraint achieved better performance for
both reconstruction and segmentation. Dynamic self-adopted loss adjustment
helps the loss value adapt to the training stage, which avoids too large loss
and gradient value at the final stage of training. Furthermore, the
performance of the re-weighted loss based on exponential operations was better
than that of the re-weighted loss based on linear functions. This is
intuitively explainable since the first few epochs may change drastically
while it becomes steady with respect to the iterations. Therefore, we selected
the re-weighted loss with exponential formations in the following experiments.

###  3.4 Evaluation and comparison on ATLAS dataset

Table 3: Comparison experiments results of different methods on ATLAS dataset. For each metric, we mark the best in bold. Method | Dice | Precision | Recall | PSNR | SSIM
---|---|---|---|---|---
SynNet | $0.357\pm 0.072$ | $0.459\pm 0.085$ | $0.352\pm 0.068$ | - | -
LI-Net | $0.312\pm 0.067$ | $0.478\pm 0.095$ | $0.278\pm 0.055$ | - | -
SERANet | $0.213\pm 0.064$ | $0.656\pm 0.179$ | $0.149\pm 0.034$ | - | -
SegNetMRI | $0.224\pm 0.067$ | $0.618\pm 0.164$ | $0.156\pm 0.034$ | $28.22\pm 0.746$ | $0.950\pm 0.006$
| U-Net
---
(full-sample)
$0.512\pm 0.085$ | $0.632\pm 0.102$ | $\textbf{0.511}\pm 0.094$ | - | -
D5C5 | - | - | - | $28.78\pm 0.689$ | $0.958\pm 0.0005$
ours | $\textbf{0.518}\pm 0.064$ | $\textbf{0.688}\pm 0.076$ | $0.479\pm 0.020$ | $\textbf{28.87}\pm 0.409$ | $\textbf{0.959}\pm 0.00005$
Figure 6: Violin-plot of comparison of segmentation results on ATLAS dataset.
Figure 7: Boxplot of comparison of segmentation results on ATLAS dataset.

To validate the effectiveness of our method, we conducted experiments on ATLAS
dataset, which is publicly available and offers a high variety of lesions.
Noting that, our proposed method and SegNetMRI could provide reconstructed
images and segmented masks simultaneously. However, SynNet, LI-Net, and
SERANet just could provide segmented masks. Table 3 and Fig. 6 show the
comparison of reconstruction and lesion segmentation performance on ATLAS
dataset. We compared our method with existing segmentation and reconstruction
from under-sampled k-space data approaches. We can see that our approach
achieved better performance on lesion segmentation and promising performance
on image reconstruction, with more that 0.15 gain on Dice score. It is worth
noticing from Fig. 6 that our approach achieved smaller variety on Dice score,
which indicates a better segmentation performance on small lesions.
Segmentation results are encouraging even compared with segmentation from
fully-sampled data.

To have a comprehensive comparison with existing segmentation from under-
sampled data methods, we reported the box-plot on Dice score for different
lesion sizes in Fig. 7. This figure shows that all of the methods perform well
on median size lesions (lesion size 2 and 3). However, our approach achieves
encouraging results on small (lesion size 0 and 1) and huge (lesion size 5)
lesions, while other methods perform worse. The promising segmentation result
suggests the effectiveness and good generalization capability of the proposed
method.

The quantitative comparison of the reconstruction performance of different
networks is shown in Table 3. As we can see, our proposed method obtained
promising quantitative metrics compared with other methods. Besides, we
selected two slices to show the reconstruction performance of different
networks, which are shown in Fig. 8. As we can see in Fig. 8, all the methods
were able to generate high-quality images. The suppression of artifacts
improved the visual quality of the reconstruction images greatly.
Nevertheless, there were differences between the reconstruction results of
different networks. Reference to the error maps and quantitative metrics, the
reconstruction ability of the proposed method was better than that of the
other method.

Figure 8: Examples of our lesion segmentation results and image reconstruction
error maps on ATLAS dataset. From left to right: label, ours, SegNetMRI, U-Net
(full-sample) (top) or D5C5 (bottom), SynNet, LI-Net, SERANet. Figure 9:
Examples of our lesion segmentation results and image reconstruction error
maps on simulated Brainweb data. From left to right: label, ours, SegNetMRI,
U-Net (full-sample) (top) or D5C5 (bottom). Figure 10: Examples of our lesion
segmentation results and image reconstruction error maps on in vivo data. From
left to right: label, ours, SegNetMRI, U-Net (full-sample) (top) or D5C5
(bottom).

###  3.5 Evaluation and comparison on Simulated Brainweb Data

To validate the effectiveness and robustness of the proposed method, we
conduct experiments on simulated Brainweb data, which is publicly available
and offers digital brain phantoms with 12 tissue types. Three tissues were
used to evaluate the segmentation performance because of their large size.
Table 4 and Fig. 9 shows the comparison of the image reconstruction and tissue
segmentation results. We can see that our method achieved better performance
on CSF segmentation and PSNR, with 0.04 and 1.2 dB improvement on CSF Dice
score and PSNR, respectively. Our method also achieved encouraging performance
on WM and GM segmentation and SSIM, with 0.9285, 0.9354 and 0.993,
respectively.

###  3.6 Evaluation and comparison on in vivo in-house Dataset

To validate the effevtiveness of the proposed on in vivo data, we conduct
experiments on in vivo in-house dataset. Table 5 and Fig 10 shows the
comparison of the image reconstruction and lesion segmentation results. We can
conclude that our approach achieved encouraging performance for both
reconstruction and segmentation, with 0.864 in Dice, 32.65 dB in PSNR and
0.983 in SSIM respectively. As we can see in Figure 10, all the methods
achieves promising reconstruction error maps and segmentation masks. Compared
with the error maps and segmentation masks achieved by other methods, our
approach was more effective in these two tasks.

Table 4: Comparison experiments results of different methods on simulated Brainweb dataset. For each metric, we mark the best in bold. Method | CSF | WM | GM | PSNR | SSIM
---|---|---|---|---|---
SegNetMRI | $0.792\pm 0.044$ | $0.915\pm 0.008$ | $0.925\pm 0.021$ | $41.48\pm 0.077$ | $0.991\pm 0.001$
D5C5 | - | - | - | $39.68\pm 0.057$ | $0.993\pm 0.001$
| U-Net
---
(full-sample)
$0.862\pm 0.013$ | $0.928\pm 0.002$ | $0.933\pm 0.001$ | - | -
ours | $\textbf{0.869}\pm 0.003$ | $\textbf{0.958}\pm 0.001$ | $\textbf{0.960}\pm 0.001$ | $\textbf{42.68}\pm 0.044$ | $\textbf{0.997}\pm 0.001$
Table 5: Comparison experiments results of different methods on in-house dataset. For each metric, we mark the best in bold. Method | Dice | precision | recall | PSNR | SSIM
---|---|---|---|---|---
ours | $\textbf{0.864}\pm 0.001$ | $\textbf{0.880}\pm 0.011$ | $\textbf{0.868}\pm 0.031$ | $\textbf{32.65}\pm 0.013$ | $\textbf{0.986}\pm 0.001$
SegNetMRI | $0.841\pm 0.004$ | $0.814\pm 0.024$ | $0.859\pm 0.033$ | $32.52\pm 0.057$ | $0.983\pm 0.006$
| U-Net
---
(Full-sample)
$0.854\pm 0.015$ | $0.830\pm 0.007$ | $0.7887\pm 0.107$ | - | -
D5C5 | - | - | - | $32.49\pm 0.044$ | $0.984\pm 0.008$

##  4 Discussion and Conclusion

In this study, we propose a multi-task MR imaging framework with iterative
teacher forcing scheme and dynamic re-weighted loss constraint. The promising
reconstruction and segmentation results were achieved through our proposed re-
weighted deep learning framework. As the comparison experiments showed, the
proposed iterative teacher forcing scheme and dynamic re-weighted loss
constraint co-prompt the quality of both the reconstruction task and
segmentation task. The main drawback of directly combing the loss function of
the corresponding task is that may lead to an unbalance between the
performance of two modules. The proposed dynamic re-weighted loss constraint
has the ability to control the training process of the network to take two
considerations on both reconstruction and segmentation tasks. It helps the
network avoid the over-large or over-small value of gradient in the training
process to make sure each task-specific loss effective. In other words, the
core of the training process of the proposed network was managed dynamically.
Furthermore, the error of the reconstruction task may influence the quality of
the segmentation task. Although we can leverage the teacher forcing scheme to
solve the problem mentioned above, the incident exposure bias and over-
correction may breakdown the performance of the network. The proposed
iterative teacher forcing scheme not only internally correct the prediction of
the low-level model but also help avoiding the occurrence of exposure bias and
over-corrections for the reconstruction model. It makes sure the quality of
the segmentation module not influenced by the reconstruction error as much as
possible. Therefore, we argue that the segmentation performance gain mainly
came from the iterative teacher forcing scheme. Moreover, compared with
existing under-sampled k-space MR image segmentation methods, the proposed
method segment small lesions more efficiently, which is crucial in clinical
practice. We demonstrate that this performance gains mainly due to the
effectiveness of the task-driven reconstruction model, which focuses on lesion
region reconstruction. In the future, we consider to explore our method for
dynamic MR imaging. Another key that should be explored is the potential
training cost of the iterative teacher forcing scheme and dynamic re-weighted
loss constraint. Alternatives will be explored in the future for slighter
training costs.

##  5 Acknowledgments

This research was partly supported by Scientific and Technical Innovation 2030
- "New Generation Artificial Intelligence" Project (2020AAA0104100,
2020AAA0104105), the National Natural Science Foundation of China (61871371,
81830056), Key-Area Research and Development Program of GuangDong Province
(2018B010109009), the Basic Research Program of Shenzhen
(JCYJ20180507182400762), Youth Innovation Promotion Association Program of
Chinese Academy of Sciences (2019351).

## References

  * [1] F. Bruno, F. Arrigoni, P. Palumbo, R. Natella, N. Maggialetti, A. Reginelli, A. Splendiani, E. Di Cesare, L. Brunese, G. Guglielmi _et al._ , “New advances in MRI diagnosis of degenerative osteoarthropathy of the peripheral joints,” _La radiologia medica_ , vol. 124, no. 11, pp. 1121–1127, 2019.
  * [2] S. A. Taylor, S. Mallett, S. Ball, S. Beare, G. Bhatnagar, A. Bhowmik, P. Boavida, J. Bridgewater, C. S. Clarke, M. Duggan _et al._ , “Diagnostic accuracy of whole-body MRI versus standard imaging pathways for metastatic disease in newly diagnosed non-small-cell lung cancer: the prospective streamline L trial,” _The Lancet Respiratory Medicine_ , vol. 7, no. 6, pp. 523–532, 2019.
  * [3] S. Debette, S. Schilling, M.-G. Duperron, S. C. Larsson, and H. S. Markus, “Clinical significance of magnetic resonance imaging markers of vascular brain injury: a systematic review and meta-analysis,” _JAMA neurology_ , vol. 76, no. 1, pp. 81–94, 2019.
  * [4] A. Oppelt, R. Graumann, H. Barfuss, H. Fischer, W. Hartl, W. Schajor _et al._ , “FISP—a new fast MRI sequence,” _Electromedica_ , vol. 54, no. 1, pp. 15–18, 1986.
  * [5] M. Lustig and J. M. Pauly, “SPIRiT: iterative self-consistent parallel imaging reconstruction from arbitrary k-space,” _Magnetic resonance in medicine_ , vol. 64, no. 2, pp. 457–471, 2010.
  * [6] M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly, “Compressed sensing MRI,” _IEEE signal processing magazine_ , vol. 25, no. 2, pp. 72–82, 2008\.
  * [7] X. Qu, Y. Hou, F. Lam, D. Guo, J. Zhong, and Z. Chen, “Magnetic resonance image reconstruction from undersampled measurements using a patch-based nonlocal operator,” _Medical Image Analysis_ , vol. 18, no. 6, pp. 843–856, 2014.
  * [8] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, and D. Liang, “Accelerating magnetic resonance imaging via deep learning,” in _2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2016, pp. 514–517.
  * [9] Y. Han, J. Yoo, H. H. Kim, H. J. Shin, K. Sung, and J. C. Ye, “Deep learning with domain adaptation for accelerated projection-reconstruction mr,” _Magnetic Resonance in Medicine_ , vol. 80, no. 3, 2018.
  * [10] H. K. Aggarwal, M. P. Mani, and M. Jacob, “Modl: Model-based deep learning architecture for inverse problems,” _IEEE Transactions on Medical Imaging_ , 2018.
  * [11] M. Kabkab, P. Samangouei, and R. Chellappa, “Task-aware compressed sensing with generative adversarial networks,” in _AAAI Conference on Artificial Intelligence_ , 2018.
  * [12] J. Adler, S. Lunz, O. Verdier, C.-B. Schönlieb, and O. Öktem, “Task adapted reconstruction for inverse problems,” _arXiv preprint arXiv:1809.00948_ , 2018.
  * [13] V. Corona, M. Benning, M. J. Ehrhardt, L. F. Gladden, R. Mair, A. Reci, A. J. Sederman, S. Reichelt, and C.-B. Schönlieb, “Enhancing joint reconstruction and segmentation with non-convex Bregman iteration,” _Inverse Problems_ , vol. 35, no. 5, p. 055001, 2019.
  * [14] R. Ramlau and W. Ring, “A Mumford–Shah level-set approach for the inversion and segmentation of X-ray tomography data,” _Journal of Computational Physics_ , vol. 221, no. 2, pp. 539–557, 2007.
  * [15] M. Burger, C. Rossmanith, and X. Zhang, “Simultaneous reconstruction and segmentation for dynamic SPECT imaging,” _Inverse Problems_ , vol. 32, no. 10, p. 104002, 2016.
  * [16] F. Lauze, Y. Quéau, and E. Plenge, “Simultaneous reconstruction and segmentation of CT scans with shadowed data,” in _International Conference on Scale Space and Variational Methods in Computer Vision_. Springer, 2017, pp. 308–319.
  * [17] E. Klann, R. Ramlau, and W. Ring, “A Mumford-Shah level-set approach for the inversion and segmentation of SPECT/CT data,” _Inverse Problems & Imaging_, vol. 5, no. 1, p. 137, 2011.
  * [18] E. Klann, “A Mumford–Shah-like method for limited data tomography with an application to electron tomography,” _SIAM Journal on Imaging Sciences_ , vol. 4, no. 4, pp. 1029–1048, 2011.
  * [19] J. Caballero, W. Bai, A. N. Price, D. Rueckert, and J. V. Hajnal, “Application-driven MRI: joint reconstruction and segmentation from undersampled MRI data,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2014, pp. 106–113.
  * [20] J. Schlemper, O. Oktay, W. Bai, D. C. Castro, J. Duan, C. Qin, J. V. Hajnal, and D. Rueckert, “Cardiac MR segmentation from undersampled k-space using deep latent representation learning,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 259–267.
  * [21] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [22] Q. Huang, D. Yang, J. Yi, L. Axel, and D. Metaxas, “FR-Net: Joint reconstruction and segmentation in compressed sensing cardiac MRI,” in _International Conference on Functional Imaging and Modeling of the Heart_. Springer, 2019, pp. 352–360.
  * [23] Q. Huang, X. Chen, D. Metaxas, and M. S. Nadar, “Brain Segmentation from k-space with End-to-end Recurrent Attention Network,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 275–283.
  * [24] L. Sun, Z. Fan, X. Ding, Y. Huang, and J. Paisley, “Joint cs-mri reconstruction and segmentation with a unified deep network,” in _International Conference on Information Processing in Medical Imaging_. Springer, 2019, pp. 492–504.
  * [25] R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” _Neural computation_ , vol. 1, no. 2, pp. 270–280, 1989.
  * [26] J. Schlemper, J. Caballero, J. V. Hajnal, A. N. Price, and D. Rueckert, “A deep cascade of convolutional neural networks for dynamic MR image reconstruction,” _IEEE transactions on Medical Imaging_ , vol. 37, no. 2, pp. 491–503, 2017.
  * [27] S. L. Liew, J. M. Anglin, N. W. Banks, M. Sondag, K. L. Ito, H. Kim, J. Chan, J. Ito, C. Jung, and N. a. Khoshab, “A large, open source dataset of stroke anatomical brain images and manual lesion segmentations,” _entific Data_ , vol. 5, p. 180011, 2018.
  * [28] C. A. Cocosco, V. Kollokian, R. K.-S. Kwan, G. B. Pike, and A. C. Evans, “Brainweb: Online interface to a 3D MRI simulated brain database,” in _NeuroImage_. Citeseer, 1997.
  * [29] E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” _IEEE Transactions on information theory_ , vol. 52, no. 2, pp. 489–509, 2006\.
  * [30] M. Lustig, D. Donoho, and J. M. Pauly, “Sparse MRI: The application of compressed sensing for rapid MR imaging,” _Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine_ , vol. 58, no. 6, pp. 1182–1195, 2007.
  * [31] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.