# FetMRQC: an open-source machine learning framework for multi-centric fetal
brain MRI quality control

Thomas Sanchez  CIBM – Center for Biomedical Imaging, Switzerland  Department
of Medical Radiology, Service of Diagnostic and Interventional Radiology,
Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
<EMAIL_ADDRESS>Oscar Esteban  Department of Medical Radiology,
Service of Diagnostic and Interventional Radiology, Lausanne University
Hospital and University of Lausanne, Lausanne, Switzerland  Yvan Gomez
BCNatal Fetal Medicine Research Center (Hospital Clínic and Hospital Sant Joan
de Déu), Universitat de Barcelona, Spain  Department Woman-Mother-Child, CHUV,
Lausanne, Switzerland  Alexandre Pron  Aix-Marseille Université, CNRS,
Institut de Neurosciences de La Timone, Marseilles, France  Mériam Koob
Department of Medical Radiology, Service of Diagnostic and Interventional
Radiology, Lausanne University Hospital and University of Lausanne, Lausanne,
Switzerland  Vincent Dunet  Department of Medical Radiology, Service of
Diagnostic and Interventional Radiology, Lausanne University Hospital and
University of Lausanne, Lausanne, Switzerland  Nadine Girard  Aix-Marseille
Université, CRMBM, UMR CNRS 7339, Marseilles, France  Service de
Neuroradiologie Diagnostique et Interventionnelle, Hôpital Timone, AP-HM,
Marseilles, France  Andras Jakab  Center for MR Research, University
Children’s Hospital Zurich, University of Zurich, Zurich, Switzerland
Neuroscience Center Zurich, University of Zurich, Zurich, Switzerland
Research Priority Project Adaptive Brain Circuits in Development and Learning
(AdaBD), University of Zürich, Zurich, Switzerland  Elisenda Eixarch  BCNatal
Fetal Medicine Research Center (Hospital Clínic and Hospital Sant Joan de
Déu), Universitat de Barcelona, Spain  IDIBAPS and CIBERER, Barcelona, Spain
Guillaume Auzias  Aix-Marseille Université, CNRS, Institut de Neurosciences de
La Timone, Marseilles, France  Meritxell Bach Cuadra  CIBM – Center for
Biomedical Imaging, Switzerland  Department of Medical Radiology, Service of
Diagnostic and Interventional Radiology, Lausanne University Hospital and
University of Lausanne, Lausanne, Switzerland

###### Abstract

Fetal brain MRI is becoming an increasingly relevant complement to
neurosonography for perinatal diagnosis, allowing fundamental insights into
fetal brain development throughout gestation. However, uncontrolled fetal
motion and heterogeneity in acquisition protocols lead to data of variable
quality, potentially biasing the outcome of subsequent studies. We present
FetMRQC, an open-source machine-learning framework for automated image quality
assessment and quality control that is robust to domain shifts induced by the
heterogeneity of clinical data. FetMRQC extracts an ensemble of quality
metrics from unprocessed anatomical MRI and combines them to predict experts’
ratings using random forests. We validate our framework on a pioneeringly
large and diverse dataset of more than 1600 manually rated fetal brain
T2-weighted images from four clinical centers and 13 different scanners. Our
study shows that FetMRQC’s predictions generalize well to unseen data while
being interpretable. FetMRQC is a step towards more robust fetal brain
neuroimaging, which has the potential to shed new insights on the developing
human brain.

Teaser. Multicentric data are an essential component towards achieving robust
quality control in fetal brain MRI.

###### keywords:

Image quality assessment and Fetal brain MRI and domain shift

##  1 Introduction

Establishing a protocol for objective image quality assessment and control for
neuroimaging studies is critical to enforce reliability, generalization and
replicability [1, 2, 3]. Quality assessment (QA) focuses on assessing and
eventually improving the quality of a process to prevent issues from
propagating, while quality control (QC) looks to find and discard problematic
outputs of that process[4]. Both steps are fundamental in magnetic resonance
imaging (MRI) studies, as insufficient MRI data quality has been shown to bias
statistical analyses and neuroradiological interpretation [5, 6, 7].

Automated QA/QC tools designed to assist data exclusion decisions for adult
brains neuroimaging studies [8, 9, 10] are becoming increasingly available.
However, these techniques are inapplicable to fetal MRI, as they rely on
priors that are not valid in utero, such as e.g., assuming that the head is
surrounded by air or the relative orientation of the brain with respect to the
stereotaxic frame defined by the scanner. In addition, fetal brain MRI
typically displays larger and uncontrolled motion of the head as fixation
techniques (e.g., padding) and real-time feedback countermeasures are only
available after birth (Fig. 1A). Moreover, fetal brain imaging greatly lacks
standardization in acquisition protocols (Fig. 1B). While consensus has
settled on 2-dimensional (2D) fast-spin echo interleaved
T${}_{\text{2}}$-weighted (T2w) MR schemes showcasing thick slices[11, 12],
specific imaging parameters such as in-plane resolution, slice thickness,
field of view, or vendor implementation of the imaging sequence greatly vary.
As a result, the appearance and quality of fetal MR images in this wild-type
data vary markedly across centers (Fig. 1b).

Figure 1: Variations in data quality illustrated. A – Comparison of data
across adult (T1w), from the ABIDE dataset[13] and from fetal acquisitions. In
the excluded scans, the adult image on the left suffers from severe motion
artifacts, while large coil artifacts corrupt the image on the right. The
fetal data suffer from strong intensity changes between multiple slices and
signal drop; in the through-plane view, strong inter-slice motion makes it
difficult to discern the brain structures. B – Examples of data acquired on
different scanners, with very different appearance. The in-plane and through-
plane resolution, the field of view, the repetition time (TR), and the echo
time (TE) can all substantially change between acquisition protocols. C –
Importance of quality control for super-resolution reconstruction (SRR),
illustrated using NiftyMIC [14], and NeSVoR [15], two SRR methods with built-
in outlier rejection. On the top row a subject is reconstructed using all
stacks available (13 for NiftyMIC, 5 for NeSVoR), and each reconstruction
shows large artifacts. On the bottom row, FetMRQC is plugged in and by
removing low quality series (6 out of 13 for NiftyMIC, 2 out of 5 for NeSVoR),
the reconstruction quality is improved.

Although fetal brain MRI can be severely affected by artifacts like inter-
slice motion, signal drops or bias field[12], only few methods dedicated to
QA/QC have been proposed. Initially, automated QA/QC has been integrated
within the super-resolution reconstruction (SRR) process[16, 17, 14, 18, 15]
[17, 14, 19, 20, 21, 14, 16]. SRR is a ubiquitous early step of the fetal MRI
processing workflow that builds a high-resolution, isotropic, 3D volume from
several differently-oriented stacks of 2D slices with low-resolution (LR)
along the through-plane axis (i.e., anisotropic resolution)[22]. Some of the
proposed approaches incorporate an automated QC stage for outlier rejection
that excludes sub-standard slices or pixels from the input low-resolution
stacks, and measure the similarity between a reconstructed slice and an input
slice using information-theoretic metrics[14, 17, 23, 15]. However, as
illustrated on Fig. 1c, sub-optimal quality stacks can remain detrimental to
the final quality of the reconstruction, even when SRR pipelines include
outlier rejection schemes. Additional QA/QC checkpoints are thus needed to
filter out low-quality raw T2 stacks before using SRR, and several deep
learning-based methods were recently proposed for this task [19, 21, 20].
These solutions aim to automatically identify problematic slices for exclusion
(QC), and, if streamlined with the acquisition, enable re-acquiring corrupted
slices on the fly[24] (QA). However, these methods operate at the slice level,
and not all artifacts can be seen by analyzing slices independently. For
instance, inter-slice motion (visible on the right of Figure 1a), a strong
bias field in the through-plane direction, or an incomplete field of view can
be spotted only when considering the entire stack of slices. Stack-wise QA/QC
methods are thus still needed.

Importantly, these methods face the challenge of deployment to unseen scanners
or acquisition settings: how will they generalize to unseen domains? Due to
the private and sensitive nature of medical data[25], building large and
diverse medical imaging datasets is difficult endeavor. As a consequence,
proposed methods are often only evaluated on locally available data, and can
fail to deal with the heterogeneity found across different centers[26, 27]. In
addition, while openly shared MRI databases have be released for adults[28,
13, 29, 30], children and adolescents[31, 32], privacy protection regulations
and ethical limitations to data-sharing are much stronger regarding fetuses,
making it even more difficult to construct robust ML models trained on
multicentric data. As today, the question of the robustness of state-of-the-
art approaches to fetal brain quality control[21, 14, 16] to unseen domains
remains open.

Beyond the need of supporting SRR, quality assessment also builds towards
reproducible neuroimaging pipelines, allowing to fairly compare different
processing steps[33]. For instance, initiative of fetal brain tissue
segmentation but lack of systematic/standardized objective evaluation of
quality input data that would support the analysis of the comparison
results[34].

The contribution of our paper is threefold. First, we introduce a framework
specifically designed for QA/QC manual annotations of T2w fetal brain MRI. It
generates a visual report for efficient stack screening and manual QA,
facilitating the work of raters. Second, we present FetMRQC, a machine
learning model based on manual ratings to automatically perform two tasks: 1)
quality assessment, where a discrete score between 0 (bad quality) to 4
(excellent) quality is predicted, and 2) quality control, where the model
predicts whether an image reaches a predefined quality threshold. QA – a
regression task in our case – and QC – a binary classification problem – are
performed automatically by a random forest that uses an ensemble of 332 image
quality metrics (IQMs), extracted from raw T2w stacks, that reflect
complementary quality features based on various statistics computed from image
intensity, brain mask and segmentation (details on IQMs extraction is
available in the Materials and Methods section). Third, by collecting and
manually annotating a very large collection of 1649 low-resolution T2w images
from 233 subjects, acquired in 13 different scanners in four different
institutions across Europe, we can benchmark the generalization of automated
QA/QC models to unseen domains, including existing baselines[14] and pre-
trained deep learning models[35, 21]. A pilot study of this work, including
fewer IQMs and only two centers, was previously presented[36]. The code and
image quality metrics are available at https://github.com/Medical-Image-
Analysis-Laboratory/fetal_brain_qc. While raw data is extremely sensitive and
thus difficult to share, FetMRQC’s IQMs are highly aggregated and do not put
patient privacy at risk.

##  2 Results

Figure 2: A look into the dataset. A – Illustration of the quality rating
interface developed in this work. B – Detailed description of the parameters
from the MRI acquisitions used in the study. Field refers to the magnetic
field of the scanner, TR is the repetition time and TE is the echo time. C –
Inter-rater agreement on the 211 stacks annotated by both raters. The global R
value is 0.75. Note that stacks from La Timone were only annotated by Rater 2.
D – Distribution of the quality ratings across the different sites considered,
on all data. The median values are respectively $1.75$ $[0.84,2.4]$ for BCN,
$1.75$ $[1,2.45]$ for CHUV and $1$ $[0.1,2.05]$ for KISPI.

### Fetal brain MRI acquisition parameters greatly vary across scanners

The analysis of large multi-centric dataset allowed us to quantify the
heterogeneity in acquisition settings. We obtained 1649 stacks of T2w slices
collected from 233 subjects at four different institutions: Lausanne
University Hospital (CHUV) in Switzerland, BCNatal at Hospital Sant Joan de
Déu in Barcelona (Spain), University Children’s Hospital Zürich (KISPI) in
Switzerland and La Timone University Hospital in Marseille, France. The data
were acquired on scanners from different vendors (Siemens at CHUV, BCNatal and
Marseille, General Electrics at KISPI), MR sequences (Half Fourier Single-shot
Turbo spin-Echo –HASTE– for Siemens scanners and Single-Short Fast Spin Echo
–SS-FSE– for GE scanners), magnetic field strengths (1.5 T and 3 T), image
resolutions, fields of view, repetition times and echo times, with both
neurotypical and pathological cases. The acquisition parameters are reported
in Figure 2B, and show a very large variability across scanners and sites. For
instance, the resolution of 1.5 T scanners changes from $1.1\times 1.1$
$\text{mm}^{2}$ (e.g. CHUV - Aera) in-plane to 0.5$\times$0.5 $\text{mm}^{2}$
(e.g. KISPI - Signa Artist), which leads to large differences in signal-to-
noise ratio. In addition, different models using the similar parameters can
also yield largely different images. Examples are shown on Figure 1B.

### FetMRQC’s reports optimize stack screening.

FetMRQC implements a quality rating interface (visual reports), to help make
the process of manual rating of quality standardized and efficient. These
visual reports, illustrated on Figure 2A, are based on the ones developed in
MRIQC[8] and are generated from the raw T2w stacks. Being HTML-based, the
reports can be visualized on any web browser, and effectively remove any bias
due to using different image visualization software. They also facilitate
annotation in an interval scale, as opposed to a categorical approach. We use
four quality ranges: [0,1): exclude – [1,2): poor – [2,3): acceptable – [3,4):
excellent. More details are available in the Materials and Methods section.

Using this interface, Rater 1 annotated 657 stacks, and rater 2 annotated 1203
stacks. 211 of these stacks selected randomly across the training dataset were
annotated by both raters to assess inter-rater reliability. Rater 1, YG, is a
maternal-fetal physician with 5 years of experience, and Rater 2, MBC, is an
engineer with 20 years of experience. The total rating time was 6h 40min for
Rater 1 (median of 36s per volume), and 14h20 for Rater 2 (median of 42s per
volume). A high inter-rater agreement was achieved in the manual quality
annotations, with Pearson’s correlation value of 0.75 overall
($\text{R}^{\text{2}}$=0.56; Figure 2). The inter-rater agreement is
consistently high within each site (2b). On CHUV data, 127 stacks were
manually rated below the exclusion threshold (Quality < 1), and 371 were rated
between poor and excellent. On BCNatal data, 155 stacks were excluded, and 353
rated above the threshold. On KISPI, 218 stacks were rated below 1, and 223
above. On La Timone, 42 stacks were rated below 1, and 161 stacks above. The
average ratio of excluded stacks is $2.04$. Regarding inclusion and exclusion
of stacks (stacks with quality above 1 are included, other are rejected), the
inter-rater agreement yielded a Cohen’s coefficient of $\kappa=0.58$ (moderate
agreement according to Landis and Koch’s interpretation [37]).

While the raters were trained to rate the overall quality of the images, they
also were instructed, but not trained, to rate specific artifacts. They were
asked to rate the degree of fetal motion (visible as discontinuities through-
plane and signal drops in-plane) and bias field, visible as a low-frequency
varying field. The impact of not training the raters for this sub-task is very
substantial: for motion rating, their Pearson’s correlation drops to
$\text{R}^{\text{2}}$=0.15, and for bias rating, $\text{R}^{\text{2}}$=0.02.
We believe that such a low reliability could be avoided by further dedicated
training. In the sequel, we will only use the overall quality rating of the
images.

### Models based on FetMRQC allow robust automated quality control and quality
assessment

Based on the ratings from FetMRQC, we considered two tasks: a quality control
(QC) task, where we aimed at predicting whether a scan should be excluded
(rating below 1), and a quality assessment (QA) task, where we predicted the
interval rating (between 0 and 4). We divided our dataset in two: 1251 stacks
were used for training and testing of the models based on cross-validation
experiments and 398 were used for assessing the generalization to unseen data,
from La Timone and two randomly selected scanners. Data from La Timone were
included in the study specifically to serve as external testing from an unseen
site. Three increasingly challenging evaluation settings are considered: (i)
Subject-wise 10-fold cross validation (CV) on the training stacks, which
quantifies the expected performance of the method on new subjects acquired on
already seen scanners. (ii) Leave-one-Scanner-out (LoSo) CV on the training
stacks, where each fold leaves out all data from a single scanner for
evaluation. This evaluates the expected performance of the method on different
scanners (iii) Pure testing on unseen scanners and an unseen site.This is the
closest to a real-world deployment setting, as the pure testing data were not
seen during the processes of design and training of the models.

For QC, we compared FetMRQC with existing baselines: (i) dl_slice, a slice-
based prediction approach by Xu et al.[21], where the slice-wise prediction
was averaged into a global score and the prediction threshold was learned by
logistic regression. (ii) dl_stack, a stack-based approach by Legorreta et
al.[35]. (iii) NiftyMIC-QC[14], that performs stack-wise evaluation at a
subject level. It uses the brain mask from all stacks of a given subject to
compute the median brain volume, and excludes stacks with a volume below
$70\%$ of the median. To our knowledge, there are no previous works that aimed
at fetal brain MRI QA. As no baseline was available, we constructed a subject-
wise oracle that returns the average rated quality of each subject as quality
assessment. The subject-wise averaged rating was correlated with the stack-
wise rating with a Pearson correlation of $R=0.59$, and served as a coarse
point of comparison for the QA performance of FetMRQC.

In addition, for both QC and QA, we assessed the added value of our proposed
IQMs as follows. First, we constructed a Base version of FetMRQC that used
exclusively a restricted set of six previously proposed quality metrics[17,
38, 21, 35, 14].

Then, we considered two variants of our model: FetMRQC used all estimated 332
IQMs and FetMRQC-20 used only 20 IQMs (selected based on their measured
feature importance on the training data). Note that as this selection was
based on the results in evaluation settings (i) and (ii), the performance of
the model was likely be inflated due to double dipping[39]. It remains
nonetheless informative on which IQMs contribute most to the performance of
FetMRQC. This is further discussed in our last experiment.

Results from the experiment are summarized in Table 1. A more detailed outlook
at the variations in performance across scanners in the LoSo cross-validation
and pure testing performance is available in Figure 3. As expected, the three
increasingly challenging evaluation settings (10-fold CV, LoSo CV, pure
testing) led to a decrease of performance. This decrease is less notable for
QC than QA.

Quality control. Overall, FetMRQC and FetMRQC-20 consistently performed best
with a performance (weighted F1) of 0.86, 0.80 and 0.82 in median for the
cross-validation, leave-one-out scanner and pure testing scenarios
respectively. This performance is consistent across the evaluation metrics
considered (1). Precision is of great interest in our case, as including bad
quality in further analysis can be greatly detrimental to further processing.
FetMRQC shows a consistently high precision in all settings considered, with
median performance of 0.86, 0.85 and 0.83 in CV, LoSo CV and pure testing
respectively.

Focusing on the scanner-wise breakdown of performance (Figure 3A and B),
FetMRQC and FetMRQC-20’s performance is very consistent across almost all
scanners considered, and does not change on new scanners from sites used in
training (Siemens’ MAGNETOM VIDA at CHUV and BCNatal - GE’s DISCOVERY MR750 at
Kispi). On the other hand, DL-based methods[35, 21], trained on homogeneous
data from a single site, fail to perform and exhibit very large variations in
performance across sites, making them generally unreliable. We note also that
a few scanners were consistently challenging for the models. On panel A, we
see that all methods except NiftyMIC-QC and FetMRQC-20 struggled on the CHUV -
Skyra scanner. On panel B, we see that FetMRQC managed to generalize well to
unseen scanners from known sites (BCN, KISPI and CHUV). However, all models,
except dl_slice, poorly generalized to data from La Timone.

Quality assessment. In the case of quality assessment, we observed that
FetMRQC’s new IQMs were instrumental in achieving a performance above the
subject-wise oracle. On Table 1B, we see that while the IQMs used in the base
model ($\text{R}^{2}$=0.49) were sufficient to outperform the subject-wise
oracle ($\text{R}^{2}$=0.33) in the subject-wise CV, using FetMRQC with either
all IQMs ($\text{R}^{2}$=0.44) or the selected 20 ($\text{R}^{2}$=0.49) was
necessary to achieve a performance over the subject-wise oracle
($\text{R}^{2}$=0.29) in the LoSo setting. This was nonetheless not sufficient
to achieve a satisfying performance in the pure testing setting, where
FetMRQC’s prediction, despite outperforming consistently over the base model,
do not outperform the subject-wise oracle. It also fails on one scanner (CHUV
- Magnetom Vida scanner, Figure 3D), but we hypothesize that such drop is
likely due to the small amount of data available from this scanner.

Table 1: Quality control and assessment results. QC (classification, left) and
QA (regression, right) results were averaged over five repetitions of the
experiment. Results are the median cross-validation performance. The number in
parentheses is the average worst-performing cross-validation fold. Three
evaluation settings were considered: 10-fold subject-wise cross-validation
(CV), LoSo CV and pure testing. Pure testing evaluation was grouped by
scanners in the testing set.

Quality control (classification)
---
| Weighted F1 ($\uparrow$) | ROC AUC ($\uparrow$) | Precision ($\uparrow$) | Recall ($\uparrow$)
10-fold subject-wise cross-validation
dl_slice [21] |  $0.64$ ($0.65$) |  $0.72$ ($0.61$) |  $0.71$ ($0.73$) |  $0.98$ ($0.86$)
dl_stack [35] |  $0.71$ ($0.72$) |  $0.77$ ($0.73$) |  $0.78$ ($0.80$) |  $0.85$ ($0.81$)
NiftyMIC-QC [14] | 0.76 (0.75) | – | 0.76 (0.77) | 0.96 (0.96)
Base |  $0.82$ ($0.78$) |  $0.88$ ($0.79$) |  $0.85$ ($0.83$) |  $0.92$ ($0.84$)
FetMRQC |  $0.86$ ($0.79$) |  $0.91$ ($0.87$) |  $0.86$ ($0.85$) |  $0.94$ ($0.86$)
FetMRQC-20 |  $0.86$ ($0.77$) |  $0.92$ ($0.87$) |  $0.86$ ($0.85$) |  $0.93$ ($0.81$)
Leave-one-Scanner-out cross-validation
dl_slice [21] |  $0.61$ $(0.47)$ |  $0.75$ $(0.60)$ |  $0.70$ $(0.62)$ |  $0.96$ $(0.93)$
dl_stack [35] |  $0.64$ $(0.53)$ |  $0.75$ $(0.62)$ |  $0.69$ $(0.47)$ |  $0.90$ $(0.871)$
NiftyMIC-QC [14] | 0.75 (0.66) | – | 0.76 (0.71) | 0.95 (0.86)
Base |  $0.78$ $(0.63)$ |  $0.80$ $(0.76)$ |  $0.80$ $(0.69)$ |  $0.84$ $(0.67)$
FetMRQC |  $0.80$ $(0.64)$ |  $0.89$ $(0.74)$ |  $0.85$ $(0.71)$ |  $0.86$ $(0.73)$
FetMRQC-20 |  $0.82$ $(0.72)$ |  $0.90$ $(0.83)$ |  $0.85$ $(0.76)$ |  $0.88$ $(0.83)$
Pure testing (KISPI + CHUV + La Timone – by scanner)
dl_slice[21] | 0.73 ($0.76$) | 0.79 ($0.79$) | 0.77 (0.77) | 0.97 (0.92)
dl_stack | 0.62 ($0.60$) | 0.72 ($0.51$) | 0.68 (0.67) | 0.97 (0.86)
NiftyMIC-QC[14] | 0.74 (0.52) | – | 0.70 (0.65) | 0.98 (1.00)
Base |  $0.77$ ($0.54$) |  $0.77$ ($0.62$) |  $0.80$ (0.65) |  $0.97$ (1.00)
FetMRQC |  $0.82$ ($0.67$) |  $0.77$ ($0.76$) |  $0.83$ (0.70) |  $0.91$ (0.91)
FetMRQC-20 |  $0.79$ ($0.56$) |  $0.74$ ($0.64$) |  $0.78$ (0.65) |  $0.93$ (0.94)

Quality assessment (regression)
---
|  $R^{2}$ ($\uparrow$) | Spearman ($\uparrow$) | MAE ($\downarrow$)
10-fold subject-wise cross-validation
Subject-wise oracle | 0.33 (0.39) | 0.53 (0.68) | 0.65 (0.61)
Base |  $0.40$ (0.38) |  $0.69$ (0.68) |  $0.59$ (0.61)
FetMRQC |  $0.60$ (0.49) |  $0.80$ (0.75) |  $0.50$ (0.56)
FetMRQC-20 |  $0.60$ (0.53) |  $0.79$ (0.78) |  $0.50$ (0.53)
Leave-one-Scanner-out cross-validation
Subject-wise oracle | 0.29 (0.40) | 0.48 (0.58) | 0.64 (0.64)
Base |  $0.29$ (0.25) |  $0.59$ (0.48) |  $0.64$ (0.66)
FetMRQC |  $0.45$ (0.39) |  $0.74$ (0.72) |  $0.56$ (0.60)
FetMRQC-20 |  $0.52$ (0.36) |  $0.77$ (0.71) |  $0.55$ (0.62)
Pure testing (KISPI + CHUV + La Timone – by scanner)
Subject-wise oracle | 0.41 ( 0.41) | 0.60 (0.60) | 0.45 (0.45)
Base |  $0.26$ ( 0.36) |  $0.45$ (0.47) |  $0.65$ (0.37)
FetMRQC |  $0.35$ (-0.74) |  $0.59$ (0.39) |  $0.51$ (0.65)
FetMRQC-20 |  $0.30$ (-0.94) |  $0.54$ (0.31) |  $0.53$ (0.68)

Figure 3: Scanner-wise results for QA/QC. A – Weighted F1 score for the QC
task for each scanner used in LoSo cross-validation (sorted from the one with
the least subjects to the most subjects). B – Weighted F1 score for the QC
task for each scanner used in the pure testing set. C – $\text{R}^{\text{2}}$
for the QA task for each scanner used in LoSo cross-validation. D –
$\text{R}^{\text{2}}$ for the QA task for each scanner used in the pure
testing set. Distribution of scores is aggregated by scanner, and the median
performance for each method is shown as the black dashed line. The red line in
the prediction task at $0$ shows the baselines for a constant predictor. These
results detail the ones presented in Table 1.

### Scanner diversity in training set greatly improves generalization

Data annotation is known to be a time-consuming process that requires highly
specialized raters[40]. Given a limited budget (in time and expertise), the
question of which data to annotate then raises naturally. In this second
experiment, we investigated how the number of scanners $n_{\text{scanner}}$
and the number of data $n_{\text{training}}$ available during training
impacted the generalization performance of FetMRQC in the context of LoSo CV.
We had in total 8 different scanners and 1251 data points. For a given
configuration ($n_{\text{scanner}}$, $n_{\text{training}}$), we performed a
LoSo CV where the data used in training were subsampled: between 1 and 7
scanners were sampled randomly from the available data and between 100 and 900
data points were then randomly sampled from the available scanners. For each
($n_{\text{scanner}}$, $n_{\text{training}}$), the experiment was repeated 20
times.

Figure 4 contains the results of the experiment, showing the minimum, maximum
and median performance with the deviation from the median, across 20
repetitions. In each case entry, the reported measure was computed as the
average across the 20 repetitions. Looking at the median performance, it is
clear that increasing the size of the training set (x axis) or the number of
scanners (y axis) both improve the generalization. Starting with best-case
generalization (maximum performance, lower row in figure 4), we see that in
every case, there is a subset of data that enables reaching the best
performance with only 100 data points. While this is not surprising, this is
also difficult to exploit: one cannot readily find ahead of time a subset of
data that will generalize well to the testing data. The worst-case
generalization is more interesting: using 100 training data points from seven
scanners reaches a similar performance as using 700 data points from four
scanners in the case of classification. Likewise, 600 data points from four
scanners in the case of regression. Overall, using multiple scanners is key to
achieving the highest performance regimes, but using more data is also greatly
valuable. However, if constrained to a limited annotation budget, we
anticipate that annotating more diverse data from various scanners will be
more helpful for generalization than gathering a large corpus from a single
scanner.

In addition, we also observe, on the median performance, that the
classification task is generally more straightforward than the regression
task: fewer data allow to reach the highest performance, while performance
keeps increasing for regression when adding more sites and more data. Thus, we
hypothesize that regression performance would further be increased by
increasing the size of training data. In contrast, the median classification
performance might stagnate, although its worth case performance might still
improve, thus making the model more robust to new scanners by further
enhancing the training dataset.

Figure 4: Performance as a function of the number of scanners and training
points. This is obtained by performing leave-one-scanner-out cross-validation
20 times, using different random subsets of data. (Top row.) Minimum (worst-
case) performance across folds (Middle row.) Median performance across folds.
The smaller plots show the corresponding median average deviation. (Bottom
row.) Maximum (best-case) performance across folds.

### A few IQMs can account for most of the performance of FetMRQC.

FetMRQC relies 332 different IQMs that are not fully independent from each
other, as shown in Figure 6. In this final experiment, we explore the IQMs
that are most important for FetMRQC QA and QC models.

We computed the feature importance of the random forest model used in each
fold of the LoSo CV and average them across folds. We grouped together the
IQMs with a correlation coefficient above 0.95 (as shown in Figure 6) to
prevent several IQMs contributing very similar information but selected by
different models in the LoSo CV for QA and QC. We then randomly selected a
single IQM from each correlated group, and arrived at the ranking shown in the
top row of Figure 5. First, we see that in the QC task (A), IQMs are generally
spread out (the top four IQMs sum up to 0.20). In the regression task (B)
however, a few IQMs capture a large part of the feature importance (the top
four IQMs sum up to 0.53). Nonetheless, three IQMs are consistently among the
top predictors: rank-based error[23], the volume of the brain mask and the
morphological closing of the brain mask. The first estimates the consistency
of the intensities across slices by computing how well a low-rank
approximation can represent the volume, the second estimates the volume of the
brain and the third estimates the degree of motion across stacks by computing
a morphological closing of the brain mask in the through-plane direction and
then subtracting the original brain mask. The first two IQMs are the ones that
have been used in NiftyMIC-QC[14] and complement each other well. Secondly, we
see that although the ranking of the most important IQMs can vary, overall 19
out of the 25 IQMs of Figure 5A and B appear in common in both tasks as the
most important IQMs. Thirdly, let us note that the best IQMs cover different
representative families of features: intensity-based, mask (or shape)-based,
and segmentation-based IQMs. Finally, note that features proposed within
FetMRQC rank highly in terms of feature importance: 14 out of the 25 IQMs
shown in Figure 5A and B were proposed in this work.

FetMRQC-20 is built on the feature importance obtained for FetMRQC (Figure 5A
and B). The IQMs were selected by averaging the feature importance from QC and
QA, and then by selecting the top-20 features. In order to keep the reduced
model as interpretable as possible, we excluded the deep learning (DL)-based
IQMs from FetMRQC-20 and replaced them with the two features that came next in
line. Results in Table 1 show that does not yield a decrease in performance.
The feature importance using only FetMRQC-20’s IQMs is shown on Figure 5C and
D and is generally consistent with FetMRQC’s results. As fewer IQMs are
available, their relative importance is generally higher, and the same IQMs
end up carrying the largest weight in decision.

Figure 5: Most important IQMs for QA/QC. Feature importance for quality
control (classification) on the left, and for quality assessment (regression)
on the right. The top row shows the top-25 IQMs from FetMRQC and the bottom
row shows the 20 selected IQMs that form FetMRQC-20. Blue IQMs are intensity-
based, orange are mask- (or shape) based, green are segmentation based, pink
are deep-learning based and brown are metadata based. Hatched features denote
the new ones proposed in this work. The error bars are the standard deviation
over the different cross-validation folds, performed over different scanners.
Note that the scales are very different between the plots: the highest feature
importance for classification is around 0.055, whereas it is around 0.23 for
regression.

##  3 Discussion

In this work, we proposed FetMRQC, a novel open-source machine learning
framework for the automated quality control and quality assessment of fetal
brain MRI. While most existing works focus on a single-center, single-scanner
setting[35, 21, 24], the evaluation in this work was carried out on a large,
multi-scanner, multi-centric dataset. These diverse data allowed us to measure
the impact of domain shift on generalization, and assess the variability in
performance across scanners. Being trained with multi-centric data FetMRQC
achieves a reliable performance in quality control over most scanners
considered, which is not the case for baseline DL-methods, trained on
homogeneous data, which exhibit a very large variability in performance. These
observations were made possible by following good practices regarding
evaluation and reporting of dataset with domain shifts[41, 27, 42]. Indeed,
cross-validation at the group level (subject or scanner in our case)[43],
computing the performance metrics at the group level and reporting the worst-
performing site were essential in unfolding the large variability in
performance, which is obfuscated when averaging across the entire testing
set[44, 45]. Designing a pure testing[27, 46] set comporting both unseen
scanners and unseen sites allowed to observe another trend: methods performing
well in the LoSo CV setting performed well on unseen scanners from known
sites, but struggled on the unseen site. Indeed, data from La Timone were very
different from the ones acquired at other institutions: only three stacks were
acquired per subject due to strong constraints on the duration of the scanning
session, and the acquisition was done at a high in-plane resolution, leading
to higher level of noise in the images compared to the rest of the data.

Beyond measuring the impact of domain shifts, several methods to correct and
compensate them on tabular data have been proposed, including group-wise
normalization of data[8], or empirical Bayes approaches, like ComBat[47]. As
shown in our supplementary experiments, we did not find these approaches to be
beneficial in our case, which is most likely due to the quality of data being
related to the scanner on which data were acquired: removing the scanner
information at the IQM level might not be helpful because it might remove
meaningful information[44]. This might be mitigated by attempting to directly
harmonize the input T2w images[45, 48] rather than the IQMs, as the IQMs were
directly extracted from images acquired with widely different imaging
parameters that could induce some confounding factors in the derived metrics.

However, FetMRQC suffers from two main limitations. As any other supervised
learning method, the first limitation comes from an often underestimated
component of machine learning pipelines, namely the quality of annotations. As
QA/QC has an inherently subjective dimension, narrowing the task at hand for
rating is key to maximize inter-rater reliability[49, 40]. The quality rating
interface is an essential tool for displaying the raw T2w fetal brain data
uniformly, and when providing the raters with a training session, can
successfully lead to high inter-rater reliability. However, our fetal motion
and bias field rating results suggest that a finer protocol is needed. The
protocol should, in particular, encourage raters to proceed in artifact-based
quality ratings: first assessing the presence and degree of various artifacts
and then deciding on a score to give rather than the opposite. Improving the
inter-rater agreement might further improve the quality of FetMRQC, in
particular on the quality assessment task, where the subject-wise CV
regression performance comes close to the level of agreement between the
raters: $\text{R}^{\text{2}}$=0.58 for the subject-wise CV and the inter-rater
agreement has $\text{R}^{\text{2}}$=0.56. A second limitation comes from the
simplicity of the model: while FetMRQC’s predictions are easily interpretable
and generally depend on a small number of IQMs, its learning capabilities are
limited by its shallow nature. A deep learning model trained directly on 3D
clinical acquisitions is likely to improve QA/QC predictions, if enough
training data is available, as it can make better use of large amounts of
training data.

Beyond addressing these limitations, future work will investigate how
preprocessing the raw T2w data might impact FetMRQC’s performance. Future work
will also include a more thorough evaluation of the impact of FetMRQC on
downstream tasks such as super-resolution reconstruction quality. FetMRQC is
only a first step towards robust tools for quantitative analysis of fetal
neuroimaging. While QA/QC starts at the raw images, it is greatly needed at
every stage of the fetal brain MRI pipeline, from acquisition to
reconstruction to surface extraction. Such checkpoints, along with community
efforts in collecting large, reality-centric datasets are key to developing
robust and reliable learning-based approaches for fetal neuroimaging and
beyond.

## Materials and Methods

FetMRQC comprehends two major elements to implement QA/QC protocols of
unprocessed (stacks of 2D slices) fetal brain MRI data. First, the tool builds
upon MRIQC’s framework and generates an individual QA report for each stack to
assist and optimize screening and annotation by experts. Second, FetMRQC
proposes to train machine learning models based on image quality metrics
(IQMs).

### Data

Our experiments are carried out on T2-weighted 2D stacks of slices from 233
subjects (1649 scans) retrieved from existing databases at four different
institutions, including both neurotypical and pathological cases. The
corresponding local ethics committees independently approved the studies under
which data were collected, and all participants gave written informed consent.
Full details of imaging parameters in the acquisitions at each institution are
provided in Figure 2B.

Lausanne University Hospital (CHUV), Switzerland, provided 61 subjects (498
scans), with an average of 7.9$\pm$3.0 stacks per subject. BCNatal (Hospital
Sant Joan de Déu, Barcelona, Spain) provided 85 subjects (508 scans), $5.8\pm
3.4$ stacks per subject. University Children’s Hospital Zürich (KISPI),
Switzerland, provided 19 subjects (441 scans) with 23.2$\pm$5.36 stacks per
subject. La Timone University Hospital, Marseille, France, provided 68
subjects (203 scans) with 3 stacks per subject. Note that the distribution of
data from La Timone is different from other sites because acquisition duration
is limited due to constraints related to the clinical routine. After the
exclusion of scanners with insufficient data (CHUV - Siemens Avanto with 5
stacks), the aggregate sample size is N=1649 stacks.

We constructed a pure testing dataset with the data from La Timone, and the
data from two randomly selected scanners from the other sites. The held-out
scanners were the Siemens Magnetom Vida, used at CHUV and BCNatal, and the GE
Discovery MR750 used at KISPI. In total, the pure testing dataset (La Timone +
held out scanners) consisted of 398 stacks.

### Manual QA of fetal MRI stacks

Akin to MRIQC[8], FetMRQC generates an HTML-based report adapted to the QA of
fetal brains for each input stack of 2D slices (Figure 2A). The input dataset
is required to comply with the Brain Imaging Data Structure (BIDS[50]), a
format widely adopted in the neuroimaging community. The reports are generated
using an image with a corresponding brain mask. This mask can be extracted
automatically, and in this work, we used MONAIfbs[51]. Each individual-stack
report has a QA utility (the so-called rating widget), with which raters can
fill in an overall quality score, the in-plane orientation, and the presence
and grading of artifacts visible in the stack. We use an interval (as opposed
to categorical) rating scale with four main quality ranges: [0,1): exclude –
[1,2): poor – [2,3): acceptable – [3,4): excellent. Interval ratings simplify
statistical modeling, set lower bounds to annotation noise, and enable the
inference task where a continuous quality score is assigned to input images
rather than broad categories. In addition, a navigation menu allows the rater
to access all reports in a centralized location, and by being able to access
the next image to be rated in a single click.

### IQMs extraction and learning

Table 2: Detailed description of the Image Quality Metrics (IQMs) computed in
FetMRQC. The number in parentheses are the total available variants on each
metric (e.g. computation on the masked image, on the central slices, etc.).
The number of IQMs sums up to 166 variants. The final number is doubled by
incorporating an indicator variable of whether a given entry failed to be
computed, resulting in a NaN (not-a-number).  Intensity-based metrics
---
rank_error[23] |  (5) |  Measure the compressibility of the image using a low-rank approximation
slice_loss |  (32) |  Use metrics commonly used for outlier rejection [17, 23, 14] to compute the difference between slices in the volume. We considered (normalized) mean averaged error, (normalized) mutual information, normalized cross correlation, (normalized) root mean squared error, peak signal-to-noise ration, structural similarity and joint entropy.
sstats [8] |  (14) |  Compute the mean, median, standard deviation, percentiles $5\%$ and $95\%$, coefficient of variation and kurtosis on brain ROI.
entropy [8] |  (2) |  Measure the overall entropy of the image.
bias |  (3) |  Level of bias estimated using N4 bias field correction [52]
filter_image |  (4) |  Estimate the sharpness by using Laplace and Sobel filters (commonly used for edge detection)
Mask-based metrics
mask_volume |  (1) |  Compute the volume of the brain mask.
centroid[38] |  (2) |  Measure the variance in the center of mass of the brain mask across slices.
closing_mask |  (2) |  Morphological closing of the brain mask in the through-plane direction, to detect inter-slice motion. Report the average difference with the original mask.
filter_mask |  (4) |  Estimate the sharpness of the brain mask using Laplace and Sobel filtering. In an ideal case, the brain mask would be smoothly varying, especially in the through-plane direction.
Segmentation-based metrics [8]
sstats[8] |  (64) |  Summary statistics on each region of the segmentation (white matter (WM), gray matter(GM) and cerebrospinal fluid (CSF). Computing mean, median, 5th and 95th percentile, kurtosis, standard deviation, mean absolute deviation and number of voxels)
volume |  (6) |  Volume of the three brain regions (WM, GM, CSF; entire brain/central slices)
SNR[8, 53] |  (10) |  Signal-to-noise computed in each region (background, WM, GM, CSF and globally)
CNR[8, 54] |  (2) |  Contrast-to-noise-ratio, to estimate the separation between GM and WM.
CJV[8, 55] |  (2) |  Coefficient of joint variation of GM and WM.
WM2Max[8] |  (2) |  White-matter to maximum intensity ratio.
Deep learning-based metrics
dl_slice[21] |  (5) |  Slice-wise deep learning-based quality assessment. Several variants are considered: full image/central-slices, uncropped/cropped image around the ROI, and using only $p_{\text{good}}$ for scoring.
dl_stack[35] |  (1) |  Stack-wise deep learning-based quality assessment.
Metadata-based metrics
im_size |  (5) |  voxel size (in-plane $x$ and $y$ and through-plane), as well as voxel-size and in-plane pixel dimension.

#### IQMs in fetal brain MRI

Leveraging the same workflow that generates the individual reports, a number
of IQMs are extracted from the entire raw stack and from the information
within the brain mask. In addition, some IQMs are computed based on a
segmentation of the 2D slices. For this, a nnUNet-v2[56] model was trained on
the FeTA dataset[33], a public dataset consisting of super-resolution
reconstructed fetal brain images along with manual segmentation. The
segmentations are done over eight different classes, which we merge into three
groups: white matter (excluding corpus callosum), cerebrospinal fluid (CSF;
intra-axial and extra-axial), and gray matter (cortical and deep). This is
done to enable the use of the segmentation-based IQMs from MRIQC[8], which
only use these three groups.

Besides this, since only a few IQMs defined by MRIQC can be applied to fetal
brains, we implement a set of IQMs previously proposed specifically to this
application: the method of Kainz et al.[23], rank_error, is based on the
estimated compressibility of a stack. In addition, we compute measures derived
from the brain mask such as the volume of the brain mask, mask_volume, used by
Ebner et al. (2020)[14] or its centroid, used by de Dumast et al. (2020)[38].
We also use recently proposed slice-wise and stack-wise deep learning-based
IQMs, dl_slice[21] and dl_stack [35]. We use the pre-trained models, as we
want to test the off-the-shelf value of these IQMs. Note that the method of
Liao et al.[20] was not included because their code is not publicly available
and we could not get in contact with the authors. The method of Xu et al.
(2020)[21] predicts simultaneously whether a slice contains some brain volume,
and whether this slice is of good quality. By default, we computed the score
as
$\frac{1}{n_{\text{slices}}}\sum_{i=1}^{n_{\text{slices}}}p_{i,\text{pass}}-p_{i,\text{fail}}$
to get a score between -1 and 1.

Along with these IQMs, we also propose additional IQMs for quality prediction
that have not previously been used in the context of fetal brain MRI. They can
be roughly categorized into four groups, and the full details are provided in
Table 2. In a nutshell, intensity-based IQMs directly rely on the voxel values
of the image. These include summary statistics[8] such as mean, median, and
percentiles. We also reuse metrics traditionally used for outlier
rejection[17, 23, 14] to quantify the intensity difference between slices in a
volume. We also compute entropy[8], estimate the level of bias using N4 bias
field correction[52] and estimate the sharpness of the image with Laplace and
Sobel filters. The second type of metrics are mask-based and operate directly
on the automatically extracted brain mask. We propose to use a morphological
closing to detect through-plane motion, as well as edge detection, to estimate
the variation of the surface of the brain mask, using Laplace and Sobel
filters. The third type of IQMs corresponds to metrics originally proposed in
the context of MRIQC[8]. These are segmentation-based and include region-wise
summary statistics, region-wise volume, region-wise signal-to-noise ratio[53],
contrast-to-noise ratio between white matter (WM) and gray matter (GM)[54],
coefficient of joint variation between gray matter and white matter[55] and
white matter to maximum intensity ratio[8].

#### Variations on the metrics

All the IQMs operate on raw T2-weighted 2D images and/or masks and can be
modified by various data transformations. For example, Kainz et al.[23]
evaluated their metrics only on the third of the slices closest to the center
of a given volume. We include various variations on the IQMs, including
considering the whole ROI instead of the centermost slices. Other variations
include keeping or masking the maternal background, aggregating point
estimates using mean, median, or other estimators, computing information
theoretic metrics on the union or intersection of masks, etc. Finally,
slice_loss metrics can be either computed as a pairwise comparison between all
slices (by default) or only on a window of neighboring slices. With all the
different variations, we obtain a total of 166 different IQMs.

#### Missing values

In addition to the previously described IQMs, we also include a boolean
variable that assesses whether a given IQM computation failed. If this occurs,
the IQM will have a zero value and a flag will be set to true. This allows to
keep all IQMs values to a real number.

This adds up to a total of 332 IQMs. Figure 6 shows a cross-correlation matrix
on the entire training dataset of the 100 IQMs most frequently used.

#### IQM normalization

Batch effects (domain shift) can induce substantial biases in IQM
computations. We experimented with the following normalization techniques:
standard, robust (median-based) and quantile scaling, standard, robust and
quantile group scaling (scaling by subject/scanner/site) and ComBat[47].

Although feature standardization is important for models such as logistic or
linear regression, this is not the case for tree-based models. We explore this
in supplementary experiments, where we see that FetMRQC does not benefit from
any of the normalization techniques. In some cases, such as subject-wise
scaling, this even proved detrimental, as the standardization was removing
some relevant differences between subjects.

#### Feature selection and dimensionality reduction

Correlated and irrelevant features can also be an obstacle for machine
learning models. In supplementary experiments, we experiment with dropping
IQMs that were highly correlated with each other (with thresholds of 0.8 and
0.9), to remove constant features, and we experimented with removing features
that do not contribute more than noise using the Winnow algorithm[57] with
extremely randomized trees[8]. Finally, we also explore using principal
component analysis to construct orthogonal features. The results show that
these steps do not improve the performance of the model. They were not
included in the final model.

### Model evaluation and selection

Model evaluation. In our initial experiments, we used nested cross-validation
to automatically perform model selection and evaluation without introducing
optimistic biases[43]. We performed model selection by ablating over the
previously mentioned feature normalization and feature selection options, as
well as various models. For the QA (regression) task, we considered linear
regression, support vector machine, random forests, and XGBoost’s regression
model[58]. For the QC (classification) task, we considered logistic
regression, support vector classifier, random forest, and XGBoost’s
classification model[58]. However, in the large majority of these initial
experiments, the best-performing configuration used no standardization, no
feature selection, and random forests for both classification and regression.
This is detailed in the ablation studies in the supplementary material. As we
decided to use a random forest without standardization or feature selection,
nested cross-validation was not required anymore and is not used in the rest
of the paper.

Baselines. For classification, we consider the following baselines. We first
include NiftyMIC-QC[14], which computes the volume of the brain for each stack
and, for each subject, excludes the stacks with a volume below 70% of the
median volume. We also include the deep learning methods of Legorreta et
al.[35] (dl_stack) and Xu et al.[21] (dl_slice). These IQMs are computed for
each individual subject. We then standardize them and train a logistic
regression model to adjust their prediction to the statistics of our dataset.
This step adjusts the threshold for prediction and can only be beneficial to
the prediction accuracy of these baselines.

For regression, as there are no baselines available, we consider a simple
model predicting only subject-wise class statistics for regression, predicting
the average subject quality for each subject. This method is an oracle,
showing a base to outperform in order to expect meaningful results.

Ablations on FetMRQC’s IQMs. Instead of considering various models, we focused
on evaluating the predictive power of different sets of IQMs. We first
considered a base model, using the state-of-the-art IQMs: the rank error of
Kainz et al.[23], the centroid of de Dumast et al.[38], the mask volume of
Ebner et al.[14], the normalized cross-correlation and mutual information
commonly used as criteria for outlier rejection[17, 14], the stack-based deep
learning method of Legorreta et al.[35] as well as the slice-based method of
Xu et al. (2020)[21]. We considered two models: FetMRQC, using all 332 IQMs
and a reduced model, FetMRQC-20, using only the 20 most important IQMs. A
summary of the methods used it provided in Table 3

Table 3: Summary of the methods compared in the paper. dl_slice |  Slice-wise deep learning (DL) quality control of Xu et al.[21], aggregated into a single score. The decision threshold is learned by logistic regression.
---|---
dl_stack |  Stack-wise DL QC of Legorreta et al.[35]. The decision threshold is learned by logistic regression.
NiftyMIC-QC |  Subject-wise QC, excluding stacks with brain volume below 70% of the median brain volume calculated for the subject.
Base |  Base version of FetMRQC using 6 IQMs: rank error[23], mask centroid[38], mask volume[14], normalized cross-correlation, mutual information[17, 14], dl_stack[35], dl_slice[21]
FetMRQC |  Full version of FetMRQC, using 332 IQMs
FetMRQC-20 |  Use the 20 best IQMs of FetMRQC – rank_error, closing_mask_full, mask_volume, filter_mask_Laplace, filter_mask_sobel_full, nRMSE_window, filter_mask_Laplace_full, filter_mask_Laplace, closing_mask, rank_error_center, seg_sstats_BG_N, centroid, rank_error_center_relative, seg_sstats_CSF_N, seg_sstats_GM_N, im_size_z, NCC_intersection, NCC_window, PSNR_window, seg_SNR_WM, seg_volume_GM.
Subject-wise oracle |  For each subject, compute the average stack quality and return this value for all the stacks of the subject.

Evaluation metrics. Our regression results are evaluated using Pearson’s
$R^{2}$ score, Spearman rank correlation, and mean absolute error (MAE). Our
classification results use a weighted F1-score, to handle imbalanced classes,
and the area under the receiver operating characteristic curve (ROC AUC), as
well as precision and recall.

Implementation. The experiments were implemented with Python 3.9.15 and
Scikit-learn 1.1.3[59].

Figure 6: Correlation matrix between the top 100 features used in FetMRQC,
according to feature importance, evaluated on the entire dataset. Blue refers
to negative correlations, and red to positive ones. The scale goes between -1
and 1. The features are clustered by similarity.

## References

  * [1] Mortamet, B. _et al._ Automatic quality assessment in structural brain magnetic resonance imaging.  _Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine_ 62, 365–372 (2009).
  * [2] Niso, G. _et al._ Open and reproducible neuroimaging: from study inception to publication.  _NeuroImage_ 119623 (2022).
  * [3] Rosen, A. F. _et al._ Quantitative assessment of structural image quality.  _Neuroimage_ 169, 407–418 (2018).
  * [4] Alfaro-Almagro, F. _et al._ Image processing and quality control for the first 10,000 brain imaging datasets from uk biobank.  _Neuroimage_ 166, 400–424 (2018).
  * [5] Power, J. D., Barnes, K. A., Snyder, A. Z., Schlaggar, B. L. & Petersen, S. E.  Spurious but systematic correlations in functional connectivity mri networks arise from subject motion.  _Neuroimage_ 59, 2142–2154 (2012).
  * [6] Reuter, M. _et al._ Head motion during mri acquisition reduces gray matter volume and thickness estimates.  _Neuroimage_ 107, 107–115 (2015).
  * [7] Alexander-Bloch, A. _et al._ Subtle in-scanner motion biases automated measurement of brain anatomy from in vivo mri.  _Human brain mapping_ 37, 2385–2397 (2016).
  * [8] Esteban, O. _et al._ MRIQC: Advancing the automatic prediction of image quality in MRI from unseen sites.  _PloS one_ 12, e0184661 (2017).
  * [9] Klapwijk, E. T., Van De Kamp, F., Van Der Meulen, M., Peters, S. & Wierenga, L. M.  Qoala-t: A supervised-learning tool for quality control of freesurfer segmented mri data.  _Neuroimage_ 189, 116–129 (2019).
  * [10] Vogelbacher, C. _et al._ Lab–qa2go: a free, easy-to-use toolbox for the quality assessment of magnetic resonance imaging data.  _Frontiers in neuroscience_ 13, 688 (2019).
  * [11] Tortori-Donati, P., Rossi, A., Girard, N. & Huisman, T. A.  Fetal magnetic resonance imaging of the central nervous system.  _Pediatric Neuroradiology: Brain_ 1219–1253 (2005).
  * [12] Gholipour, A. _et al._ Fetal MRI: a technical update with educational aspirations.  _Concepts in Magnetic Resonance Part A_ 43, 237–266 (2014).
  * [13] Di Martino, A. _et al._ The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism.  _Molecular psychiatry_ 19, 659–667 (2014).
  * [14] Ebner, M. _et al._ An automated framework for localization, segmentation and super-resolution reconstruction of fetal brain mri.  _NeuroImage_ 206, 116324 (2020).
  * [15] Xu, J. _et al._ NeSVoR: Implicit neural representation for slice-to-volume reconstruction in MRI.  _IEEE Transactions on Medical Imaging_ (2023).
  * [16] Uus, A. U. _et al._ Automated 3d reconstruction of the fetal thorax in the standard atlas space from motion-corrupted mri stacks for 21–36 weeks ga range.  _Medical image analysis_ 80, 102484 (2022).
  * [17] Kuklisova-Murgasova, M., Quaghebeur, G., Rutherford, M. A., Hajnal, J. V. & Schnabel, J. A.  Reconstruction of fetal brain MRI with intensity matching and complete outlier removal.  _Medical image analysis_ 16, 1550–1564 (2012).
  * [18] Tourbier, S. _et al._ An efficient total variation algorithm for super-resolution in fetal brain mri with adaptive regularization.  _NeuroImage_ 118, 584–597 (2015).
  * [19] Lala, S. _et al._ A deep learning approach for image quality assessment of fetal brain MRI.  In _Proceedings of the 27th Annual Meeting of ISMRM, Montréal, Québec, Canada_ , 839 (2019).
  * [20] Liao, L. _et al._ Joint image quality assessment and brain extraction of fetal mri using deep learning.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part VI 23_ , 415–424 (Springer, 2020).
  * [21] Xu, J. _et al._ Semi-supervised learning for fetal brain mri quality assessment with roi consistency.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part VI 23_ , 386–395 (Springer, 2020).
  * [22] Uus, A. U. _et al._ Retrospective motion correction in foetal MRI for clinical applications: existing methods, applications and integration into clinical practice.  _The British Journal of Radiology_ 95, 20220071 (2022).
  * [23] Kainz, B. _et al._ Fast volume reconstruction from motion corrupted stacks of 2d slices.  _IEEE transactions on medical imaging_ 34, 1901–1913 (2015).
  * [24] Gagoski, B. _et al._ Automated detection and reacquisition of motion-degraded images in fetal haste imaging at 3 t.  _Magnetic resonance in medicine_ 87, 1914–1922 (2022).
  * [25] Willemink, M. J. _et al._ Preparing medical imaging data for machine learning.  _Radiology_ 295, 4–15 (2020).
  * [26] Sambasivan, N. _et al._ “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai.  In _proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , 1–15 (2021).
  * [27] Varoquaux, G. & Cheplygina, V.  Machine learning for medical imaging: methodological failures and recommendations for the future.  _NPJ digital medicine_ 5, 48 (2022).
  * [28] Mueller, S. G. _et al._ Ways toward an early diagnosis in alzheimer’s disease: the alzheimer’s disease neuroimaging initiative (adni).  _Alzheimer’s & Dementia_ 1, 55–66 (2005).
  * [29] Markiewicz, C. J. _et al._ The openneuro resource for sharing of neuroscience data.  _Elife_ 10, e71774 (2021).
  * [30] Van Essen, D. C. _et al._ The wu-minn human connectome project: an overview.  _Neuroimage_ 80, 62–79 (2013).
  * [31] Makropoulos, A. _et al._ The developing human connectome project: A minimal processing pipeline for neonatal cortical surface reconstruction.  _Neuroimage_ 173, 88–112 (2018).
  * [32] Casey, B. J. _et al._ The adolescent brain cognitive development (abcd) study: imaging acquisition across 21 sites.  _Developmental cognitive neuroscience_ 32, 43–54 (2018).
  * [33] Payette, K. _et al._ An automatic multi-tissue human fetal brain segmentation benchmark using the fetal tissue annotation dataset.  _Scientific Data_ 8, DOI: 10.1038/s41597-021-00946-3 (2021).
  * [34] Payette, K. _et al._ Fetal brain tissue annotation and segmentation challenge results.  _Medical Image Analysis_ 88, 102833 (2023).
  * [35] Legorreta, I., Samal, S., Zhang, J. & Im, K.  Automatic fetal brain MRI quality assessment. Github repository: https://github.com/FNNDSC/pl-fetal-brain-assessment.
  * [36] Sanchez, T., Esteban, O., Gomez, Y., Eixarch, E. & Bach Cuadra, M.  Fetmrqc: Automated quality control for fetal brain mri.  In _Perinatal, Preterm and Paediatric Image Analysis_ , 3–16 (Springer Nature Switzerland, Cham, 2023).
  * [37] Landis, J. R. & Koch, G. G.  An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers.  _Biometrics_ 363–374 (1977).
  * [38] de Dumast, P. _et al._ Translating fetal brain magnetic resonance image super-resolution into the clinical environment.  _European Congress of Magnetic Resonance in Neuropediatrics_ (2020).
  * [39] Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. & Baker, C. I.  Circular analysis in systems neuroscience: the dangers of double dipping.  _Nature neuroscience_ 12, 535–540 (2009).
  * [40] Rädsch, T. _et al._ Labelling instructions matter in biomedical image analysis.  _Nature Machine Intelligence_ 5, 273–283 (2023).
  * [41] Roberts, M. _et al._ Common pitfalls and recommendations for using machine learning to detect and prognosticate for covid-19 using chest radiographs and ct scans.  _Nature Machine Intelligence_ 3, 199–217 (2021).
  * [42] Zech, J. R. _et al._ Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study.  _PLoS medicine_ 15, e1002683 (2018).
  * [43] Varoquaux, G. _et al._ Assessing and tuning brain decoders: cross-validation, caveats, and guidelines.  _NeuroImage_ 145, 166–179 (2017).
  * [44] Dockès, J., Varoquaux, G. & Poline, J.-B.  Preventing dataset shift from breaking machine-learning biomarkers.  _GigaScience_ 10, giab055 (2021).
  * [45] Zhou, K., Liu, Z., Qiao, Y., Xiang, T. & Loy, C. C.  Domain generalization: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2022).
  * [46] Kapoor, S. & Narayanan, A.  Leakage and the reproducibility crisis in ml-based science.  _arXiv preprint arXiv:2207.07048_ (2022).
  * [47] Johnson, W. E., Li, C. & Rabinovic, A.  Adjusting batch effects in microarray expression data using empirical bayes methods.  _Biostatistics_ 8, 118–127 (2007).
  * [48] Wang, J. _et al._ Generalizing to unseen domains: A survey on domain generalization.  _IEEE Transactions on Knowledge and Data Engineering_ (2022).
  * [49] Esteban, O., Poldrack, R. A. & Gorgolewski, K. J.  Improving out-of-sample prediction of quality of MRIQC.  In _International Workshop on Large-scale Annotation of Biomedical data and Expert Label Synthesis_ , 190–199 (Springer, 2018).
  * [50] Gorgolewski, K. J. _et al._ The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments.  _Scientific data_ 3, 1–9 (2016).
  * [51] Ranzini, M., Fidon, L., Ourselin, S., Modat, M. & Vercauteren, T.  Monaifbs: Monai-based fetal brain mri deep learning segmentation.  _arXiv preprint arXiv:2103.13314_ (2021).
  * [52] Tustison, N. _et al._ N4itk: Improved N3 bias correction.  _IEEE Transactions on Medical Imaging_ 29, DOI: 10.1109/TMI.2010.2046908 (2010).
  * [53] Dietrich, O., Raya, J. G., Reeder, S. B., Reiser, M. F. & Schoenberg, S. O.  Measurement of signal-to-noise ratios in mr images: influence of multichannel coils, parallel imaging, and reconstruction filters.  _Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine_ 26, 375–385 (2007).
  * [54] Magnotta, V. A., Friedman, L. & BIRN, F.  Measurement of signal-to-noise and contrast-to-noise in the fbirn multicenter imaging study.  _Journal of digital imaging_ 19, 140–147 (2006).
  * [55] Ganzetti, M., Wenderoth, N. & Mantini, D.  Intensity inhomogeneity correction of structural mr images: a data-driven approach to define input algorithm parameters.  _Frontiers in neuroinformatics_ 10, 10 (2016).
  * [56] Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J. & Maier-Hein, K. H.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  _Nature methods_ 18, 203–211 (2021).
  * [57] Littlestone, N.  Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm.  _Machine learning_ 2, 285–318 (1988).
  * [58] Chen, T. & Guestrin, C.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , 785–794 (2016).
  * [59] Pedregosa, F. _et al._ Scikit-learn: Machine learning in Python.  _the Journal of machine Learning research_ 12, 2825–2830 (2011).
  * [60] Vovk, U., Pernus, F. & Likar, B.  A review of methods for correction of intensity inhomogeneity in mri.  _IEEE transactions on medical imaging_ 26, 405–421 (2007).

## Acknowledgments

TS and MBC acknowledge access to the facilities and expertise of the CIBM
Center for Biomedical Imaging, a Swiss research center of excellence founded
and supported by CHUV, UNIL, EPFL, UNIGE and HUG.

### Funding

  * Era-net Neuron MULTIFACT – Swiss National Science Foundation grant 31NE30_203977 (TS)

  * ERA-net NEURON MULTIFACT – French National Research Agency, Grant ANR-21-NEU2-0005

  * Swiss National Science Foundation grant SNSF #185872 (OE)

  * National Institute of Mental Health grant RF1MH12186 (OE)

  * Chan Zuckerberg Initiative grant EOSS5/‘NiPreps’ (OE)

  * SICPA foundation (YG)

  * Instituto de Salud Carlos III (ISCIII) grant AC21_2/00016 (EE)

  * French National Research Agency, SulcalGRIDS Project, Grant ANR-19-CE45-0014

### Author contributions

  * Conceptualization: MBC, TS, OE

  * Data: EE, AJ, NG, MK, VD, GA

  * Annotations: YG, MBC

  * Methodology: TS, MBC, OE

  * Software: TS

  * Evaluation: TS, AP

  * Supervision: MBC

  * Writing—original draft: TS, MBC, OE, GA

  * Writing—review & editing: All authors

### Competing interests

### Data and materials availability

Raw fetal brain MRI cannot readily be shared because of patient privacy.
Derived measures, such as extracted IQMs will be shared along with the codes
on Zenodo (under CC BY 4.0 license) and GitHub (under an Apache 2.0 license)
respectively, for the results to be reproduced.

## Supplementary material

### Quality rating protocol

In this section, we describe and illustrate the protocol that we used for
quality rating in the study. The quality rating is defined in the context of
downstream super-resolution reconstruction (SRR) from multiple low-resolution
acquisitions in different orientations (axial, coronal, sagittal). The quality
assessment specifically aims at quantify how suitable a given raw T2 weighted
volume is for SRR, and is not a radiological assessment of the image.

Rating protocol Using a FetMRQC report, our quality rating was consisted of
going through the following questions.

  * •

Motion-related artifacts. Fetal motion can induce in-plane and through-plane
artifacts.

    * –

Is there in-plane motion: signal drop/void, blurring, aliasing, ringing
artifacts?

    * –

Is there through-plane motion: loss of structural continuity in neighbouring
slices[12, 22]?.

  * •

Bias related artifacts. Bias field is typically described as a smoothly
varying spatial inhomogeneity that alters image intensities that otherwise
would be constant for the same tissue type regardless of its position in the
image [60]. In practice, it often appears like a shade on a part of an image,
and is visible both in-plane and through-plane.

    * –

Is there in-plane bias: differences in intensity within a single tissue on a
given slice?

    * –

Is there through-plane bias: difference in intensity within a single tissue
across slices?

  * •

Miscellanous.

    * –

Is the image paritcularly noisy: grainy appearance?

    * –

Is the brain entirely contained on the image?

After answering the questions, the rater is asked to provide a score for
global quality. Examples below show the scores that we assigned to various
images.

#### Example cases

We now review six cases from our data, featuring different gestational ages
and orientations with explanation of the artifacts in the captions. This
review does not aim to be exhaustive. Figure 7 shows two cases of excellent
quality acquisitions. The other images show typical artifact patterns that can
be found in fetal images. Figure 8 shows acceptable and poor-to-acceptable
quality images. Figure 9 shows poor quality images.

Figure 7: Excellent quality images. (Left) Acceptable to excellent quality
sagittal image. No clear artifacts are visible in the in-plane view, and no
large patterns of motion are view are present on the through-plane views.
Moreover, the structure of the brain is clearly distinguished on all three
planes. (Right) Excellent quality axial image. No artifacts are visible on any
part of the image. The intensity of the tissues is spatially homogeneous,
indicating a low bias, and no in-plane or through-plane affecting structural
integrity are visible.

Figure 8: Acceptable quality images. (Left) Acceptable quality sagittal image.
Although some ringing artifacts are visible outside the brain on slices 6 and
7, all brain structures are clearly visible in-plane. A strong bias field can
be viewed between the top and bottom row of in-plane slices, as well as
through-plane view. There is also moderate motion, viewed in the through-plane
view 268 where one sees various blocks of slices look disconnected from each
other. (Right) Poor to Acceptable quality coronal image. On both in-plane and
through-plane images, a clear intensity discontinuity is visible, suggesting a
strong bias bield. In addition, one sees on slice 14 a signal drop. No stair-
like motion is visible, but a sharp discontinuity is seen on slice 160. This
was rated as moderate motion.

Figure 9: Poor quality images. (Left) Poor-to-Exclude quality sagittal image.
Multiple slices are affected by signal drops suggesting heavy motion.
Unsurprisingly, this leads to a poor structural integrity on the through-plane
slices: it is very difficult to recognize the brain structure. (Right) Low
Poor quality coronal image. A strong bias field is visible on all in-plane
slices (the bottom of each slice is much darker than the top for the same
tissue), and also through-plane: on 279, the left is darker than the right,
and on 333, the left is clearer than the right. In addition, a typical
"staircase" motion is viewed through-plane, as well as a signal drop on slice
6.

### FetMRQC ablation studies

Various components such as IQM standardization, feature selection,
dimensionality reduction can be included in FetMRQC. We used nested cross-
validation to automatically perform model selection and evaluation without
introducing optimistic biases[43]. We then had three variants of FetMRQC to
compare:

  * •

Vanilla FetMRQC: No preprocessing, use all IQMs and fit a random forest.

  * •

Nested CV FetMRQC: Preprocessing, feature selection and various possible
models, with selection performed using nested CV. The parameters are the one
of Table 4 below.

  * •

ComBat + Nested CV FetMRQC: ComBat[47], preprocessing, feature selection and
various possible models, with selection performed using nested CV. The
parameters are the one of Table 4.

We evaluated each of these methods in a leave-one-scanner-out (nested) CV and
report the results on Table 5. The differences between Vanilla FetMRQC and the
variants were tested with Welch’s t-test (to take into account the unequal
variance across samples), but none of the differences were found to be
statistically significant. The breakdown of the results is shown on Figure 10,
where we see that no method manages to provide an improvement for all
scanners. While some scanners get a better performance, the performance is
also decreases for other scanners. This is most clearly seen for Combat +
Nested CV in the QA task (Figure 10B).

The full nested cross-validation very largely increases the computational time
required to train the model. Given the IQMs, vanilla FetMRQC takes around 5 to
10 seconds to be trained. Nested CV evaluates 1004 models (regression) and
1344 models (classification), and parts like the Winnow algorithm make the
overall training slower. Our simple implementation, using 5 parallel workers,
took around one day to run. While this could certainly be greatly improved, it
is clear that nested CV brings a much larger computational burden compared to
vanilla FetMRQC. In this cas, as it did not bring any significant benefit, we
chose to only rely on the vanilla version of FetMRQC: using all IQMs without
scaling and with a random forest.

Table 4: Parameters automatically optimized by the inner loop of the nested
CV.

Model step | Parameters
---|---
Remove correlated features |  $\text{Threshold}\in\\{0.8,0.9\\}$; Disabled
Data Scaling |  Standard (group) scaling, Robust (group) scaling, Quantile (group) scaling No scaling
Winnow algorithm | Enabled, Disabled
PCA | Enabled, Disabled
Regression models |  Linear regression, Gradient boosting, Random Forest
Classification models |  Logistic regression, Random Forest, Gradient Boosting, AdaBoost

Table 5: Quality control and assessment ablation study. LoSo CV was performed
using vanilla FetMRQC, as well as nested cross-validation for hyperparameter
tuning. A third variant preprocessed the data using ComBat[47] prior to
performing nested CV. Results are the median cross-validation performance. The
number in parentheses is the average worst-performing cross-validation fold.

Quality control (classification)
---
| Weighted F1 ($\uparrow$) | ROC AUC ($\uparrow$) | Precision ($\uparrow$) | Recall ($\uparrow$)
Leave-one-Scanner-out cross-validation
Nested CV | 0.81 (0.68) | 0.79 (0.64) | 0.85 (0.73) | 0.83 (0.79)
ComBat + Nested CV | 0.80 (0.73) | 0.79 (0.71) | 0.86 (0.71) | 0.89 (0.79)
Vanilla | 0.81 (0.62) | 0.78 (0.58) | 0.89 (0.69) | 0.83 (0.79)

Quality assessment (regression)
---
|  $R^{2}$ ($\uparrow$) | Spearman ($\uparrow$) | MAE ($\downarrow$)
Leave-one-Scanner-out cross-validation
Nested CV | 0.50 (0.36) | 0.73 (0.70) | 0.55 (0.54)
ComBat + Nested CV | 0.49 (0.18) | 0.75 (0.68) | 0.57 (0.60)
Vanilla | 0.45 (0.39) | 0.74 (0.71) | 0.56 (0.51)

Figure 10: Scanner-wise results for the QA/QC ablation study.. This is the
breakdown of Table 5. A – Weighted F1 score for the QC task, for each scanner
used in LoSo CV. B – $\text{R}^{2}$ for the QA task for each scanner used in
LoSo CV.