∎

11institutetext: P. Handa 22institutetext: Dept. of ECE, DTU, Delhi
22email<EMAIL_ADDRESS>33institutetext: M. Mathur 44institutetext:
Dept. of ECE, IGDTUW, Delhi
44email<EMAIL_ADDRESS>55institutetext: N. Goel 66institutetext:
Dept. of ECE, IGDTUW, Delhi
66email<EMAIL_ADDRESS>
# Open and free EEG datasets for epilepsy diagnosis

Palak Handa  Monika Mathur  Nidhi Goel

(Received: date / Accepted: date)

###### Abstract

The Epilepsies are a common, chronic neurological disorder affecting more than
50 million individuals across the globe. It is characterized by unprovoked,
recurring (similar or different type) seizures which are commonly diagnosed
through clinical EEGs. Good-quality, open-access and free EEG data can act as
a catalyst for on-going state-of-the-art (SOTA) research works for detection,
prediction and management of epilepsy and seizures. They can also aid in
improving the quality of life (QOL) of these diseased individuals and
contribute research in healthcare multimedia, data analytics and Artificial
Intelligence (AI) in personalized medicine. This paper presents widely used,
available, open and free EEG datasets available for epilepsy and seizure
diagnosis. A brief comparison and discussion of open and private datasets has
also been done. Such datasets will help in development and evaluation of
automatic computer-aided system in healthcare.

###### Keywords:

Open datasets Biomedical multimedia EEG signals Epilepsy diagnosis Seizure
Performance

##  1 Introduction

The Epilepsies are a chronic neurological disorder characterized by
unprovoked, recurring (similar or different type) seizures. A clinical EEG
setting is used by doctors to observe different types of epileptic activity as
it leaves distinct impressions in the form of interictal epileptiform
discharges, peri-ictal activities and high frequency oscillations _etc_ Thomas
et al. [2001].

The annual economic burden of epilepsy is enormous in developing countries
like India where it is estimated to be 88.2% of gross national product (GNP)
per capita and 0.5% of the overall GNP Thomas et al. [2001]. Hence, early
diagnosis through recent technologies like Artificial Intelligence (AI),
feature engineering, data analytics and multimedia is vital and can aid in
Quality Of Life (QOF) of patients and their associated caretakers.

Secured, reproducible AI algorithm, good quality data and efficient computing
horse power are the major elements for development of early detection and
prediction of epileptic wave forms through EEG signals. There are several
types of EEGs such as intracranial, scalp, ambulatory, _etc._ They are
recorded in video, image and signal format depending on their use and
application in hospitals.

There is a huge demand of real-time biomedical multimedia tools for data
analysis, and pattern recognition of such formats. Bonn EEG time series
database Andrzejak et al. [2001] was the first EEG dataset to be publically
available for research applications in this field. It remains as the benchmark
dataset for most research works due to its availability. Several datasets
discussed in this paper encourage scientific advances in this field.

The data quality of biomedical datasets is measured through various factors
such as presence of artefacts and noise, missing values, descriptive
information, annotations by health experts, pre-fined data structure,
processing and robustness to outliers _etc._

The datasets mentioned in the section 2 are freely available (except temple
university which requires a login), re-distributable for research purposes,
previously freely available and or became private or removed from the portal
based on adult and paediatric population. Table 1 shows a comparison of
publically available, open-access (except temple university which requires a
login) human EEG datasets for epilepsy diagnosis.

The source and availability of these were verified on 26-07-2021, which may
change in the future. They were found using different keywords like ‘EEG
datasets for epilepsy’, ‘datasets for epilepsy detection’, ‘EEG based epilepsy
diagnosis’, and ‘open EEG datasets’ on Pub med and google scholar search
engine.

##  2 EEG datasets for epilepsy diagnosis

There are several EEG datasets for epilepsy diagnosis which are freely
available and private due to various reasons such as lack of ethical
clearance. This section lists all the existing EEG datasets with their URLs
for adult and paediatric population.

###  2.1 Adult datasets

The adult population consists of affected individuals above 20 years of age.
There are several databases like American Epilepsy Society Seizure Prediction
Challenge database Howbert et al. [2014], dataset of EEG recordings of
pediatric patients with epilepsy based on the 10-20 system et al. [2021] and
Karunya University Selvaraj et al. [2014] which contain both adult and
paediatric EEG data. They are mentioned in section 2.2.

####  2.1.1 Bonn EEG time series databaseAndrzejak et al. [2001]

This database comprises of 100 single channels EEG of 23.6 seconds with
sampling rate of 173.61 Hz. Its spectral bandwidth range is between 0.5 Hz and
85 Hz. It was taken from a 128 channel acquisition system. Five patients EEG
sets were cut out from a multi-channel EEG recording and named A, B, C, D and
E. Set A and B are the surface EEG recorded during eyes closed and open
situation of healthy patients respectively. Set C and D are the intracranial
EEG recorded during a seizure free from within seizure generating area and
from outside seizure generating area of epileptic patients respectively. Set E
is the intracranial EEG of an epileptic patient during epileptic seizures.
Each set contains 100 text files wherein each text file has 4097 samples of 1
EEG time series in ASCII code. A band pass filter with cut off frequency as
0.53 Hz and 40 Hz has been applied on the data. It is an artifact free data
and hence no prior pre-processing is required for the classification of
healthy (non-epileptic) and un-healthy (epileptic) signals. The strong eye
movement’s artefacts were omitted. It was made available in 2001. The extended
version of this data is now a part of EPILEPSIA project. Available link: a and
b

####  2.1.2 Bern-Barcelona EEG database Andrzejak et al. [2012]

This multi channel EEG database was recorded using specialized electrodes and
consists of five patients with longstanding pharmacoresistant temporal lobe
epilepsy. The patients underwent epilepsy surgery. The sampling rate was
either 512 or 1024 Hz based upon whether they were recorded with less or more
than 64 channels of EEG system. Three out of five attained complete seizure
freedom. Two types of EEG are present in the data _i.e.,_ focal and non-focal.
Each file has about 10240 samples for a time duration of 20 seconds. Available
link: c and d

####  2.1.3 Temple University EEG corpus Obeid and Picone [2016]

Temple University EEG corpus is the largest free EEG data available for
epilepsy and seizure types diagnosis till date. It consists of data acquired
from 2000 to 2013 using different EEG clinical settings for about 10,874
patients. This community has developed various software products such as
annotation tools, toolboxes for seizure detection, and EDF browser for data
analysis of EEG, EMG, and ECG _etc_ signals. EDF browser helps to view EEG
recording in a video form. There are various datasets available such as IBM
Features For Seizure Detection (IBMFT), the TUH EEG epilepsy corpus, seizure
corpus, slowing corpus, and events corpus _etc_. A user ID and password is
required to get access to these datasets. Available link.

####  2.1.4 Neurology and sleep center, New Delhi EEG dataset Swami et al.
[2016]

This database comprises of 5.12 seconds EEG data. It was recorded using 57 EEG
channel Grass Tele-factor Comet AS40 Amplification System; sampled at 200 Hz.
It’s spectral bandwidth range is between 0.5 Hz and 70 Hz. Time series EEG
datasets are categorized into three major MATLAB file folder namely ictal,
pre-ictal and inter-ictal stages. Each MAT file has 1024 samples. A subset of
this database is publically available. Available link.

####  2.1.5 Epileptic Seizure Recognition Data Set Andrzejak et al. [2001]

The time series EEG dataset consists of 11500 instances of EEGs of 4 subjects
suffering from epilepsy. This data has been removed from the UCI machine
learning repository recently and was released in 2017. It is a simplified
version of the original data released by Andrzejak et al. [2001]. It consists
of 5 subjects (4 unhealthy and 1 healthy) performing different activities and
experience epileptic seizures except subject 1. The time duration for each EEG
was 23.5 seconds.

####  2.1.6 Siena Scalp EEG Database Detti et al. [2020], Detti [2020]

This multi-channel EEG database of 14 epileptic patients (9 males and 5
females) was recorded using specialized amplifiers, and reusable electrodes.
The signals were recorded with a sampling rate of 512 Hz and stored in EDF
files. The data has been acquired from Unit of Neurology and Neurophysiology
of the University of Siena, Italy and focuses on seizure prediction. It is an
integral part of national interdisciplinary research project PANACEE. This
data contains 47 seizures from 128 hours of video EEG recording. The start and
end time of a seizure was also recorded and contains the list of electrodes
present on the scalp of a patient during event recording. Three types of
seizures namely focal onset with and without impaired awareness, and focal to
bilateral tonic–clonic (FBTC) were found and recorded in the diseased
patients. Available link.

####  2.1.7 Single electrode EEG data of healthy and epileptic patients Panwar
et al. [2019], Panwar [2020]

This dataset was generated with a motive to build predictive epilepsy
diagnosis model and publically available since 2020. It was generated on a
similar acquisition and settings _i.e.,_ sampling frequency, bandpass
filtering and number of signals and time duration as of University of Bonn. It
has overcome the limitations faced by University of Bonn dataset such as
different EEG recording (inter-cranial and scalp) for healthy and epileptic
patients Panwar et al. [2019]. All the data were taken exclusively using
surface EEG electrodes for 15 healthy and epileptic patients. Available link.

####  2.1.8 Epileptic EEG Dataset Nasreddine [2021]

This multi-channel, long term EEG database was recorded for 6 patients
suffering from focal epilepsy. They were undergoing pre-surgical evaluation
for possible epilepsy surgery. Different EEG segments of a seizure like ictal,
pre-ictal, inter-ictal and its onsets have been included in the data. The
signals were recorded with a sampling rate of 500 Hz and stored in EDF files.
Labelled and classified data points (train and test set) have been mentioned
for complex partial electrographic, and video-detected seizures. All the EEG
signals underwent band pass filtering of range 1-70 Hz where 50 Hz (utility
frequency) was also removed. Available link.

###  2.2 Paediatric datasets

The paediatric EEG database consists of affected individuals from age 1 month
- 20 years.

####  2.2.1 Children’s hospital Boston–MIT database Goldberger et al. [2000]

This database comprises of 844 hour continuous EEG. 23 pediatric patients from
age 1.5-19 who underwent scalp multi-channel EEG recording. It is the first
paediatric EEG database available for epilepsy and seizure diagnosis. The
patients were given anti-seizure medications. About 200 seizures were recorded
in a universal bio-polar montage with about 24-27 EEG channels. Sampling
frequency was kept to be 256 Hz. Each EEG segment is called as a record which
usually is for duration of one hour. There are 9-42 edf files from a single
subject. Additional vagal nerve stimulus signals are also present. Separate
file name and montages have been mentioned for seizure v/s non seizure episode
in EEG segments. Available link.

####  2.2.2 Karunya University Selvaraj et al. [2014]

This database comprises of 18 channel EEG data with segments of normal, focal
and generalized epileptic seizure activities from 1–107 years of patients. It
was released in 2014 but the website is not available for research use now.
Each segment has 2056 sample points. Sampling frequency was kept to be 256 Hz.
The EEG recordings vary from 40 minutes to one hour. It was collected from a
diagnostic center based in Coimbatore, India. Available link.

####  2.2.3 A dataset of neonatal EEG recordings with seizures annotations
Stevenson et al. [2018]

This database consists of multi-channel, good quality EEG recordings of 79
term neonates where 39 of them suffered from neonatal seizures in the NICU of
Helsinki University Hospital, Finland. The recordings were captured with
NicOne EEG amplifier, and 19 EEG channel cap. The signals were recorded with a
sampling rate of 256 Hz and stored in EDF files. It consists of seizure
annotations by healthcare experts for seizure detection purpose. The data was
pre-processed using butterworth high-pass filtering. The data also contains
natural artefacts. Available link.

Table 1: Comparison of existing EEG datasets for epilepsy diagnosis

Ref. Availability Type Source Year Size No. of No. of Sampling EEG segments
channels patients frequency Andrzejak et al. [2001] Freely available Adult
e-repositori upf. 2001 3.05 MB 100 single 5 173.61 Hz seizure states, healthy
Goldberger et al. [2000] Freely available Paediatric PhysioNet repository 2010
40 GB 23-26 22 256 Hz Intractable seizures Andrzejak et al. [2012] Freely
available Adult e-repositori upf. 2012 814 MB 64 5 512 Hz Focal, Non-focal
Howbert et al. [2014] Freely available Dog and human Kaggle 2014 105 GB - - -
different types Selvaraj et al. [2014] Not available Adult and Paediatric
Website 2014 - - - 256 Hz normal, focal and generalized epileptic seizures
Obeid and Picone [2016] Free but Adult Website 2015 572 GB 20-31 10,874 250,
different types requires login 256, 512 Hz Swami et al. [2016] Freely
available Adult Researchgate 2016 604 KB 57 10 200 Hz Ictal, inter-ictal, pre-
ictal EEGs Andrzejak et al. [2001] Removed Adult UCI repository 2017 3 MB 100
single 5 173.61 Hz seizure related, healthy Stevenson et al. [2018] Freely
available Paediatric (neonates) Zenedo 2018 4.3 GB 19 79 256 Hz seizure onset
Raghu et al. [2019] Private Adult - 2019 - 19 115 128 Hz epileptic and healthy
Panwar et al. [2019] Private Adult - 2019 - - 50 250, 256 Hz generalized and
focal epilepsies Wu et al. [2019] Private Adult - 2019 - 21 5 500 Hz focal and
tonic-clonic Avcu et al. [2019] Private Pediatric - 2019 - - 29 200, 500 Hz
typical absence seizures Yan et al. [2019] Private Adult - 2019 - - 12 256 Hz
seizure events Choi et al. [2019] Private - - 2019 - 21 25 200 Hz seizure
events Cao et al. [2019] Private - - 2019 - 18 10 256 Hz seizure states Bilal
et al. [2019] Private - - 2019 - 22 22 250 Hz ictal, non-ictal Panwar [2020]
Freely available Adult Zenedo 2020 20 MB - 15 173.61 Hz Inter-ictal Yedurkar
and Metkar [2020] Private - - 2020 - 21 - 250 Hz seizure onsets Das et al.
[2020] Private Adult - 2020 - 21 150 256 Hz seizure and normal Detti [2020]
Freely available Adult PhysioNet repository 2020 20 GB 29 14 512 Hz Epileptic
seizures (focal onset, tonic-clonic) Nasreddine [2021] Freely available Adult
Mendeley repository 2021 3133 MB 21 6 500 Hz Complex partial, electrographic
and video-detected seizures et al. [2021] Freely available Paediatric and
Adult Open neuro repository 2021 15 GB 52 30 2000 Hz HFO markings

####  2.2.4 Dataset of EEG recordings of pediatric patients with epilepsy
based on the 10-20 system et al. [2021]

This dataset consists of scalp EEG recordings to study the impact age on
observed High Frequency Oscillations (HFO) in pediatric epileptic patients.
Three hours of pediatric and adult EEG sleep data was recorded for 30 focal or
generalised epileptic patients. The signals were recorded with a sampling rate
of 2000 Hz and stored in EDF files. Different sleep stage annotations are
available in this database. Available link.

###  2.3 Others

####  2.3.1 American Epilepsy Society Seizure Prediction Challenge database
Howbert et al. [2014]

This database consists of intra-cranial EEG segments from dogs and humans with
different acquisitions of sampling rate, duration of EEGs, and no. of
electrodes _etc._ It was released as a part of the kaggle challenge hosted by
the American Epilepsy Society in 2014 for development of seizure forecasting
systems and witnessed about 504 teams. Different seizure segments of ictal,
pre-ictal, post-ictal, inter-ictal were provided in MATLAB files. The data
storage was about 105 GB. Additional annotated EEG data was also provided by
the University of Pennsylvania and the Mayo Clinic. Available link.

####  2.3.2 Private databases

Several private databases have also been recorded for epilepsy diagnosis using
EEG signals Panwar et al. [2019], Das et al. [2020], Raghu et al. [2019], Wu
et al. [2019], Avcu et al. [2019], Bilal et al. [2019], Yedurkar and Metkar
[2020], Yan et al. [2019], Choi et al. [2019], Cao et al. [2019]. The European
Epilepsy database Ihle et al. [2012] is a private database which consists of
high quality, annotated EEG signals from University of Bonn Andrzejak et al.
[2001], Freiburg Schelter et al. [2006], Flint hills, and many multi-modal
like MRI imaging data. The website of Selvaraj et al. [2014] is not available.
EEG data in Zwoliński et al. [2010] was freely available till 2015 when its
portal crashed.

##  3 Conclusion

Diagnosis, treatment and management of epilepsy is still a challenging task
for the scientific and healthcare community. It’s detection by visual
introspection of long hour EEG is not only time taking but a very tedious and
subjective task. Artificial intelligence can help in escalating this process
and lead to successful detection of different types of epilepsies through
efficient, high quality and annotated EEG data. This paper has presented all
the existing EEG datasets for epilepsy diagnosis with its availability and
brief comparisons. Such datasets motivate scientific research in early
diagnosis of epilepsy through robust techniques.

## Conflict of interest

The authors declare that they have no conflict of interest.

## References

  * Thomas et al. [2001] SV Thomas, PS Sarma, M Alexander, L Pandit, L Shekhar, C Trivedi, and B Vengamma.  Economic burden of epilepsy in india.  _Epilepsia_ , 42(8):1052–1060, 2001.
  * Andrzejak et al. [2001] Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian E Elger.  Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state.  _Physical Review E_ , 64(6):061907, 2001.
  * Howbert et al. [2014] J Jeffry Howbert, Edward E Patterson, S Matt Stead, Ben Brinkmann, Vincent Vasoli, Daniel Crepeau, Charles H Vite, Beverly Sturges, Vanessa Ruedebusch, Jaideep Mavoori, et al.  Forecasting seizures in dogs with naturally occurring epilepsy.  _PloS one_ , 9(1):e81920, 2014.
  * et al. [2021] Dorottya Cserpan et al.  Dataset of eeg recordings of pediatric patients with epilepsy based on the 10-20 system, 2021.
  * Selvaraj et al. [2014] Thomas George Selvaraj, Balakrishnan Ramasamy, Stanly Johnson Jeyaraj, and Easter Selvan Suviseshamuthu.  Eeg database of seizure disorders for experts and application developers.  _Clinical EEG and neuroscience_ , 45(4):304–309, 2014.
  * Andrzejak et al. [2012] Ralph G Andrzejak, Kaspar Schindler, and Christian Rummel.  Nonrandomness, nonlinear dependence, and nonstationarity of electroencephalographic recordings from epilepsy patients.  _Physical Review E_ , 86(4):046206, 2012.
  * Obeid and Picone [2016] Iyad Obeid and Joseph Picone.  The temple university hospital eeg data corpus.  _Frontiers in neuroscience_ , 10:196, 2016.
  * Swami et al. [2016] P Swami, B Panigrahi, S Nara, M Bhatia, and T Gandhi.  Eeg epilepsy datasets, 2016.
  * Detti et al. [2020] Paolo Detti, Giampaolo Vatti, and Garazi Zabalo Manrique de Lara.  Eeg synchronization analysis for seizure prediction: A study on data of noninvasive recordings.  _Processes_ , 8(7):846, 2020.
  * Detti [2020] P. Detti.  Siena scalp eeg database (version 1.0.0), 2020.  URL https://doi.org/10.13026/5d4a-j060.
  * Panwar et al. [2019] Siddharth Panwar, Shiv Dutt Joshi, Anubha Gupta, and Puneet Agarwal.  Automated epilepsy diagnosis using eeg with test set evaluation.  _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , 27(6):1106–1116, 2019.
  * Panwar [2020] Siddharth Panwar.  Single electrode EEG data of healthy and epileptic patients, February 2020.  URL https://doi.org/10.5281/zenodo.3684992.
  * Nasreddine [2021] Wassim Nasreddine.  Epileptic eeg dataset, 2021.  URL https://doi.org/10.17632/5pc2j46cbc.1.
  * Goldberger et al. [2000] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley.  Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals.  _circulation_ , 101(23):e215–e220, 2000.
  * Stevenson et al. [2018] Nathan Stevenson, Karoliina Tapani, Leena Lauronen, and Sampsa Vanhatalo.  A dataset of neonatal EEG recordings with seizures annotations, June 2018.  URL https://doi.org/10.5281/zenodo.2547147.
  * Raghu et al. [2019] Shivarudhrappa Raghu, Natarajan Sriraam, Yasin Temel, Shyam Vasudeva Rao, Alangar Satyaranjandas Hegde, and Pieter L Kubben.  Performance evaluation of dwt based sigmoid entropy in time and frequency domains for automated detection of epileptic seizures using svm classifier.  _Computers in biology and medicine_ , 110:127–143, 2019\.
  * Wu et al. [2019] Duanpo Wu, Zimeng Wang, Lurong Jiang, Fang Dong, Xunyi Wu, Shuang Wang, and Yao Ding.  Automatic epileptic seizures joint detection algorithm based on improved multi-domain feature of ceeg and spike feature of aeeg.  _IEEE Access_ , 7:41551–41564, 2019.
  * Avcu et al. [2019] Mustafa Talha Avcu, Zhuo Zhang, and Derrick Wei Shih Chan.  Seizure detection using least eeg channels by deep convolutional neural network.  In _ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 1120–1124. IEEE, 2019.
  * Yan et al. [2019] Peter Z Yan, Fei Wang, Nathaniel Kwok, Baxter B Allen, Sotirios Keros, and Zachary Grinspan.  Automated spectrographic seizure detection using convolutional neural networks.  _Seizure_ , 71:124–131, 2019.
  * Choi et al. [2019] Gwangho Choi, Chulkyun Park, Junkyung Kim, Kyoungin Cho, Tae-Joon Kim, HwangSik Bae, Kyeongyuk Min, Ki-Young Jung, and Jongwha Chong.  A novel multi-scale 3d cnn with deep neural network for epileptic seizure detection.  In _2019 IEEE International Conference on Consumer Electronics (ICCE)_ , pages 1–2. IEEE, 2019.
  * Cao et al. [2019] Jiuwen Cao, Jiahua Zhu, Wenbin Hu, and Anton Kummert.  Epileptic signal classification with deep eeg features by stacked cnns.  _IEEE Transactions on Cognitive and Developmental Systems_ , 12(4):709–722, 2019.
  * Bilal et al. [2019] Muhammad Bilal, Muhammad Rizwan, Sajid Saleem, Muhammad Murtaza Khan, Mohammed Saeed Alkatheir, and Mohammed Alqarni.  Automatic seizure detection using multi-resolution dynamic mode decomposition.  _IEEE Access_ , 7:61180–61194, 2019.
  * Yedurkar and Metkar [2020] Dhanalekshmi P Yedurkar and Shilpa P Metkar.  Multiresolution approach for artifacts removal and localization of seizure onset zone in epileptic eeg signal.  _Biomedical Signal Processing and Control_ , 57:101794, 2020\.
  * Das et al. [2020] Khakon Das, Debashis Daschakladar, Partha Pratim Roy, Atri Chatterjee, and Shankar Prasad Saha.  Epileptic seizure prediction by the detection of seizure waveform from the pre-ictal phase of eeg signal.  _Biomedical Signal Processing and Control_ , 57:101720, 2020\.
  * Ihle et al. [2012] Matthias Ihle, Hinnerk Feldwisch-Drentrup, César A Teixeira, Adrien Witon, Björn Schelter, Jens Timmer, and Andreas Schulze-Bonhage.  Epilepsiae–a european epilepsy database.  _Computer methods and programs in biomedicine_ , 106(3):127–138, 2012.
  * Schelter et al. [2006] Björn Schelter, Matthias Winterhalder, Thomas Maiwald, Armin Brandt, Ariane Schad, Jens Timmer, and Andreas Schulze-Bonhage.  Do false predictions of seizures depend on the state of vigilance? a report from two seizure-prediction methods and proposed remedies.  _Epilepsia_ , 47(12):2058–2070, 2006.
  * Zwoliński et al. [2010] Piotr Zwoliński, Marcin Roszkowski, Jaroslaw Żygierewicz, Stefan Haufe, Guido Nolte, and Piotr J Durka.  Open database of epileptic eeg with mri and postoperational assessment of foci—a real world verification for the eeg inverse solutions.  _Neuroinformatics_ , 8(4):285–299, 2010.


# Ensemble CNN and Uncertainty Modeling to Improve Automatic
Identification/Segmentation of Multiple Sclerosis Lesions in Magnetic
Resonance Imaging

Giuseppe Placidi1 Luigi Cinque2 Daniela Iacoviello3 Filippo Mignosi4 Matteo
Polsinelli1

( 1 A2VI-lab, c/o Department of MeSVA, University of LAquila, Via Vetoio
Coppito 2, 67100 - LAquila, ITALY
<EMAIL_ADDRESS>
2 Department of Computer Science, Sapienza University of Rome, Via Salaria
113, 00198 - Roma, ITALY
3 Department of Computer, Control and Management Engineering Sapienza
University of Rome, Rome, Italy 4 Department of DISIM, University of LAquila,
Via Vetoio Coppito, 67100 - LAquila, ITALY
)

###### Abstract

To date, several automated strategies for identification/segmentation of
Multiple Sclerosis (MS) lesions with the use of Magnetic Resonance Imaging
(MRI) have been presented, but they are outperformed by human experts, from
whom they act very differently. This is mainly due to: the ambiguity
originated by MRI instabilities; peculiar variability of MS; non specificity
of MRI regarding MS. Physicians partially manage the uncertainty generated by
ambiguity relying on radiological/clinical/anatomical background and
experience. To emulate human diagnosis, we present an automated framework for
identification/segmentation of MS lesions from MRI based on three pivotal
concepts: 1. the modelling of uncertainty; 2. the proposal of two, separately
trained, CNN, one optimized for lesions and the other for lesions with respect
to the environment surrounding them, respectively repeated for axial, coronal
and sagittal directions; 3. the definition of an ensemble classifier to merge
the information collected by different CNN.

The proposed framework is trained, validated and tested on the 2016 MSSEG
benchmark public data set from a single imaging modality, the FLuid-Attenuated
Inversion Recovery (FLAIR). The comparison with the ground-truth and with each
of 7 human raters, proves that there is no significant difference between the
automated and the human raters.

##  1 Introduction

Multiple Sclerosis (MS) is a degenerative disease mainly affecting white
matter (WM) and spinal cord, with a very heterogeneous clinical presentation
across patients both in severity and symptoms [46].

The origins of the disease are not well understood but characteristic signs of
tissue degeneration are recognizable by the presence of lesions and brain
atrophy. Most of MS signs can be observed through Magnetic Resonance Imaging
(MRI) which has become the elective mini-invasive tool for MS [16]. Focal
lesions are primarily visible in the white matter on structural MRI,
observable as hyper-intensities on T2-weighted images (T2w), on proton-density
images (PD), or on FLuid-Attenuated Inversion Recovery images (FLAIR), and as
hypo-intensities on T1-weighted images (T1w).

Physicians often use FLAIR images for WM lesion detection and other modalities
mostly to ascertain the presence of cortical lesions.

An examination consists in thousands of images mostly collected pre and post
contrast administration. MRI is used routinely in clinical practice but it is
unspecific for MS and not well correlated to the impairment progression, to
the neuro-plasticity and to the effects of demyelinization of nerves, the last
being a critical effect which is invisible to MRI. In a MS patient, the
"healthy" brain tissue is usually referred as "apparently healthy" [16]. In
addition, often lesions and healthy tissue are present in the same place, thus
resulting in the partial volume effect (PVE). Regarding MRI, healthy
anatomical structures similar to lesions and close to lesions could contribute
to create further ambiguity. Moreover, there is a huge variability in images
due to differences in scanners, magnetic field strength/homogeneity and tuning
of parameters [37]. Efforts have been made to standardize MRI amplitude [10]
but results are still unsatisfactory due to the huge set of parameters to be
tuned. The consequence of the above phenomenons are ambiguities and
disagreement among radiologists (inter-raters variability) as well as
uncertainty in a same radiologist (intra-rater variability) mainly in defining
the borders of the lesions, but also in labeling whole regions. This could
make the manual segmentation, besides long and boring, also inaccurate,
especially when quantitative evaluations are required, despite the ability of
radiologists in relying on unexpressed anatomical/clinical background, on
knowledge about MS and its symptoms.

Several automatic frameworks have been recently proposed and reviewed [13, 11,
38, 60, 19, 3, 20]. Despite the huge efforts, to date the results are still
far from those of human experts. Actually, this has led to an increase in the
model complexity not corresponding to the expected improvement [34]. Automated
strategies are not robust to MRI variability, not even sufficiently able to
model medical knowledge, human operational capacity and flexibility.
Furthermore, the reasoning methodology used by radiologists for volume
analysis, consisting in 2D axial slices with a continuous view of coronal and
sagittal slices for confirmation [49, 16], is not fully imitated by automatic
strategies. A recent paper [33] highlights the importance of using 3D CNN in
MS lesion segmentation, though other recent frameworks [48, 56] prefer
ensembles of 2D U-net models, in a way which is similar to the human
methodology. Human methodology is empowered by the fact that spatial
resolution is often not the same along the three axes with a prevalence for
axial planes. However, the uncertainty affecting radiologists during
classification is not reported in the public data sets: a binary choice is
often insufficient to represent the evaluation of an expert. If represented,
the uncertainty could greatly help an automatic strategy to better segment
also undoubted lesions. This pushed to investigate on uncertainty in medical
data [4] and on the effect that the rater style could transfer in terms of
uncertainty to an automated strategy [53].

Implicit information in automated methods is difficult to be modelled and
more, if introduced with external supports (dictionary, anatomical atlases,
etc.) [31], is often insufficient to fill the gap with human experts.

We aim at filling this gap, both in performance and in reasoning, by proposing
a framework which includes: 1) the classification of uncertainty as an
intermediate class between the background and lesions; 2) the optimization of
two CNN (2D U-net models), one for the class lesion and one for the class
background to contextualize lesions with respect to the surrounding anatomical
structures for the three spatial directions (axial, coronal and sagittal); 3)
the definition of an ensemble classifier to merge the information collected by
all CNN.

To obtain our goal we: use the publicly available large-scale benchmark MRI
database and corresponding ground truth proposed in 2016 MICCAI MS Lesion
Segmentation Challenges, MSSEG [1]; define the uncertain regions with the help
of the binary classifications of 7 human raters in MSSEG; apply the framework
just on FLAIR images.

These choices allow to: compare the proposed framework with competitive
automated strategies and human experts; demonstrate that the uncertain
reasoning modelling could help in reducing the ambiguity and complexity of
the, intrinsically uncertain, problem; demonstrate that a single imaging
sequence, FLAIR, is sufficient for MS lesion identification/segmentation in
WM. It is implicit that the framework applies to WM lesions and not on
cortical lesions, being these last absent in MSSEG.

The manuscript is structured as follows: Section 2 presents a review of
automatic approaches to MS lesion identification/segmentation, in particular
those using CNN; Section 3 describes the proposed framework in the context of
the used data set, the defined three-class consensus used for training, the
proposed CNN architecture and the ensemble system; Section 4 details the
metrics used for comparison; Section 5 reports and discusses experimental
results; Section 6 concludes the paper and presents some constructive hints
for future improvements.

##  2 Related work

Medical image analysis is greatly performed with automated methods, mostly
involving deep learning [30, 36, 32, 55, 29]. Automated MS lesion
identification$/$segmentation is still an active field of research and several
methods have been provided in the last decade and well reviewed along time
[13, 11, 28, 60] with an emerging role of AI-based methods [2].

Automated strategies can be classified into three main groups: methods using
pre-selected features modelling (PSFM), methods using a-priori information
modelling (APIM) and methods using deep learning modelling (DLM).

PSFM calculate pre-selected features and learn from previously segmented
training images to separate lesions from healthy tissue [62]. Some PSFM use a
large set of features and select the more discriminant ones through labelled
training. Others use an atlas-based technique, employing topological and
statistical atlases for WM lesion segmentation [43, 54]; another includes the
usage of Decision Random Forests [18]. Similarly, a framework for segmentation
of contrast-agent enhanced lesions using conditional random fields is defined
in [26]. The work in [8] proposes a set of features, including contextual
features, registered atlas probability maps and an outlier map, to
automatically segment MS lesions through a voxel by voxel approach. A
rotation-invariant multi-contrast non-local means segmentation is proposed in
[21] for the identification and segmentation of lesions from 3D MRI images.
Supervised learning by PSFM has been widely employed in tasks where the
training database and the pre-selected feature set cover all possible cases
[9]. Nevertheless, when the heterogeneity of the disease and the potential
variability of imaging are large, as it occurs for MS and MRI, the dimension
of the training database and, mostly, the choice of the pre-selected features
are critical.

APIM does not require labelled training data to perform segmentation, but
usually exploit some a-priori information, such as the intensity
clustering[58], to model tissue distribution. In [6], a likelihood estimator
to model the distribution of intensities in healthy brain MR images is
presented. Other methods use threshold with post processing refinement [41] or
are based on probabilistic models [47]. A big challenge for APIM is that the
outliers are not specific for lesions because they could be due to artifacts,
intensity inhomogeneity and small anatomical structures like blood vessels:
this often produces false positives [7]. Moreover, APIM is strongly based on
the information extracted and simplified by the knowledge of specific experts.

During the last years DLM has gained popularity in medical imaging especially
with U-nets ad their variants [39, 63, 35, 23]. Though the dimension of the
training database is also crucial in DLM, this has no concern regarding the
pre-selection of features as in PSFM or regarding a-priori information
modelling as in APIM. In particular CNN, compared to machine learning
approaches, has achieved remarkable success in biomedical image analysis [7,
36, 55, 56]. DLM trains and learns to design features directly from data [5]
and provides best results in MS lesion identification/segmentation, as
recently confirmed in [13, 11, 28, 60, 25, 52, 59, 20, 3].

CNN applied to MS often use 2D spatial convolutional layers [5], others use 3D
convolutional layers to incorporate 3D spatial information simultaneously [50,
22, 40, 51] or merge spatial with temporal information [38]. All these methods
perform segmentation with a minimum lesion volume threshold to avoid the
inclusion of small false positives.

Some methods improve the MS segmentation by retraining the CNN from the first
layers, in a kind of self supervision [15], by hypothesizing that the those
layers are more sensitive.

Among other recent works, hybrid models [14] allow to detect MS lesions by
using textural features in combination with machine learning approaches, as
Decision Tree (DT) and Support Vector Machines (SVM).

However, CNN performance is still far from that obtained by human experts or
dramatically drops with other data sets [34].

##  3 The proposed framework

The framework we propose, sketched in Figure 1, consists of the following
steps: 1) deep learning automatic classification, of the images (2D) composing
the MRI model, in three classes: Background, Uncertainty and Lesion (capital
letter to imply the concept ’class’), optimized for Lesion (lesions from
inside) and for Lesion in the context of the surrounding environment,
separately for axial, coronal and sagittal directions (resulting in 6
classifiers); 2) class fusion (separately for Lesion and Uncertainty, starting
from Lesion) by performing the Union of the 2 axial segmentation (step 2a in
Figure 1), followed by a majority vote taken from the remaining segmentations
and used for confirmation of the class (if the class is not confirmed, this is
downgraded (step 2b in Figure 1); 3) final output.

For the framework we propose, the following three hypotheses hold:

1) the MSSEG pre-processed data from just one single MRI modality, FLAIR, are
the input of the framework;

2) the binary labelled ground-truth is revised to contain, besides Lesion and
Background, also Uncertainty which is created from part of the original
Background, leaving Lesion unchanged (see detailed description below);

3) the three considered classes are supposed to be ordered, Background $<$
Uncertainty $<$ Lesion: in our case, just Lesion and Uncertainty are the
subjects of fusion and, for this reason, their downgrading consists in the
passage from Lesion to Uncertainty and from Uncertainty to Background,
respectively: the process starts from Lesion to allow Uncertainty fusion on
the upgraded data set.

Step 2a of Figure 1 serves to include, besides common information, also
complementary ones coming from the specificity of each of the two axial CNN
and, at the same time, to model the reasoning of the radiologists that use
axial orientation to make the first hypotheses. Step 2b is used to vote for
each object resulting from the axial processing (Lesion or Uncertainty), its
permanence in the assigned class, or its downgrading. Objects are confirmed
when at least two of the other four raters (two coronal and two sagittal)
agree with the axial classification. This ensures that false positives are
greatly reduced, that 3D contextualization with the environment is maintained
and that the model agrees with the radiologist’s reasoning.

The choices regarding the usage of one single imaging modality, the
classification in three classes and the use of an ensemble framework are
clarified below.

Being supervised, each classifier needs training, validation and test carried
on by using data from a public data set. In what follows we first describe the
used data set, the ternary ground truth, the CNN architecture, the used loss
function, the hyper-parameters optimization and the ensemble, final,
classification of Figure 1.

Figure 1:  The proposed identification/segmentation pipeline which divides the
brain tissue in three classes: healthy tissue (Background), tissue that has
uncertain nature (Uncertainty), and MS lesions (Lesion). The strategy operates
independently on axial, coronal and sagittal images, each processed by two
separately trained U-nets, one optimized for Lesion, to directly focus on
lesions, and the other optimized for contextualizing lesions with respect to
the environment. It recombines the results by using the Union of axial volumes
followed by a majority vote strategy on the coronal and sagittal volumes, for
confirmation. Voxels whose classification is not confirmed are downgraded
(Lesion becomes Uncertainty and Uncertainty becomes Background). The framework
operates separately for Lesion and Uncertainty, starting from Lesion. In step
2b, the procedure is applied voxel by voxel: $L$ is referred to each single
voxel of the class $c\in\\{Lesion,Uncertainty\\}$.

###  3.1 MSSEG data set

To allow a direct comparison and benchmark of the proposed framework with the
state-of-the-art segmentation methodologies, we use the 2016 MSSEG data set
[1]. In MSSEG, data were collected with several MRI scanners using different
magnetic field strengths: 1.5T Siemens Aera, 3T Siemens Verio, 3T Philips
Ingenia and 3T General Electric Discovery. The data set contains, for each
examination, $T_{1}$-w, $T_{1}$-w gadolinium enhanced ($T_{1}$-w Gd),
$T_{2}$-w, $T_{2}$-FLAIR and $PD$-w images.

We have chosen MSSEG because it is a benchmark data set including 7 different
classifications made by 7 different expert human raters and a consensus, the
ground-truth, making it possible to compare our framework with human
classifications, as well as with state of the art automatic strategies. For
MSSEG, it has been asked the human raters to perform a binary segmentation
(each voxel was identified as lesion/not lesion): the annotated data set has
been composed by 15 training cases. Another subset of data, composed of 38
testing cases, has been left without annotation.

The images are anonymized and provided both in original form and pre-processed
for user convenience.

Pre-processing refers to a series of mathematical adjustments to MR images
before segmentation [45] for reducing the effects of noise and imaging
artifacts, equalizing space, eliminating outliers and stabilizing the
contrast.

Though the MSSEG data set contains images from 4 imaging modalities, in our
framework we opt for the usage of one single modality (FLAIR) to:
detect/segment MS lesions in WM; demonstrate that a single modality is
sufficient for MS lesion identification/segmentation in WM; avoid that the
potential gain of using multiple imaging modalities could be overcome by the
drop in robustness and performance due to residual MRI instabilities after
pre-processing.

Further, we divide the annotated MSSEG data set, composed by 15 subjects, in
three subsets respectively for training, validation and test, as follows:

  * •

the training data set contains examinations from 9 subjects, 3 subjects of
each centre;

  * •

the validation data set contains data from 3 subjects, one for each centre;

  * •

the test data set contains the remaining data, from 3 subjects, one for each
centre.

An exhaustive cross validation is performed while maintaining the proportions
between centres. Once the data set is established, it is augmented by adding,
for each image, 2 random rotations (between -13 and 13 degrees, 1 degree in
resolution), 1 random scaling (between 1.1 and 1.3, 0.01 in resolution) and 1
gaussian random noise addition, 0 mean and 0.001A variance, where A is the
maximum amplitude value in the examined volume.

The augmented data set, for each validation and orientation (axial, coronal
and sagittal), contains 5216 images for training, 418 images for validation
and 435 images for test.

The augmented data originated from the same dataset [1] used to train,
validate and test the state of the art methods. In this way, we can indirectly
compare our framework with the state of the art strategies by using, for them,
the performance evaluation reported in the original manuscripts.

###  3.2 Ternary ground-truth

In medical imaging it can be assumed that there is a single, unknown, true
segmentation map of the underlying anatomy, and each human rater produces an
approximation with variations reflecting individual experience.

In the opposite case, it can be assumed that the variable annotations from
experts are all realistic and acceptable instances of the true segmentation.

As it often occurs, the truth is in the middle: some ambiguities are indeed
due to human subjectivity or imperfections (extrinsic), while some others are
due to the problem itself (intrinsic). In our problem, both are important, but
intrinsic ambiguities have the highest role, being due both to MS presentation
and to MRI non specificity: lesions are not well separated from healthy tissue
in MS (due to PVE) and MRI is neither sufficiently specific for MS nor
sufficiently precise. Regarding human subjectivity, it produces differences
that are due to a mix of prior assumptions, like experience in the field,
greater or lesser exploitation of additional meta-information (such as
anatomical/radiological/clinical knowledge), mistakes or oversights which
often are concentrated on small and/or low intensity lesions and lesion
borders.

When raters are forced to provide a binary segmentation, as in MSSEG, they
cannot express any doubt, whatsoever is the cause. The binary segmentation
does not allow the representation of the intrinsic uncertainty and,
furthermore, induces a human rater to assume polarized decisions which, from
one side, could not correspond to what the rater really believes in and, from
the other side, could be confusing and misleading for an automated strategy.
In fact, ambiguous decisions might have been assumed by the rater in similar
situations (an uncertain region could be considered healthy tissue in one case
and lesion in another) which could influence the automated strategy [53].

If we want to train an automated method to recognize the problem-specific
uncertainty, we have to integrate the binary ground-truth with human
uncertainty (doubts). To maintain the original ground-truth, we just consider
as Uncertainty the voxels that at least three of the seven human raters
considered as lesions and the binary consensus had not. In this way, since we
create space for the Uncertainty from the Background, the Lesion class of the
binary consensus remains unchanged: this allows the comparison with state of
the art strategies, performed on Lesion. The use of Uncertainty in this
specific context is intended to reduce the uncertainty in identifying Lesion,
hence to improve classification. A more accurate definition of the
uncertainty, though interesting, is out the scope of the manuscript and
deserves specific exploration.

The method we propose is quite different from other strategies used to define
the uncertainty [27, 20], for the following motivations: 1) to maintain the
original structure of Lesion; 2) to account for the uncertainty affecting both
the problem and the raters; 3) to avoid the new class Uncertainty could
prevent a direct comparison with other methods; 4) to quantify the performance
gain the proposed framework could effectively get when the Uncertainty is
introduced; 5) to allow the learning strategy to consider as uncertain not
only lesion borders, as other Authors do [27], but also whole regions not
necessarily connected to lesions. In fact, Uncertainty could regard both the
lesion borders, where damaged tissues could coexist with healthy tissues
(PVE), and whole structures, where doubts are due to MRI unspecific nature for
MS.

Fig.2 reports a FLAIR image example with the corresponding binary consensus
and the proposed ternary consensus.

Figure 2: A sample FLAIR image from the MSSEG data set (left), the binary
consensus (middle) and the proposed ternary consensus (right). The Lesion is
annotated in red. In the ternary consensus, the Uncertainty is indicated in
yellow.

Uncertainty is in yellow in the ternary consensus, the last used as ground
truth to train our framework in reaching a ternary classification.

###  3.3 CNN architecture

Being the problem at hand the classification of 2D slices of a FLAIR brain
volume into the three classes, Background, Uncertainty and Lesion, in this
work we use the U-Net 2D architecture [39] depicted in Figure 3, being U-nets
specifically designed for these tasks.

Figure 3:  The used U-net "D architecture. The architecture is the same for 6
classifiers, though they have been trained separately.

Compared to the traditional U-net architecture, we insert a batch
normalization layer in each block to mitigate the effects of the gradient
amplification [42] in the regions surrounding the lesions, though with a
relevant increase of computational costs (about 30%).

To optimize the number of blocks, $n$, we have performed preliminary training,
with $n\in\\{3,4,5\\}$. We have not gone outside this set because for $n=5$
the U-Net started to overfit, even when using high values of
L2-Regularization, and for $n=2$ a dramatic drop of performance occurred. With
$n=4$, the problems related to overfitting have disappeared and the
performance was good but some redundancy remained. For this reason, we have
trained the CNN with $n=3$ and verified that redundancy was greatly reduced
and training converged faster than for $n=4$: hence, $n=3$ is the number of
selected blocks and used thereinafter.

###  3.4 Loss function and process optimization

The architecture we use has to solve a three class automatic annotation, for
which a Multi-label Cross Entropy Loss Function is necessary, defined as
follows:

$loss=\frac{1}{N}\sum_{n=1}^{N}\sum_{i=1}^{K}(T_{ni}log(Y_{ni})+(1-T_{ni})log(1-Y_{ni}))$
(1)

where $N$ and $K$ are the numbers of observations and classes, respectively.

The use of three classes, besides the problem stabilization (the presence of
Uncertainty gives better confidence in defining both Lesion and Background),
allows also to consider another important aspect. Indeed, we can optimize two
CNN, sharing the same architecture and the same loss function (Eq.1), but with
a different learning focuses: Lesion and Background. The Uncertainty is used
as a sort of "_buffer class_ ". In the case of a binary classification this
would not have been possible: the optimization of one would automatically lead
to the optimization of the other (what is not Lesion is Background and vice-
versa). The usage of the Uncertainty gave to both CNN a new choice to break
that constraint.

To achieve faster training and better performance, in this work the training
process of each CNN is controlled by the following hyperparameters [44, 61]:

  1. 1.

Starting Learning Rate: it is related to the data set and to the type of
neural network.

  2. 2.

L2-Regularization: it prevents overfitting.

  3. 3.

Class balancing: it optimizes the amplification factor for the represented
classes and improves training. Here we have three classes, hence two weights
are sufficient (the Lesion weight and the Background one).

Of the above, the first two are standard for CNN, while Class balancing is
specific for our CNN because it helps to differentiate the path of
optimization.

The selected hyperparameters are automatically optimized through a Bayesian
approach [44] applied to the following optimization problem:

$x^{*}=argmin_{x\in X}f(x)$ (2)

where $X$ is the domain of $x$, $f(x)$ represents an objective function to be
minimized and $x^{*}$ is the hyperparameter setting that yields the optimal
value of $f(x)$.

In this work $f(x)$ is defined as

$f(x)=1-IoU(x)$ (3)

where $IoU$ is the Intersection over Union score [13] defined in Section 4.

For the two CNN used therein, the $IoU$ is calculated on Lesion for the first
and on Background for the second.

The hyperparameter settings for the optimized CNN in each direction are
different and justify different training paths for the CNN and different
points of convergence for each of them. Figure 4 shows the different behaviour
of the two CNN in the segmentation grad-cams of a sample image, for the axial
direction. The CNN optimized for Lesion, tends to enlarge Lesion and
Uncertainty with respect to the CNN optimized for Background, which surrounds
lesions from outside.

Figure 4:  Grad-cam representation for an axial sample image for both CNN,
Lesions from inside (In) and from outside (Out), respectively. Grad-cams are
shown for the three classes, Lesion (first column), Uncertainty (second
column) and Background (third column). The fourth column shows the resulting
classification.

###  3.5 Ensemble Classification

It is well known that ensemble classifiers often perform better and more
robustly than their single components [57, 48]. For classification, we use 2D
slices of the whole volume, with specific CNN trained separately for Lesion
and Background, each for axial, radial and sagittal orientations,
respectively. In this way, we avoid that a particular orientation could be
favourable to lesions or to the classifier. Further, it serves to ensure 3D
continuity and to consider the context of lesions in the surrounding
environment.

We obtain a set of 6 classifiers, whose classification has to be merged to
produce a single output resembling the reasoning of the radiologists: though
3D FLAIR data are collected following sagittal planes to account for
clinical/physical issues, radiologists often use axial images for data
interpretation and the other orientations for confirmation [49, 16].
Accordingly, we prefer axial classifications and use coronal and sagittal
output for confirmation.

Regarding axial classifications, since each of the two CNN referring to the
same direction operates in the same scenario but with different focuses, they
collect specific information where reasoning is specific. Since both specific
contributions are important, besides common findings, a Union operation
between the two classifications is required. This is in accordance with the
double reading procedure followed by radiologists [17].

However, being the classified volume a three values data set, the Union does
not correspond to the classic binary union operation. In our case, the Lesion
is privileged, then comes the Uncertainty and, finally, the Background. In
fact, a voxel is classified as part of a lesion if at least one of the two
classifications considers it as a lesion; elsewhere, if at least one of the
two classifications considers it as uncertain, it is classified as
Uncertainty, elsewhere, it is considered as Background.

After Union operation, false positives are more present than in each single
classifier: their number is reduced by using the majority vote between the
other 4 classifications (two coronal and 2 sagittal, being the comparison
performed along axial planes). In fact, for each voxel the class is maintained
if at least two of the other classifiers confirm it, elsewhere it is
downgraded by one (a potential Lesion becomes Uncertainty, a potential
Uncertainty becomes Background): a double step is not allowed. This means that
first a decision is made on the Lesion and, then, on the Uncertainty by using
the data set resulting from the application of the process to the Lesion.

The ensemble of different classifiers is justified both by the fact that the
Union has to join common information, as well as specific information from
each of the two axial classifiers, and because each potentially positive voxel
needs confirmation from the coronal and sagittal classifiers (in this case,
also 3D spatial information is considered). We have preferred to resemble the
usual procedure used by radiologists and to rely on the usual benefits of
using an ensemble classifier.

In the proposed automatic pipeline, we have copied the human behavior by
privileging axial sections with respect to the others, but we have also
performed trials regarding the preference of the other orientations in the
fusion process and the results, not shown, confirm that the axial preference
gives the best results, closely followed by the coronal and, at a great
distance, by the sagittal, though this is the direction used for 3D FLAIR data
collection. This could be explained, at least partially, by the fact that
axial and coronal slices show highly symmetrical shape both regarding brain
anatomy and lesion shapes, also across subjects, thus making the learning
process easier than for sagittal slices. In sagittal directions symmetry is
absent and a huge variation of the image content could correspond to a little
rotation of the head.

##  4 Performance Parameters

In the binary classification problem the voxels can be positive (P, or Lesion)
or negative (N or Background). In a ternary ground-truth, P represents the
voxels of the class we are considering at present (Lesion or Uncertainty),
while N represents the negative voxels (those of the other two classes).
According to this definition, the same rules apply for each rater (with
respect to the class considered at present): $TP$ are the true positive
voxels, $TN$ are the true negative voxels, $FP$ are the false positive voxels
and $FN$ are the false negative voxels. Referring to a given class, a
classified voxel can be just in one of the states $TP$, $TN$, $FP$ and $FN$.

As far as we need an exhaustive comparison between all the raters involved
therein (artificial, single humans and ground truth), and being a unique
performance parameter unavailable, we define and calculate all the mostly
known scores and metrics. Despite of a certain redundancy, we use the rule of
thumb that it is better to over-describe the results than under-describe them
in order to highlight any potential sorts of errors the model could make and
to avoid that masking these errors could result in an advantage for the model
itself. In what follows, we define all the used parameters by separating
scores, defined in $[0,1]$ and whose ideal value is 1, from metrics, defined
in $[0,\infty)$ and whose best value is 0. The two groups are distinguished
for graphical purposes. For more details about the reported metrics, please
refer to [12, 11, 13].

###  4.1 Scores (convergent to 1)

Sensitivity (also called recall or true positive rate) is defined as:

$SENS=\frac{TP}{TP+FN}$ (4)

$SENS$ measures the portion of positive voxels that are correctly
identified.We also distinguish an object sensitivity, $OSENS$, defined as:

$OSENS=\frac{TP_{o}}{TP_{o}+FN_{o}}$ (5)

in which the prefix $O$ and the subscript $o$ indicate we are referring to
whole objects and not to single voxels. An object is considered as $TP$ if the
intersection with the corresponding object in the ground-truth is not empty.

Specificity ($SPEC$) is defined as:

$SPEC=\frac{TN}{TN+FP}$ (6)

$SPEC$ represents the portion of negative voxels $N$ that have been correctly
identified. For the treated case, where classes are strongly unbalanced,
$SPEC$ is biased by the fact that most of the image surface is covered by
background (unbalancing).

For the accuracy ($ACC$),we use the following normalized definition to reduce
unbalancing effects:

$ACC=\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right)/2$ (7)

Positive Predicted Value ($PPV$), also called Precision, is defined as:

$PPV=\frac{TP}{TP+FP}$ (8)

$PPV$ represents the portion of voxels identified as positives which are
really positives ($TP$).

As for $OSENS$, we define an object-based $PPV$, $OPPV$ as follows:

$OPPV=\frac{TP_{o}}{TP_{o}+FP_{o}}$ (9)

in which the prefix $O$ and the subscript $o$ indicate we are referring to
whole objects not to single objects, as above.

Dice score, also called Sorensen–Dice coefficient, is defined as:

$Dice=\frac{2*TP}{2*TP+FP+FN}$ (10)

$Dice$ measures the similarity between two data sets. This index is widely
used in AI for the validation of image segmentation algorithms.

$Image$ $Dice$ score uses the same equation of $Dice$ but, while $Dice$ is
calculated on the whole data set, $Image$ $Dice$ is applied on each image
separately and the results are averaged on the number of images. $Image$
$Dice$ allows to the so called per-image evaluation [12]. Per-image
evaluations are important because they tend to highlight the local behaviour.
For our comparisons, we calculate both $Dice$ and $Image$ $Dice$.

A score similar to $Dice$ is the Intersection Over Union ($IoU$):

$IoU=\frac{TP}{TP+FP+FN}$ (11)

where the difference is in the weight of $TP$.

The $F1$ Score (calculated for whole objects and not for single voxels) is
defined as:

$F1=2*\frac{OSENS*OPPV}{OSENS+OPPV}$ (12)

where $OSENS$ and $OPPV$ are defined above.

The Boundary F1 score ($BF$) is a per-image version of $F1$ score.

Pearson Correlation Coefficient ($PCC$), between two data sets $A$ and $B$, is
defined as:

$PCC(A,B)=\frac{cov(A,B)}{\sigma_{A}*\sigma_{B}}$ (13)

where $cov(A,B)$ is the covariance of $A$ and $B$ and $\sigma_{A}$ and
$\sigma_{B}$ are the standard deviation of $A$ and $B$, respectively. $PCC$
ranges in the interval $[-1,1]$ and a negative value of $PCC$ indicates a
similarity of the object $A$ with the negative version of the object $B$.

###  4.2 Metrics (convergent to 0)

The following metrics are those used in the present manuscript in which the
ideal value is 0.

Extra Fraction ($EF$), is defined as:

$EF=\frac{FP}{TP+FN}$ (14)

Detection error rate ($DER$) is defined as:

$DER=\frac{DE}{MTA}$ (15)

where the detection error $DE$ is calculated as the sum of the voxels of
connected regions wrongly marked as positive by the rater and the mean total
area $MTA$ is defined as the average between the number of positive voxels
from the rater and the ground-truth. $DER$ measures the disagreement in
detecting the same regions between a rater under and the ground-truth.

Outline Error Rate ($OER$) is defined as:

$OER=\frac{OE}{MTA}$ (16)

where $OE$ is the outline error calculated as the difference between the
number of voxels of the union and that of the intersection between the
positive connected regions of both the rater and the ground-truth. $OER$
measures the disagreement in outlining the same objects between a rater and
the ground truth.

False Detection Ratio ($FDE$) is defined as:

$FDE=\frac{FP}{P}$ (17)

Relative Area Error ($RAE$) is defined as:

$RAE=\frac{TP+FP-P}{P}$ (18)

Hausdorff Distance ($HD$) between two objects $A$ and $B$ is defined as:

$HD(A,B)=max(h(A,B),h(B,A))$ (19)

where $h(A,B)$ is:

$h(A,B)=max_{a\in A}min_{b\in B}\parallel a-b\parallel$ (20)

$HD$ measures how far two subsets are from each other. In other words, two
sets are close with respect to $HD$ if every point of one set is close to a
certain point of the other set.

Euclidean Distance ($ED$) between two objects $A$ and $B$ is defined as:

$ED(A,B)=max(d(A,B),d(B,A))$ (21)

where $d(A,B)$ is defined as:

$d(A,B)=\frac{1}{N}\sum_{a\in A}min_{b\in B}\parallel a-b\parallel$ (22)

Surface Distance ($SD$) is defined as:

$SD=\frac{\sum_{i\in A_{S}}d(x_{i},G_{S})+\sum_{j\in
G_{S}}d(x_{j},A_{S})}{N_{A}+N_{G}}$ (23)

where $A_{S}$ and $G_{S}$ are two segmentations (one is the rater segmentation
and the other is the ground truth), $d$ denotes the minimal $ED$ between
voxels on both surfaces, while $N_{A}$ and $N_{G}$ denote the number of points
of each surface.

##  5 Results and Discussion

The proposed framework has been trained, validated and tested on the ternary
ground-truth defined above. As far as the ground-truth maintains unaltered
Lesion of the original binary ground-truth, we guarantee a direct comparison
with human raters and, at the same time, with already existing automated
methods.Regarding the segmented Uncertainty, a comparison is possible just
with respect to the ternary ground-truth, since for the human raters
Uncertainty is unavailable.

The evaluation of the proposed framework and of the human radiologists, with
respect both to the ground-truth and to each-other, is performed by applying
the cross-validation approach defined in Subsection 3.1. Average and standard
deviation values are calculated for the evaluation parameters defined in
Section 4 and divided in two groups: those whose ideal value is 1 and those
whose ideal value is 0.

The first results, reported in Fig 5, are those between the raters and the
ground-truth performed on the Lesion class. This is also an indirect
comparison, through the ground-truth, between the proposed framework and the
human raters.

Figure 5: Comparison between the raters and the ground truth, performed on the
Lesion class. The reported metrics are separated in those whose ideal value is
1 (a) and those whose ideal value is 0 (b). Average and standard deviation are
reported. Euclidean, Hausdorff and Surface distances are shown in cm units.

For a better overview, the mean values are also shown in Fig. 6 by using a
radar visualization: they confirm that the behaviour of the proposed method is
inside the inter-rater variability.

Figure 6: Comparison between raters in the same conditions of Fig. 5. For
graphic purposes, only the average values are reported and the line of the
proposed framework (red) is highlighted with respect to the others.

As it can be observed, our framework is never the best or the worst, for at
least one of the metrics, as instead occurs for human raters. This can be
explained, at least partially, by the fact that it has been trained with the
consensus that, for its nature, tends to average pros and cons of the raters
from which it has been derived.

A Wilcoxon signed-rank test of the vectors of metric values confirms, with a
significance level of 0.01, that there is no significant difference between
the behaviour of our method and that of the 7 human raters, with respect to
the ground-truth, on the Lesion class. This means that, if data are shown
without labels, it would be impossible to recognize the automated rater from
humans.

As far as the lesion size could greatly affect the performance of the
classification [11] and the previously reported results are averaged with
respect to the lesion volume, we repeat the comparison by changing the lesion
volume. To this aim, we consider all the lesions separately and the
calculations are performed lesion by lesion, by maintaining separated also the
lesions of the same volume. In this way we can: 1) visualize potential
outliers; 2) represent lesion density; 3) avoid local averaging that could
mask specific contributions. The results, reported just for the most commonly
used evaluation parameters [11], are shown in Fig. 7.

Figure 7: Dice score (a), F1-score (b) and Surface Distance (c), calculated
for each lesion and shown with respect to the lesion volume for the human
raters and the proposed framework. To improve readability; lesion volume is
represented in logarithmic scale; framework’s values (red) are highlighted.

It is confirmed the analogy of behavior between the proposed framework and the
7 human raters, though for some volumes our framework has exhibited results
close to the borders of oscillation of the 7 human raters. Indeed, a greater
dispersion can be observed for F1-score and a relatively high average value is
observable for $SD$, though in line with that of some human raters.

The above good results are not sufficient alone to state that our framework
behaves like human raters because the comparison is through the ground-truth.
In other words, our framework could be at the same ’distance’ as the human
raters are from the ground-truth, but from opposite sides. For this reason, a
direct comparison is necessary to finally confirm the similarity between the
proposed framework and the human raters. To this aim, we perform the
experiment of comparing all the raters to each other by considering ground-
truth all of them, including our framework, in rotation. Results, for the most
frequently used metrics, are reported in Fig. 8.

Figure 8: Comparison between all the raters with respect to each other,
including our framework and consensus, each alternately considered as the
ground-truth. For readability purposes, only some metrics are reported. In
this representation, placing together metrics converging to 1 and to 0, the
angular position indicates the metric value: clock wise versus for metrics
converging to 1, anti-clock wise versus for metrics converging to 0. Radial
information is only used to separate the current ground-truth raters. Ground-
truth raters have colored bullets placed on the vertical line.

These results confirm that the proposed framework behaviour does not differ
from that of the other human raters and that it is not polarized toward a
specific rater or toward the ground-truth. Moreover, as other Authors have
highlighted [20], results show similarities among some human raters (R4 with
R5 and R6 with R7). Fortunately, results also confirm that the ground-truth is
not biased by the similarity between some raters, and that it maintains a
"human" behaviour, being it very close to the raters R1 and R2. This is a
fundamental aspect because it means that all the people who are attempting to
train automated systems with respect to the ground-truth, including ourselves,
are not following a "chimera" to which, paradoxically, the closer we get, the
more we move away from the proper objective.

A visual overview of the behaviour of the proposed framework in the whole
process of identification/segmentation, both for Lesion and for Uncertainty,
with respect to the ternary ground truth, is shown in Fig. 9. The ternary
ground-truth is reported on the left side, the corresponding segmentation
obtained with the proposed framework is presented on the right side, for the
same subject and slices. Both for Lesion and Uncertainty, the proposed
framework selects more than necessary ($FP$ are evident). Interestingly, $FN$
are almost absent from the segmented volume. The other interesting property
shown by the proposed framework is the good spatial continuity of the lesion
structures in the 3D model of Lesion (upper right panel, where Lesion is red
colored while Uncertainty is yellow colored).

Figure 9: Comparison between the ternary ground truth (left) and the proposed
automated framework (right). Lesion is red and Uncertainty is yellow. For
readability purposes, the upper right panel of each side shows just the
healthy brain and Lesion in 3D.

To complete our discussion, we compare the proposed framework with recently
proposed automated strategies. To this aim, Table 1 contains this indirect
comparison on the metrics calculated for at least one of the other methods.
The necessary condition for a method to be considered in Table 1 is to have
been trained, validated and tested on the MSSEG data set. In this way, we can
ensure that the comparison is homogeneous and performed on the same conditions
of that obtained with respect to the 7 human raters.

Table 1: Comparison between our framework and the state of the art methods.
Just averages are reported and the symbol ’-’ is for unavailable data. For the
state of the art strategies, the performance values are those reported in the
original manuscripts, while for Team Fusion (TF) the metrics are those
reported in [11].

Method | MRI mod. | Sens | OSens | TPR | Acc | PPV | OPPV | Dice | F1 | SD
---|---|---|---|---|---|---|---|---|---|---
| TS [11]
---
| FLAIR, PD, T2
---
T1, G-E T1
0.71 | 0,60 | 0.99 | - | 0.65 | 0.53 | 0.64 | 0.50 | 0.91
[19] |  | FLAIR, PD, T2
---
T1, G-E T1
0.65 | - | 0.86 | 0.97 | - | - | 0.76 | - | -
[51] | FLAIR, T1 | 0.55 | - | - | - | - | 0.79 | 0.63 | - | -
[3] |  | FLAIR, PD
---
T1, G-E T1
0.76 | - | - | - | - | - | 0.82 | - | -
[34] | FLAIR, T1, T2 | - | - | - | - | - | - | 0.76 | 0.59 | -
Our | FLAIR | 0.88 | 0.77 | 0.98 | 0.88 | 0.81 | 0.81 | 0.77 | 0.72 | 0.27

Though a global ranking is difficult, data reported in Table 1 are clear: the
proposed framework is the most stable with respect to different metrics and it
generally outperforms the other methods, including those using multiple
imaging. This could have an interesting implication: FLAIR would contain
sufficient information, not only necessary, to identify and segment all MS
lesions occurring in the WM. Potentially positive consequences are: a) due to
the huge variability of MRI and of its modalities, the usage of a single
modality could increase the performance, above using multiple modalities,
because it could greatly contribute to stabilize automatic
identification/classification; b) the acquisition time and stress for the
patient can be reduced.

An important aspect that has determined the outstanding performance of the
proposed framework is the use of the ternary ground-truth. Indeed, Fig. 10
shows the average performance results when the proposed framework is trained
without the Uncertainty (on the binary consensus) as compared to those
obtained when trained on the ternary consensus. The ensemble method trained
without the Uncertainty outperforms similar automated strategies (team fusion
in [11]), though it is still far from humans: the step which places the
proposed framework among humans is the inclusion in the pipeline of the class
Uncertainty.

Figure 10: The proposed method when trained without and with Uncertainty,
compared both on metrics whose ideal value is 1 (a) and for metrics whose
ideal value is 0 (b).

This is in line with what reported in [20]: the framework learns better what
is surely Lesion, what is surely Background and uses Uncertainty for doubts,
as a buffer class. Indeed, the polarized and ambiguous classification of
doubtful voxels, sometimes as Lesion and others as Background, disorients any
automated strategy and deviates it from the human reasoning.

##  6 Conclusion and future work

An automated framework for the identification/segmentation of MS lesions from
FLAIR MRI images has been presented. We have demonstrated that traditional CNN
architectures, if placed in a context emulating the procedures of human
specialists, could effectively behave like a human expert. The strength points
of the proposed framework are the following: 1) to train the system to
recognize both the lesions as they are and with respect to the environment
they are immersed in, thus allowing to incorporate also a sort of meta-
information regarding the environment where MS lesions mostly occur; 2) to
resemble radiologists in consulting axial slices to discover potential lesions
and to check radial and sagittal slices for confirmation, as well as to
maintain 3D continuity to their findings; 3) to use an ensemble classification
that usually performs better than its components; 4) to use an artificially
generated Uncertainty class to improve the performance of an automated
strategy and to make it more similar to the human reasoning; 5) to operate
just on FLAIR images.

Results have shown that the proposed framework resembles human raters both in
behaviour and in performance, when compared with the MSSEG consensus on
Lesion. Indeed, Wilcoxon statistical test has assessed the framework ability
to exhibit a behaviour that is equivalent to, or indistinguishable from, that
of a human rater. Furthermore, the proposed framework outperforms the state of
the art strategies which have been trained, validated and tested on the MSSEG
data set. Finally, the usage of the uncertainty during training has greatly
improved the performance of the framework.

In a recent report [24], the JASON Advisory Group has identified several key
recommendations for advancing computation technology into routine clinical
practice. One of them is that new technologies should address a significant
clinical need, be practical in use and reduce medical system costs. The
demonstration that a better performance is possible by including some concepts
(Uncertainty and Ensemble) to enrich traditional CNN architectures goes in the
direction of the above recommendations.

Future directions of exploration will be: 1) an accurate definition of
"Uncertainty" and a deep study of its role in the intrinsically uncertain
decision-making process; 2) the use of specific pre-processing strategy for
the FLAIR images to further improve the robustness of the method with respect
to MRI and MRI scanners variability; 3) how multiple imaging could affect the
classification performance; 4) how the use of a different loss functioncould
better deal the unbalancing problem 5) to study a "soft" consensus based on a
single class (lesion) with different probability valuesto reduce problem
complexity and to explore a "soft" loss function; 6) to test the proposed
framework to identify/segment also cortical lesions and for longitudinal
studies (temporal evolution of the disease); 7) to explore specific pre-
processing strategies to make all the used modalities robust with respect to
MRI and MRI scanners; 8) to study the role of additional meta-information
informationin improving lesion identification.

## Acknowledgments

The Authors wish to thank Dr Maria Silvia Marottoli for her contribution to
the improvement of the manuscript.

## References

  * Com [2016] _MSSEG Challenge Proceedings: Multiple Sclerosis Lesions Segmentation Challenge Using a Data Management and Processing Infrastructure_ , Athènes, Greece, 2016.
  * Afzal et al. [2020] H. M. Rehan Afzal, Suhuai Luo, Saadallah Ramadan, and Jeannette Lechner-Scott.  The emerging role of artificial intelligence in multiple sclerosis imaging.  _Multiple Sclerosis Journal_ , page 135245852096629, October 2020\.
  * Alijamaat et al. [2021] Ali Alijamaat, Alireza NikravanShalmani, and Peyman Bayat.  Multiple sclerosis lesion segmentation from brain mri using u-net based on wavelet pooling.  _International Journal of Computer Assisted Radiology and Surgery_ , pages 1–9, 2021.
  * Alizadehsani et al. [2021] Roohallah Alizadehsani, Mohamad Roshanzamir, Sadiq Hussain, Abbas Khosravi, Afsaneh Koohestani, Mohammad Hossein Zangooei, Moloud Abdar, Adham Beykikhoshk, Afshin Shoeibi, Assef Zare, Maryam Panahiazar, Saeid Nahavandi, Dipti Srinivasan, Amir F. Atiya, and U. Rajendra Acharya.  Handling of uncertainty in medical data using machine learning and probability theory techniques: a review of 30 years (1991–2020).  _Annals of Operations Research_ , March 2021.
  * Aslani et al. [2019] Shahab Aslani, M. Dayan, L. Storelli, M. Filippi, Vittorio Murino, M. Rocca, and Diego Sona.  Multi-branch convolutional neural network for multiple sclerosis lesion segmentation.  _NeuroImage_ , 196:1–15, 2019.
  * Aït-Ali et al. [2005] L. S. Aït-Ali, S. Prima, P. Hellier, B. Carsin, G. Edan, and C. Barillot.  STREM: A robust multidimensional parametric method to segment MS lesions in MRI.  In _Lecture Notes in Computer Science_ , pages 409–416. Springer Berlin Heidelberg, 2005.
  * Brosch et al. [2016] T. Brosch, L. Y. W. Tang, Y. Yoo, D. K. B. Li, A. Traboulsee, and R. Tam.  Deep 3d convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation.  _IEEE Transactions on Medical Imaging_ , 35(5):1229–1239, 2016.
  * Cabezas et al. [2014] Mariano Cabezas, Arnau Oliver, Sergi Valverde, Brigitte Beltran, Jordi Freixenet, Joan C. Vilanova, Lluís Ramió-Torrentà, Àlex Rovira, and Xavier Lladó.  BOOST: A supervised approach for multiple sclerosis lesion segmentation.  _Journal of Neuroscience Methods_ , 237:108–117, nov 2014\.
  * Carass et al. [2017] Aaron Carass, Snehashis Roy, Amod Jog, Jennifer L. Cuzzocreo, Elizabeth Magrath, Adrian Gherman, Julia Button, James Nguyen, Ferran Prados, Carole H. Sudre, Manuel Jorge Cardoso, Niamh Cawley, Olga Ciccarelli, Claudia A.M. Wheeler-Kingshott, Sébastien Ourselin, Laurence Catanese, Hrishikesh Deshpande, Pierre Maurel, Olivier Commowick, Christian Barillot, Xavier Tomas-Fernandez, Simon K. Warfield, Suthirth Vaidya, Abhijith Chunduru, Ramanathan Muthuganapathy, Ganapathy Krishnamurthi, Andrew Jesson, Tal Arbel, Oskar Maier, Heinz Handels, Leonardo O. Iheme, Devrim Unay, Saurabh Jain, Diana M. Sima, Dirk Smeets, Mohsen Ghafoorian, Bram Platel, Ariel Birenbaum, Hayit Greenspan, Pierre-Louis Bazin, Peter A. Calabresi, Ciprian M. Crainiceanu, Lotta M. Ellingsen, Daniel S. Reich, Jerry L. Prince, and Dzung L. Pham.  Longitudinal multiple sclerosis lesion segmentation: Resource and challenge.  _NeuroImage_ , 148:77–102, mar 2017.
  * Carré et al. [2020] Alexandre Carré, Guillaume Klausner, Myriam Edjlali, Marvin Lerousseau, Jade Briend-Diop, Roger Sun, Samy Ammari, Sylvain Reuzé, Emilie Alvarez Andres, Théo Estienne, Stéphane Niyoteka, Enzo Battistella, Maria Vakalopoulou, Frédéric Dhermain, Nikos Paragios, Eric Deutsch, Catherine Oppenheim, Johan Pallud, and Charlotte Robert.  Standardization of brain MR images across machines and protocols: bridging the gap for MRI-based radiomics.  _Scientific Reports_ , 10(1), jul 2020.
  * Commowick et al. [2018] Olivier Commowick, Audrey Istace, Michaël Kain, Baptiste Laurent, Florent Leray, Mathieu Simon, Sorina Camarasu Pop, Pascal Girard, Roxana Améli, Jean-Christophe Ferré, Anne Kerbrat, Thomas Tourdias, Frédéric Cervenansky, Tristan Glatard, Jérémy Beaumont, Senan Doyle, Florence Forbes, Jesse Knight, April Khademi, Amirreza Mahbod, Chunliang Wang, Richard McKinley, Franca Wagner, John Muschelli, Elizabeth Sweeney, Eloy Roura, Xavier Lladó, Michel M. Santos, Wellington P. Santos, Abel G. Silva-Filho, Xavier Tomas-Fernandez, Hélène Urien, Isabelle Bloch, Sergi Valverde, Mariano Cabezas, Francisco Javier Vera-Olmos, Norberto Malpica, Charles Guttmann, Sandra Vukusic, Gilles Edan, Michel Dojat, Martin Styner, Simon K. Warfield, François Cotton, and Christian Barillot.  Objective evaluation of multiple sclerosis lesion segmentation using a data management and processing infrastructure.  _Scientific Reports_ , 8(1), sep 2018.
  * Csurka et al. [2013] Gabriela Csurka, Diane Larlus, Florent Perronnin, and France Meylan.  What is a good evaluation measure for semantic segmentation?.  In _Bmvc_ , volume 27, pages 10–5244, 2013.
  * Danelakis et al. [2018] Antonios Danelakis, Theoharis Theoharis, and Dimitrios A. Verganelakis.  Survey of automated multiple sclerosis lesion segmentation techniques on magnetic resonance imaging.  _Computerized Medical Imaging and Graphics_ , 70:83 – 100, 2018.  ISSN 0895-6111.
  * ElSebely et al. [2021] Randa ElSebely, Ahmed Hassan Yousef, Ashraf A Salem, and Bassem Abdullah.  Automatic segmentation of multiple sclerosis lesions in brain mr images using ensemble machine learning.  In _2021 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC)_ , pages 28–33. IEEE, 2021.
  * Fenneteau et al. [2020] Alexandre Fenneteau, Pascal Bourdon, David Helbert, Christine Fernandez-Maloigne, Christophe Habas, and Remy Guillevin.  Learning a cnn on multiple sclerosis lesion segmentation with self-supervision.  In _International Symposium on Electronic Imaging 2020 3D Measurement and Data Processing_ , pages 1–8. HAL open science, 2020.
  * Filippi et al. [2019] Massimo Filippi, Paolo Preziosa, Brenda L Banwell, Frederik Barkhof, Olga Ciccarelli, Nicola De Stefano, Jeroen J G Geurts, Friedemann Paul, Daniel S Reich, Ahmed T Toosy, Anthony Traboulsee, Mike P Wattjes, Tarek A Yousry, Achim Gass, Catherine Lubetzki, Brian G Weinshenker, and Maria A Rocca.  Assessment of lesions on magnetic resonance imaging in multiple sclerosis: practical guidelines.  _Brain_ , 142(7):1858–1875, 06 2019.  ISSN 0006-8950.
  * Geijer and Geijer [2018] Håkan Geijer and Mats Geijer.  Added value of double reading in diagnostic radiology, a systematic review.  _Insights into Imaging_ , 9(3):287–301, March 2018.
  * Geremia et al. [2011] Ezequiel Geremia, Olivier Clatz, Bjoern H. Menze, Ender Konukoglu, Antonio Criminisi, and Nicholas Ayache.  Spatial decision forests for MS lesion segmentation in multi-channel magnetic resonance images.  _NeuroImage_ , 57(2):378–390, July 2011.
  * Ghosal et al. [2019] Palash Ghosal, Pindi Krishna Chandra Prasad, and Debashis Nandi.  A light weighted deep learning framework for multiple sclerosis lesion segmentation.  In _2019 Fifth International Conference on Image Information Processing (ICIIP)_ , pages 526–531. IEEE, 2019.
  * Gros et al. [2021] Charley Gros, Andreanne Lemay, and Julien Cohen-Adad.  Softseg: Advantages of soft versus binary training for image segmentation.  _Medical Image Analysis_ , 71:102038, 2021.
  * Guizard et al. [2015] Nicolas Guizard, Pierrick Coupé, Vladimir S. Fonov, Jose V. Manjón, Douglas L. Arnold, and D. Louis Collins.  Rotation-invariant multi-contrast non-local means for MS lesion segmentation.  _NeuroImage: Clinical_ , 8:376–389, 2015.
  * Hashemi et al. [2019] S. R. Hashemi, S. S. Mohseni Salehi, D. Erdogmus, S. P. Prabhu, S. K. Warfield, and A. Gholipour.  Asymmetric loss functions and deep densely-connected networks for highly-imbalanced medical image segmentation: Application to multiple sclerosis lesion detection.  _IEEE Access_ , 7:1721–1735, 2019.
  * Huang et al. [2021] Yi-Jie Huang, Qi Dou, Zi-Xian Wang, Li-Zhi Liu, Ying Jin, Chao-Feng Li, Lisheng Wang, Hao Chen, and Rui-Hua Xu.  3-d roi-aware u-net for accurate and efficient colorectal tumor segmentation.  _IEEE Transactions on Cybernetics_ , 51(11):5397–5408, 2021.
  * JASON [2017] JASON.  Artificial intelligence for health and heath care.  _JSR-17-Task-002_ , 2017.
  * Kang et al. [2020] Guixia Kang, Beibei Hou, Yiyuan Ma, Fabrice Labeau, Zichen Su, et al.  Acu-net: A 3d attention context u-net for multiple sclerosis lesion segmentation.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1384–1388. IEEE, 2020.
  * Karimaghaloo et al. [2012] Z. Karimaghaloo, M. Shah, S. J. Francis, D. L. Arnold, D. L. Collins, and T. Arbel.  Automatic detection of gadolinium-enhancing multiple sclerosis lesions in brain mri using conditional random fields.  _IEEE Transactions on Medical Imaging_ , 31(6):1181–1194, 2012.
  * Kats et al. [2019] Eytan Kats, Jacob Goldberger, and Hayit Greenspan.  Soft labeling by distilling anatomical knowledge for improved MS lesion segmentation.  In _2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019)_. IEEE, apr 2019.
  * Kaur et al. [2020] Amrita Kaur, Lakhwinder Kaur, and Ashima Singh.  State-of-the-art segmentation techniques and future directions for multiple sclerosis brain lesions.  _Archives of Computational Methods in Engineering_ , feb 2020.
  * Lian et al. [2022] Chunfeng Lian, Mingxia Liu, Yongsheng Pan, and Dinggang Shen.  Attention-guided hybrid network for dementia diagnosis with structural mr images.  _IEEE Transactions on Cybernetics_ , 52(4), 2022.
  * Litjens et al. [2017] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen A.W.M. van der Laak, Bram van Ginneken, and Clara I. Sánchez.  A survey on deep learning in medical image analysis.  _Medical Image Analysis_ , 42:60–88, dec 2017.
  * Liu et al. [2021] Chenghao Liu, Xiangzhu Zeng, Kongming Liang, Yizhou Yu, and Chuyang Ye.  Improved brain lesion segmentation with anatomical priors from healthy subjects.  pages 186–195. Springer International Publishing, 2021.
  * Liu et al. [2020] Mingxia Liu, Jun Zhang, Chunfeng Lian, and Dinggang Shen.  Weakly supervised deep learning for brain disease prognosis using mri and incomplete clinical scores.  _IEEE Transactions on Cybernetics_ , 50(7):3381–3392, 2020.
  * Macar et al. [2021] U. Macar, Enamundram Naga Karthik, Charley Gros, Andr’eanne Lemay, and Julien Cohen-Adad.  Team neuropoly: Description of the pipelines for the miccai 2021 ms new lesions segmentation challenge.  _ArXiv_ , abs/2109.05409, 2021.
  * McKinley et al. [2021] Richard McKinley, Rik Wepfer, Fabian Aschwanden, Lorenz Grunder, Raphaela Muri, Christian Rummel, Rajeev Verma, Christian Weisstanner, Mauricio Reyes, Anke Salmen, et al.  Simultaneous lesion and brain segmentation in multiple sclerosis using deep neural networks.  _Scientific reports_ , 11(1):1–11, 2021.
  * Milletari et al. [2016] F. Milletari, N. Navab, and S. Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In _2016 Fourth International Conference on 3D Vision (3DV)_ , pages 565–571, 2016.
  * Nie et al. [2019] Dong Nie, Li Wang, Ehsan Adeli, Cuijin Lao, Weili Lin, and Dinggang Shen.  3-d fully convolutional networks for multimodal isointense infant brain image segmentation.  _IEEE Transactions on Cybernetics_ , 49(3):1123–1136, 2019.
  * Placidi [2012] G. Placidi.  Mri: Essentials for innovative technologies (1st ed.), 2012.
  * Placidi et al. [2019] Giuseppe Placidi, Luigi Cinque, Matteo Polsinelli, Alessandra Splendiani, and Emanuele Tommasino.  Automatic framework for multiple sclerosis follow-up by magnetic resonance imaging for reducing contrast agents.  In _Image Analysis and Processing – ICIAP 2019_ , pages 367–378, Cham, 2019. Springer International Publishing.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, _Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015_ , pages 234–241, Cham, 2015. Springer International Publishing.
  * Rosa et al. [2019] Francesco La Rosa, Mário João Fartaria, Tobias Kober, Jonas Richiardi, Cristina Granziera, Jean-Philippe Thiran, and Meritxell Bach Cuadra.  Shallow vs deep learning architectures for white matter lesion segmentation in the early stages of multiple sclerosis.  In _Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries_ , pages 142–151. Springer International Publishing, 2019\.
  * Roura et al. [2015] Eloy Roura, Arnau Oliver, Mariano Cabezas, Sergi Valverde, Deborah Pareto, Joan C. Vilanova, Lluís Ramió-Torrentà, Àlex Rovira, and Xavier Lladó.  A toolbox for multiple sclerosis lesion segmentation.  _Neuroradiology_ , 57(10):1031–1043, jul 2015\.
  * Santurkar et al. [2018] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry.  How does batch normalization help optimization?  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , NIPS’18, page 2488–2498, Red Hook, NY, USA, 2018. Curran Associates Inc.
  * Shiee et al. [2010] Navid Shiee, Pierre-Louis Bazin, Arzu Ozturk, Daniel S. Reich, Peter A. Calabresi, and Dzung L. Pham.  A topology-preserving approach to the segmentation of brain images with multiple sclerosis lesions.  _NeuroImage_ , 49(2):1524–1535, jan 2010.
  * Snoek et al. [2012] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams.  Practical bayesian optimization of machine learning algorithms.  In _Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’12, page 2951–2959, Red Hook, NY, USA, 2012. Curran Associates Inc.
  * Špiclin et al. [2018] Žiga Špiclin, Franjo Pernuš, Boštjan Likar, Tim Jerman, and Domen Ravnik.  Dataset variability leverages white-matter lesion segmentation performance with convolutional neural network.  In Elsa D. Angelini and Bennett A. Landman, editors, _Medical Imaging 2018: Image Processing_. SPIE, mar 2018.
  * Steinman [1996] M.D. Lawrence Steinman.  Multiple sclerosis: A coordinated immunological attack against myelin in the central nervous system.  _Cell_ , 85(3):299–302, 2021/01/07 1996.
  * Strumia et al. [2016] M. Strumia, F. R. Schmidt, C. Anastasopoulos, C. Granziera, G. Krueger, and T. Brox.  White matter ms-lesion segmentation using a geometric brain model.  _IEEE Transactions on Medical Imaging_ , 35(7):1636–1646, 2016.
  * Sundaresan et al. [2021] Vaanathi Sundaresan, Giovanna Zamboni, Peter M. Rothwell, Mark Jenkinson, and Ludovica Griffanti.  Triplanar ensemble u-net model for white matter hyperintensities segmentation on MR images.  73:102184, October 2021.
  * Traboulsee et al. [2016] A Traboulsee, JH Simon, L Stone, E Fisher, DE Jones, A Malhotra, SD Newsome, Jiwon Oh, DS Reich, N Richert, et al.  Revised recommendations of the consortium of ms centers task force for a standardized mri protocol and clinical guidelines for the diagnosis and follow-up of multiple sclerosis.  _American Journal of Neuroradiology_ , 37(3):394–401, 2016.
  * Valverde et al. [2017] Sergi Valverde, Mariano Cabezas, Eloy Roura, Sandra González-Villà, Deborah Pareto, Joan C. Vilanova, Lluís Ramió-Torrentà, Àlex Rovira, Arnau Oliver, and Xavier Lladó.  Improving automated multiple sclerosis lesion segmentation with a cascaded 3d convolutional neural network approach.  _NeuroImage_ , 155:159–168, jul 2017.
  * Valverde et al. [2019] Sergi Valverde, Mostafa Salem, Mariano Cabezas, Deborah Pareto, Joan C. Vilanova, Lluís Ramió-Torrentà, Àlex Rovira, Joaquim Salvi, Arnau Oliver, and Xavier Lladó.  One-shot domain adaptation in multiple sclerosis lesion segmentation using convolutional neural networks.  _NeuroImage: Clinical_ , 21:101638, 2019.
  * Vang et al. [2020] Yeeleng S Vang, Yingxin Cao, Peter D Chang, Daniel S Chow, Alexander U Brandt, Friedemann Paul, Michael Scheel, and Xiaohui Xie.  Synergynet: a fusion framework for multiple sclerosis brain mri segmentation with local refinement.  In _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_ , pages 131–135. IEEE, 2020.
  * Vincent et al. [2021] Olivier Vincent, Charley Gros, and Julien Cohen-Adad.  Impact of individual rater style on deep learning uncertainty in medical imaging segmentation.  _ArXiv_ , abs/2105.02197, 2021.
  * Wu et al. [2021] Huisi Wu, Xiujuan Chen, Ping Li, and Zhenkun Wen.  Automatic symmetry detection from brain mri based on a 2-channel convolutional neural network.  _IEEE Transactions on Cybernetics_ , 51(9):4464–4475, 2021.
  * Xue et al. [2021] Jie Xue, Kelei He, Dong Nie, Ehsan Adeli, Zhenshan Shi, Seong-Whan Lee, Yuanjie Zheng, Xiyu Liu, Dengwang Li, and Dinggang Shen.  Cascaded multitask 3-d fully convolutional networks for pancreas segmentation.  _IEEE Transactions on Cybernetics_ , 51(4):2153–2165, 2021.
  * Yang et al. [2021a] Yun Yang, Yuanyuan Hu, Xingyi Zhang, and Song Wang.  Two-stage selective ensemble of cnn via deep tree training for medical image classification.  _IEEE Transactions on Cybernetics_ , 37(3):1–14, 2021a.
  * Yang et al. [2021b] Yun Yang, Yuanyuan Hu, Xingyi Zhang, and Song Wang.  Two-stage selective ensemble of cnn via deep tree training for medical image classification.  _IEEE Transactions on Cybernetics_ , 2021b.
  * Yoo et al. [2014] Y. Yoo, T. Brosch, A. Traboulsee, D. Li, and R. Tam.  Deep learning of image features from unlabeled data for multiple sclerosis lesion segmentation.  In _MLMI_ , 2014.
  * Zhang et al. [2021] Hang Zhang, Jinwei Zhang, Rongguang Wang, Qihao Zhang, Susan A Gauthier, Pascal Spincemaille, Thanh D Nguyen, and Yi Wang.  Geometric loss for deep multiple sclerosis lesion segmentation.  In _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_ , pages 24–28. IEEE, 2021.
  * Zhang and Oguz [2021] Huahong Zhang and Ipek Oguz.  Multiple sclerosis lesion segmentation - a survey of supervised cnn-based methods.  In Alessandro Crimi and Spyridon Bakas, editors, _Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries_ , pages 11–29, Cham, 2021. Springer International Publishing.
  * Zhang et al. [2019] Xiang Zhang, Xiaocong Chen, Lina Yao, Chang Ge, and Manqing Dong.  Deep neural network hyperparameter optimization with orthogonal array tuning.  In Tom Gedeon, Kok Wai Wong, and Minho Lee, editors, _Neural Information Processing_ , pages 287–295, Cham, 2019. Springer International Publishing.
  * Zurita et al. [2018] Mariana Zurita, Cristian Montalba, Tomás Labbé, Juan Pablo Cruz, Josué Dalboni da Rocha, Cristián Tejos, Ethel Ciampi, Claudia Cárcamo, Ranganatha Sitaram, and Sergio Uribe.  Characterization of relapsing-remitting multiple sclerosis patients using support vector machine classifications of functional and diffusion MRI data.  _NeuroImage: Clinical_ , 20:724–730, 2018.
  * Özgün Çiçek et al. [2016] Özgün Çiçek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger.  3d u-net: Learning dense volumetric segmentation from sparse annotation.  In _Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016_ , pages 424–432. Springer International Publishing, 2016.