# A review of deep learning-based information fusion techniques for multimodal
medical image classification

Yihao Li Mostafa El Habib Daho<EMAIL_ADDRESS>Pierre-Henri
Conze Rachid Zeghlache Hugo Le Boité Ramin Tadayoni Béatrice Cochener Mathieu
Lamard Gwenolé Quellec LaTIM UMR 1101, Inserm, Brest, France  University of
Western Brittany, Brest, France  IMT Atlantique, Brest, France  Sorbonne
University, Paris, France  Ophthalmology Department, Lariboisière Hospital,
AP-HP, Paris, France  Paris Cité University, Paris, France  Ophthalmology
Department, CHRU Brest, Brest, France

###### Abstract

Multimodal medical imaging plays a pivotal role in clinical diagnosis and
research, as it combines information from various imaging modalities to
provide a more comprehensive understanding of the underlying pathology.
Recently, deep learning-based multimodal fusion techniques have emerged as
powerful tools for improving medical image classification. This review offers
a thorough analysis of the developments in deep learning-based multimodal
fusion for medical classification tasks. We explore the complementary
relationships among prevalent clinical modalities and outline three main
fusion schemes for multimodal classification networks: input fusion,
intermediate fusion (encompassing single-level fusion, hierarchical fusion,
and attention-based fusion), and output fusion. By evaluating the performance
of these fusion techniques, we provide insight into the suitability of
different network architectures for various multimodal fusion scenarios and
application domains. Furthermore, we delve into challenges related to network
architecture selection, handling incomplete multimodal data management, and
the potential limitations of multimodal fusion. Finally, we spotlight the
promising future of Transformer-based multimodal fusion techniques and give
recommendations for future research in this rapidly evolving field.

###### keywords:

Multimodality fusion, Deep learning, Medical image classification , Computer-
aided diagnosis

††journal: /

##  1 Introduction

###  1.1 Context

In recent years, the field of medical image analysis has seen a surge in
efforts to apply deep learning-based methods to the classification of various
diseases, notably related to the brain [102, 251, 126], breasts [174, 47,
173], prostate [109, 243, 143] and eyes [119, 248, 86]. The ability to
accurately classify and diagnose diseases from medical images has the
potential to revolutionize healthcare by improving diagnostic accuracy,
reducing human error, and enabling more personalized treatment planning. This
trend has highlighted the need for robust and efficient methods for analyzing
medical images across multiple imaging modalities.

With advances in medical image acquisition systems, many new imaging
modalities have been developed to diagnose patients [151, 16, 79], resulting
in larger and more diverse datasets. An imaging modality alone does not often
provide all the information needed to ensure accurate clinical diagnosis.
Therefore, clinicians increasingly base their diagnosis on images obtained
from a variety of sources: a combination of abundant information can be used
in clinical practice with more confidence. Following this trend and to improve
diagnosis results, artificial intelligence-based classification models are
increasingly being developed by combining data from multiple sources to take
advantage of both redundancies and complementarities across modalities.

Several surveys have been conducted in recent years to analyze the trends in
the application of multimodality in various fields [184, 19], among which the
field of medicine is gaining a great deal of attention. In medicine, several
such survey papers focus on specific image analysis tasks: image fusion [16],
image synthesis [234], image segmentation [259], or image registration [59].
However, medical image classification was never addressed in a comprehensive
manner. A few surveys target specific fields such as neurology [197] or
oncology [124], but they do not provide a comprehensive discussion of how
multimodal fusion may be applied to other fields. To fill this gap, we propose
to review deep learning-based information fusion techniques for multimodal
medical classification across all medical fields. We restrict this analysis to
classification, given the sufficient coverage of other analysis tasks [16,
234, 259, 59]. We include in the scope of classification methods any method
assigning class labels, possibly with probabilities, to a patient or a region
of interest in the patient, regardless of the application (diagnosis,
prognosis, risk estimation, etc.). Throughout our review, we summarize and
discuss the advantages and disadvantages of various information fusion methods
that can be applied to various organs and imaging modalities. As the first
review to examine the use of deep learning in multimodal medical
classification, this paper aims to guide future investigations into medical
diagnosis using multiple imaging modalities.

###  1.2 Traditional methods

Information fusion not based on deep learning strategies, relying on
traditional image processing and machine learning, has been reviewed in a
previous survey [99]. We summarize hereafter the main developments and
highlight the benefits of non-deep learning-based information fusion.

Input fusion is the most commonly used strategy among traditional methods. It
involves the fusion of images from various modalities into structured data and
fuses them into different categories depending on the fusion domain: spatial
fusion [59, 205, 25, 75, 21], frequency fusion [170, 162, 186, 48, 134, 233]
and sparse representation [252, 264, 133]. In spatial fusion, multimodal
images are combined at the pixel level, but this approach often leads to
spectral degradation [147] and color distortion [24]. Frequency fusion, which
involves transforming the input image into the frequency domain, is more
complex and results in limited spatial resolution [193]. Sparse
representation, on the other hand, can be sensitive to registration errors and
lacks attention to details [24].

Other strategies include intermediate and output fusion, which do not require
registration of the input images. Intermediate fusion involves extracting
features from different imaging modalities, concatenating them, and feeding
them into a classifier, generally a support vector machine (SVM), for
diagnosis [111, 213, 178]. This approach requires extensive testing and rich
domain knowledge for feature extraction and selection. On the other hand,
output fusion involves stacking the data results from unimodal models and
combining them [107]. While this approach circumvents the need for early
integration, it presents its own set of challenges. Individual models in
output fusion may be heavily influenced by their respective modality-specific
idiosyncrasies, potentially introducing biases into the final combined output.
Moreover, if these unimodal models yield correlated or redundant information,
the utility of stacking them diminishes, as it might not deliver significant
additional value.

Traditional methods typically involve complex pre-processing steps paired with
relatively simple model structures. Such a combination frequently leads to
information loss during feature extraction, thereby complicating efforts to
fully leverage the synergies between various imaging modalities.

Besides requiring domain knowledge, these traditional multimodal fusion
approaches do not fully utilize the complementarity between multimodal
features. These limitations highlight the need for more advanced techniques,
such as deep learning-based multimodal fusion methods, able to overcome the
challenges faced by traditional methods. Deep learning network architectures
offer complex models that can explore more possibilities for multimodal
fusion. Furthermore, various end-to-end models significantly reduce the amount
of domain knowledge required for diagnosis purposes, albeit at the cost of
interpretability [187].

###  1.3 Development trends

Recognizing the potential of deep learning-based methods for multimodal
medical image classification, researchers have increasingly focused on this
area. In order to obtain more accurate diagnoses, multimodal medical image
analysis have also become a growing trend. Fig. 2 shows the number of
publications about multimodal medical classification each year, which was
queried on February 27, 2024, on PubMed. As illustrated by the figure, the
number of papers has increased yearly from 2016 to 2023, indicating that
multimodal medical classification tasks based on deep learning have gained
greater attention in recent years. Furthermore, we report the number of
publications on different organs in multimodal diagnosis tasks in Fig. 3. We
found that brain-related publications currently account for a substantial
portion of multimodal studies. This is due to the disclosure of many large
multimodal image datasets on the brain. On the other hand, not many studies
were conducted on other organs, except whenever a public dataset was released.
This finding motivated us to focus the review on studies performed on public
datasets from various organs. One advantage is to allow direct quantitative
comparisons between methods.

Fig. 1: Overview and proportion of deep learning-based information fusion
techniques for multimodal medical image classification presented in this
paper. Fig. 2: Number of publications on medical multimodal image
classification. Per-year statistics obtained using PubMed from 2016 to 2023.
Fig. 3: Number of publications dealing with medical multimodal image
classification on human organs, from 2016 to 2023. Tags: organ, number of
publications, percentage.

###  1.4 Paper selection

Table 1: A list of multimodal image datasets. The list is sorted by the number
of publications on PubMed (Keywords: dataset name AND ’multimodal’). Details
about imaging modalities are given in Tab.2

Dataset |  Year |  Modalities | Body Organ(s) | Medical Diagnosis | EHR
---|---|---|---|---|---
ADNI1 |  (2004-2009) (2010-2016) (2016-2022) (2023-2027) | sMRI, fMRI, PET | Brain | Alzheimer’s Disease | Available
BraTS2 |  (2012-2023) yearly | MRI (TI, T2, T1c, FLAIR) | Brain | Brain Tumor | N/A
TCIA3 | 2014 |  CT, MRI, PET, US, etc. |  Brain, Breast, Lung, Kidney, Head-Neck, Liver, Pancreas, etc. | Common Cancer Disease | Available
OASIS4 | 2007 | MRI, PET | Brain | Alzheimer’s Disease | Available
SPC5 | 2018 |  Dsc, Clinical Image, Metadata | Skin | Skin Lesion | Available
TCGA6 | 2006 |  Pathological data, Genomic data | Brain, Lung, etc. | Common Cancer Disease | Available
ABIDE7 | 2012 | sMRI, fMRI | Brain |  Autism Spectrum Disorder (ASD) | Available
ADHD-2008 | 2011 | sMRI, fMRI | Brain |  Attention Deficit Hyperactivity Disorder (ADHD) | Available
COBRE9 | 2012 | sMRI, fMRI | Brain |  Schizophrenia | Available
GAMMA10 | 2021 | OCT, Fundus Image | Eye |  Glaucoma | N/A
CPM-RadPath11 |  2019, 2020 | MRI (TI, T2, T1c, FLAIR) | Brain |  Brain Tumor | N/A
ISIT-UMR12 | 2019 |  White Light RGB, Narrow Band Imaging (NBI) | Digestive Tract |  Gastrointestinal Lesions | N/A
MRNet13 | 2018 | MRI (TI, T2) | Knee | Knee Injuries | N/A
CTU-UHB14 | 2014 | CT (FHR, UC) | Uterus | Fetal Distress Diagnosis | Available

  * 1

https://adni.loni.usc.edu/

  * 2

http://braintumorsegmentation.org/

  * 3

https://www.cancerimagingarchive.net/

  * 4

https://www.oasis-brains.org/

  * 5

https://derm.cs.sfu.ca/Welcome.html

  * 6

https://www.cancer.gov/ccg/research/genome-sequencing/tcga

  * 7

http://fcon_1000.projects.nitrc.org/indi/abide/

  * 8

http://fcon_1000.projects.nitrc.org/indi/adhd200/

  * 9

https://fcon_1000.projects.nitrc.org/indi/retro/cobre.html

  * 10

https://aistudio.baidu.com/aistudio/competition/detail/90/0/introduction

  * 11

https://zenodo.org/records/3718894

  * 12

http://www.depeca.uah.es/colonoscopy_dataset/

  * 13

https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002699

  * 14

https://physionet.org/content/ctu-uhb-ctgdb/1.0.0/

In our initial literature search, we identified a total of 14 public
multimodal image datasets. These datasets are detailed in Sect. 2.2, with a
summary presented in Tab. 1. The methodology for finalizing the list of papers
for this review was as follows:

  1. 1.

For each of the 14 datasets, we conducted a search on PubMed for publications
that mentioned the dataset name, coupled with any of the following terms:
(multimodality), (multimodal), (multi-modal), (multiparametric), or (multi-
parametric).

  2. 2.

We then concatenated the 14 resulting lists.

  3. 3.

Based on the abstracts, we handpicked articles that addressed multimodal
information fusion through deep learning methods.

Notably, there are gaps in the availability of public multimodal datasets
focused on classification tasks for certain organs—namely, the breast, lung,
prostate, kidneys, larynx, heart, and liver, even though they are frequently
discussed in the multimodal medical image analysis literature. To ensure a
comprehensive review, we expanded our scope to include 19 pertinent articles
that target these organs but utilize private datasets. This brought our final
tally to 114 publications.

###  1.5 Highlights

Through our examination of the deep learning-based multimodal image
classification literature (overview presented in Fig. 1), we propose in this
paper an updated taxonomy for multimodal information fusion. As discussed in
Sect. 1.2 and other surveys [184, 151, 28], multimodal fusion methods are
traditionally classified as input fusion, intermediate fusion or output
fusion, based on the stage of information fusion in the classification
pipeline, as in Fig. 4(a). Note that some publications refer to input fusion
as early fusion, while intermediate fusion may be considered as feature-level
fusion, and output fusion is equivalent to decision-level fusion or late
fusion [28, 184, 119]. Our analysis points to intermediate fusion as the
prevailing category at present. To grant readers a more in-depth understanding
of multimodal deep learning networks, we further segment intermediate fusion
into single-level fusion, hierarchical fusion, and attention-based fusion, as
illustrated in Fig. 4(b). The proposed taxonomy is detailed and discussed in
Sect. 4.1: it covers the majority of the current multimodal classification
network architectures, providing insight into their stages and styles of
information fusion.

Fig. 4: (a) Unimodal classification task flow and different types of
multimodal fusion based on the level in which they perform information fusion.
(b) Information fusion networks for the three types of multimodal fusion,
inputs to information fusion, and the implementation of information fusion.

In this paper, we present the following contributions:

  * (1)

Identify the process of medical multimodal classification.
The methodological approach of deep learning-based multimodal classification
can divided into four steps: data processing, deep learning network,
multimodal information fusion, and the final classification algorithm.
Crucially, the classification of multimodal information fusion methods hinges
on the sequential positioning of these stages. Propose network architectures
apt for generic multimodal fusion classification endeavors.

  * (2)

Propose network architectures for generic multimodal fusion classification
tasks.
In order to address medical classification tasks involving different organs
and imaging modalities, we summarized five strategies of multimodal fusion:
input fusion, single-level fusion, hierarchical fusion, attention-based
fusion, and output fusion. These fusion methods can be applied to any
multimodal classification problem in medicine, allowing for greater
flexibility and potential for improved results.

  * (3)

Present the prevailing challenges and predict future trends.
While it’s apparent that multimodal fusion is still in its early stages, our
paper analyzes specific challenges tied to this domain and predicts future
trends in the field.

The remainder of the paper is organized as follows: Sect. 2 describes commonly
used multimodal data for medical multimodal classification tasks and their
publicly available datasets. Sect. 3 describes the multimodal medical image
classification task process mentioned in contribution 1. A review of papers
implementing each of the five fusion strategies of contribution 2 is presented
in Sect. 4. The purpose of Sect. 5 is to discuss the existing problems and to
make predictions regarding future fusion methods in contribution 3. Finally,
Sect. 6 contains our concluding comments. A list of frequently used
abbreviations throughout the paper is shown in Tab. LABEL:tab_terms.

##  2 Multimodal medical images

###  2.1 Imaging modalities

For medical diagnosis purposes, each imaging modality has its own
characteristics and information. Different medical imaging modalities use
different frequency bands of the electromagnetic spectrum in order to screen
and diagnose different medical conditions in the human body [16]. There are
different wavelengths and frequencies associated with each imaging modality,
as well as different characteristics (structure, function, etc.) [200].
Furthermore, medical imaging modalities can be classified as invasive or non-
invasive. Invasive methods involve inserting an object into the body through
an incision or needle injection in order to examine an organ, while non-
invasive methods utilize some form of radiation or sound [16]. Table 2 shows
some modalities that appear in multimodal medical image datasets.

Table 2: Some examples of imaging modalities and organs found in the multimodal medical image analysis literature. Modalities |  Body organs examined |  Invasive/ Non-invasive | Description
---|---|---|---
Magnetic Resonance Image (MRI) |  Brain, Prostate, Breast, etc. | Non-Invasive |  In addition to high spatial resolution and exquisite soft tissue contrast, MRI can also display dynamic physiologic changes in three dimensions [167].
Positron Emission Tomography (PET) |  Brain, Prostate, Breast, etc. | Invasive |  The PET provides information about the organs’ activity, as well as its sugar use as energy[17].
Computed Tomography (CT) |  Lung, Bone, Oral, etc. |  Non-Invasive (harmful) |  CT is an excellent tool for detecting bone, joint, and soft tissue lesions that may affect bone, joints, or soft tissues [29].
Ultrasound (US) |  Abdomen, Breast, etc. |  Non-Invasive |  In addition to showing the activity and function of certain organs in the body, US can also identify whether a tissue or organ contains fluid or gas [112].
Optical Coherence Tomography (OCT) |  Eye, Heart |  Non-Invasive |  Biological tissues can be visualized in high-resolution with OCT scanning in two-dimensional or three-dimensional modes [83].
Dermatoscope (Dsc) |  Skin |  Non-Invasive |  Dsc allows better visualization of subsurface structures and improved identification of skin diseases [139].
Color Fundus Photographs (CFP) |  Eye |  Non-Invasive |  CFP monitors the progression of eye disorders using color photographs taken with a fundus camera [23].

Due to their complementary nature, there has been a significant focus on the
following combinations of modalities targetting various diseases: (1) multi-
parametric MRI (TI, T2, T1C, and FLAIR) [50, 245, 101, 238, 230], (2) MRI and
PET [131, 62, 65, 71], (3) PET and CT [175], (4) multi-view ultrasound (US
B-mode, US color Doppler) [174, 173], (5) Color Fundus Photographs (CFP) and
Optical Coherence Tomography (OCT) [228, 119, 60], (6) Dermatoscope (Dsc) and
Clinical Images [212, 96, 225], and (7) combined diagnosis of Image Data and
Clinical Data [244, 168, 219, 132]. The complementary relationships between
these modalities will be briefly discussed.

Neurology and neurosurgery frequently use MRI. Different MRI images can be
obtained by changing the factors affecting the magnetic resonance (MR) signal,
and these different images are referred to as sequences. Depending on the
sequence used, the behavior of tumors may vary, and it is essential to use
multiple sequences to accurately determine tumor location and size [158].
T1-weighted (T1) and T2-weighted (T2) MRIs are the most common MRI sequences.
Tomographic anatomical maps can be observed with the T1 sequence, and the T2
sequence clearly shows the location and size of the lesion [123]. The Fluid
Attenuated Inversion Recovery (Flair) sequence provides better visualization
of the area around the tumor site, making it easier to detect the tumor’s
boundaries [78]. Furthermore, contrast-enhanced T1-weighted (T1c) sequences
can be used to detect intra-tumor conditions and distinguish tumors from non-
tumorigenic lesions [104]. T2 and Flair are suitable for detecting tumors with
peritumoral edema, while T1 and T1c are suitable for detecting tumors without
peritumoral edema [259].

Diffusion-weighted imaging (DWI) is another useful sequence designed to detect
the random movements of water protons. Therefore, DWI sequence is a highly
sensitive method for detecting acute strokes [169]. An increased apparent
diffusion coefficient (ADC) value with lower signals of DWI images could
reveal the fast diffusion of water molecules [194]. In addition to using
multiple sequences, co-diagnosis using structural MRI (sMRI) and functional
MRI (fMRI) is becoming increasingly popular [7, 130]. fMRI measures the small
changes in blood flow that occur with brain activity. This test can be used to
determine which parts of the brain are performing critical functions and to
determine the effects of strokes and other diseases on the brain [20].

The combination of PET and MRI, PET and CT has been recognized as a valuable
method for screening and diagnosing various diseases [31, 127, 87, 239, 12].
The PET scan is preceded by the administration of a radioactive agent to the
patient. This allows doctors to determine the metabolic processes in which the
brain tissue is involved [17]. Compared to other imaging methods such as CT
and MRI, PET has a high sensitivity and can detect lesions even if MRI/CT does
not yet show abnormalities. PET also has high specificity, making it possible
to determine whether a tumor is malignant based on its metabolism at the time
of MRI/CT detection [150]. However, because PET scan lacks information about
organ anatomy, they should be conducted in conjunction with CT/MRI scans [7].
Indeed, the combination of PET and MRI/CT scans provides structural and
functional information related to various diseases, improving the
effectiveness of diagnosis. Fig. 5 shows the images of PET, CT, and MRI, as
well as several sequences of MRI.

Fig. 5: (a)-(c) are the images of PET, CT, and MRI. (d)-(g) are the different
sequences of MRI. Images from Zhou et al. [259].

Availability, low cost, and safety make ultrasonography the most widely used
clinical diagnostic tool, with applications ranging from breast cancer
diagnosis to cervical lymph node detection. Conventional B-mode imaging is
used to examine abnormal masses in tissues, Color Doppler imaging shows the
distribution of blood vessels within tissues [268], while Strain Elastography
(SE) is a qualitative technique and provides information on the relative
stiffness between one tissue and another. For example, the combined use of
Conventional B-mode imaging and Color Doppler is common in identifying
cervical lymph nodes [4], diagnosing breast cancer [174, 173], and so forth
[137, 191, 192]. Fig. 6 shows the US images of Conventional B-mode, Color
Doppler, and Strain Elastography.

Fig. 6: Breast ultrasound images of a 37-y-old woman with fibroadenoma. (a)
Conventional B-mode, (b) Color Doppler, and (c) Strain elastography (SE) image
[116].

In the diagnosis of ophthalmic diseases, CFP and OCT are the two most cost-
effective methods [119]. These imaging modalities provide prominent biomarkers
that can be used to identify glaucoma suspects, such as the vertical cup-to-
disc ratio (vCDR) on fundus images and the retinal nerve fiber layer thickness
(RNFL) on an OCT image. A more accurate and reliable diagnosis, compared to a
single modality, is often achieved by taking both screenings in clinical
practice [228]. Fig. 7 shows the images of CFP and OCT.

Fig. 7: Images of CFP and OCT from GAMMA challenge [228].

In the diagnosis of skin cancer, a combination of dermoscopic and clinical
images is often used [212]. The clinical image is captured using a digital
camera and shows the visualized feature in different views and lighting
conditions. On the other hand, dermoscopic images provide a clear view of the
skin’s subsurface structures and are obtained using a specific skin imaging
technique in contact with the skin [67]. Fig. 8 shows examples of the
dermoscopic and clinical images.

Fig. 8: Dermoscopic and clinical image. Image from public datasets SPC [96].

In addition to multimodal image combinations, clinical information regarding
the patient’s medical history and symptoms can significantly contribute to the
diagnosis of the disease. These data may contain implicit features that may
improve the model’s classification performance. Electronic Health Records
(EHR) are commonly used to detect brain diseases by integrating image analysis
features [168, 219]. Similarly, skin cancer detection also relies heavily on
metadata [212, 244].

###  2.2 Multimodal image datasets

In multimodal medical diagnosis, multimodal datasets are particularly valuable
for testing various networks and developing fusion methods. However, the
privacy and cost of medical images often make obtaining more comprehensive
multimodal datasets challenging for researchers. Fortunately, there are
several freely available multimodal datasets. These datasets provide
information regarding the diagnosis of diseases at various locations in the
body, as well as the analysis of various multimodal combinations. These
datasets are expected to contribute to the analysis of fusion methods and
serve as a foundation for the future development of multimodal fusion methods.

Alzheimer’s Disease Neuroimaging Initiative (ADNI) is a multi-center
longitudinal study to discover clinical, imaging, genetic, and biochemical
biomarkers for Alzheimer’s disease (AD). ADNI has three stages: ADNI 1
included 400 subjects diagnosed with mild cognitive impairment (MCI), 200
subjects with early AD, and 200 elderly control subjects [166]; ADNI 2 added
new participant groups: 150 elderly controls, 100 EMCI subjects, 150 late mild
cognitive impairment (LMCI) subjects, and 150 mild AD patients [22]; ADNI 3
added hundreds of new MCI subjects, mild AD subjects, and elderly controls
[226].
The MRI Brain Tumor Segmentation (BraTS) challenge has been held since 2012
and currently includes classification tasks in addition to tumor segmentation
[144]. Each subject has four MRI modalities (T1, T1C, T2, and T2 FLAIR), human
annotation of tumor segmentation, and tumor grade.
The Cancer Imaging Archive (TCIA) is a large-scale public database containing
medical images of common tumors (lung cancer, prostate cancer, etc.) and
corresponding clinical information (treatment protocol, genetics, pathology,
etc.) [41].
Open Access Series of Imaging Studies (OASIS) seeks to make neuroimaging
datasets freely accessible to the scientific community [141]. OASIS-3 contains
755 cognitively normal adults and 622 individuals at various stages of
cognitive decline ranging in age from 42-95 years [108].
Seven-point Criteria Evaluation Database (SPC) provides a database for
evaluating computerized image-based prediction of the 7-point malignancy
checklist for skin lesions. The dataset contains more than 2000 clinical and
dermoscopy color images and structured metadata for training and evaluating
computer-aided diagnosis systems [96].
As part of the Cancer Genome Atlas (TCGA), an internationally recognized
cancer genomics project, more than 20,000 primary cancer samples and matched
normal samples were molecularly characterized [227, 214].
The Autism Brain Imaging Data Exchange (ABIDE) initiative now includes two
large-scale collections, ABIDE I and ABIDE II, whose ultimate goal is to
facilitate discovery science and comparative analysis across samples. ABIDE I
contains 1112 datasets, including 539 from individuals with ASD and 573 from
typical controls (ages 7-64 years, median 14.7 years across groups) [53].
ABIDE II contains 1114 datasets from 521 individuals with ASD and 593 controls
(age range: 5-64 years) [52].
ADHD-200 Sample is a grassroots initiative that aims to improve scientific
understanding of the neural basis of ADHD through the implementation of open
data sharing and discovery-based research methods [42].
The Center for Biomedical Research Excellence (COBRE) is providing raw
anatomical and functional magnetic resonance imaging data from 72 patients
with schizophrenia and 75 healthy controls (ages ranging from 18 to 65 in each
group) [32].
The Glaucoma Grading from Multimodality Images (GAMMA) Challenge is intended
to facilitate the development of fundus and OCT-based glaucoma grading [229].
GAMMA contains 2D fundus images and 3D OCT images of 300 patients.
Computational Precision Medicine: Radiology-Pathology Challenge on Brain Tumor
Classification 2020 (CPM-RadPath) is a brain tumor classification challenge.
There are 221 cases in the training dataset, each with a paired radiology and
digital pathology image. Within the 221 cases, there are 54, 34, and 133 cases
for lower grade astrocytoma, IDH-mutant, oligodendroblioma, 1p/19q codeltion,
and glioblastoma and diffuse astrocytic glioma with molecular features of
glioblastoma, IDH-wildtype, respectively [81, 105]. The CPM-RadPath 2020
challenge also contains 35 and 73 validation and testing sets, respectively.
Each patient contains multiple MRI sequences: T1, post-contrast T1-weighted
(T1Gd), T2, and FLAIR.
ISIT-UMR is a dataset for the classification of gastrointestinal lesions in
regular colonoscopy. The dataset consists of 76 polyps with white light and
NBI videos from the same polyp [145].
The MRNet dataset consists of 1,370 knee MRI exams performed at Stanford
University Medical Center between January 1, 2001, and December 31, 2012.
There were 1,104 (80.6%) abnormal exams in the dataset, with 319 anterior
cruciate ligament (ACL) tears and 508 meniscal tears [27].
CTU-CHB Intrapartum Cardiotocography is a database containing 552 cardiac
tomography recordings from the Czech Technical University (CTU) in Prague and
the University Hospital in Brno (UHB). As part of each CT, a fetal heart rate
time series (FHR), as well as a uterine contraction (UC) signal, are recorded
[40].

The previously mentioned datasets provide valuable resources for developing
and testing multimodal fusion methods. They contain images of different
medical modalities of the same patient, as well as images of different
patients. Access to these datasets is available upon request and at no cost.
In this review, we summarize the fusion methods presented in 53 articles that
use ADNI, 11 articles that use TCIA, 7 articles that use BraTS (2015, 2017,
2019 and 2021 editions), 7 article that uses OASIS, 4 articles that use COBRE,
4 articles that use SPC, 4 articles that use ABIDE, 3 articles that use
ADHD-200, 2 articles that use CPM-RadPath (2020 edition), 2 articles that use
GAMMA, 2 articles that use MRNet, 1 article that uses TCGA, 1 article that
uses CTU-UHB and 1 article that uses ISIT-UMR. As mentioned earlier, 19 papers
discussed in this review are not based on public datasets.

##  3 Multimodal classification pipeline

Multimodal fusion of biomedical data using deep learning remains an evolving
field. The terminology used to describe fusion methods often varies between
publications, leading to ambiguity. For instance, terms like input,
intermediate, and output fusion are commonplace, but their interpretations may
differ. To bring clarity and standardization to the multimodal classification
area, we adopt the five-stage pipeline proposed in Sleeman IV et al. [201],
referenced in Tab. 3. This pipeline offers a structured approach to
encapsulate all medical multimodal classification tasks. Within this section,
we elucidate each of these stages, detailing their definitions and the
methodologies for their implementation. Subsequently, based on the sequence
and structure of the information fusion stage paired with the deep learning
(DL) backbone stage, we categorize multimodal fusion techniques into five
distinct strategies in Section 4.

Table 3: Multimodal classification pipeline. Stage | Description
---|---
Data preprocessing |  The initial step of the classification task is to perform operations such as registration, denoising, and data augmentation on the raw data.
DL backbone |  Extraction of high-dimensional features of data by the deep learning network structure.
Information fusion |  Fusion of multimodal data/features by different methods.
Final classifier |  The final stage of generating classification results from multimodal data.
Model evaluation |  Different metrics are used to evaluate the performance of multimodal models.

###  3.1 Pre-processing

Image pre-processing is crucial for multimodal medical classification tasks,
as it enhances DL network efficiency and effectiveness in extracting features.
Pre-processing techniques, such as image registration, cropping, denoising,
resampling, intensity normalization, regions-of-interest (ROI) extraction [44,
49, 100], and feature selection [129, 128, 7, 174], prepare the data for more
accurate and efficient analysis by DL models.

To further improve the performance of these models, data augmentation
techniques play an essential role in the pre-processing pipeline. For example,
data augmentation helps prevent overfitting [223] using methods like random
cropping, flipping, and rotation during training. In addition, increasing the
training dataset’s diversity improves the model’s generalization capabilities.

Considering the large volumes of data generated by multimodal medical images,
it is noteworthy that only a small fraction is relevant to diagnosing
diseases. Therefore, feature selection emerges as a crucial pre-processing
step, aiming to reduce data dimensionality while retaining pertinent
information. Common feature selection methods include manual selection [266,
196, 98] and Principal Component Analysis (PCA) [114, 61, 258].

Another critical aspect of pre-processing multimodal medical images is image
registration. It involves aligning images from different modalities (e.g.,
MRI, CT, and PET) into a common coordinate system, enabling the accurate
matching of corresponding anatomical structures across image types [15, 59].
Such alignment facilitates comprehensive data analysis and becomes
particularly critical for input-level fusion, where combining complementary
information from different modalities requires proper alignment [119].
Image registration in this context presents several challenges. A significant
one is the lack of ample training datasets for supervised deep learning.
Another is defining accurate similarity measures, especially with the varied
appearance of different modalities. The registration process can be further
complicated when trying to align images from different patients or even the
same patient over time due to factors like changes in anatomy and metabolic
processes.
In tackling these challenges, deep similarity metrics have shown promise,
especially in traditional frameworks. While multimodal registration has seen
advancements, direct transformation prediction lags behind, especially
compared to single-modality methods. One innovative solution is using
Generative Adversarial Networks (GANs) to make multimodal images more
consistent.

###  3.2 Information fusion

A key component of multimodal image classification is information fusion.
Based on the level at which information is fused, information fusion can be
divided into input fusion, intermediate fusion, and output fusion. And there
are two ways to achieve fusion [201], namely concatenation and merge.
Concatenation involves the concatenation of data from different modalities
into a single tensor for the next step. Merge involves complex calculations
such as adding data from different modalities, and the final result is a
smaller amount of data. Fig. 9 illustrates the two types of fusion. Our study
focuses on the fusion of different medical imaging modalities, and in Sect. 4,
we will examine the different fusion methods in greater detail.

Fig. 9: Two types of fusion. Orange and green: data of different modalities.
Blue: the output fused data.

###  3.3 Deep learning backbone

DL backbones are used to extract high-dimensional features of modalities
during the classification process. Over recent years, several high-performing
network architectures have emerged, including AlexNet [103], VGG [198],
GoogLeNet [209], ResNet [76], DenseNet [84], AE [208, 206, 242], ViT [236],
and others, providing state-of-the-art performance in classification. A
summary of the common architectures for DL is presented in Tab. 4. DL has
developed rapidly due to several factors, including the development of
hardware devices like graphics processing units (GPUs) and tensor processing
units (TPUs), which have greatly improved the training speed of DL networks.
Additionally, publicly available datasets such as ImageNet [51] have
facilitated the training and testing of various models. Furthermore, DL is
capable of learning advanced features directly from data without requiring
extensive expertise or prior experience, making it easily adaptable across
various domains.

Table 4: Some common architectures of deep neural networks. Different architectures are more suitable for different types of data. Architecture | Description
---|---
Fully Connected Neural Network (FCNN) |  FCNN are the most traditional deep neural networks. Every neuron in a layer is connected to every neuron in the layer below it [68].
Convolutional Neural Network (CNN) |  CNN can model spatial structures, such as images or volumes. Convolutional kernels model local information by sliding over input data [68].
Autoencoders (AE) |  By compressing and reconstructing the input data, AE learns low-dimensional encoding. There are different types of layers, such as convolutional and fully connected [18].
Transformer |  Transformer is a model that uses a multi-headed attention mechanism. Feature extraction is solely based on attention [217].

In input fusion, a single backbone can extract features from fused modalities.
However, in other fusion schemes such as intermediate or output fusion,
multiple DL backbones may be used to extract features from different
modalities. In current multimodal fusion research, Convolutional Neural
Networks (CNN) are the preferred choice of the majority of researchers due to
their effectiveness in feature extraction from medical images. Many pre-
trained models have already been tested on large datasets, making them
suitable for use in medical imaging research. In the articles analyzed, CNNs
were used in 65 articles, Fully Connected Neural Networks (FCNN) in 10
articles, Auto-Encoders (AE) in 8 articles, and Transformers in 6 articles.

###  3.4 Final classifier

Multimodal classification employs a final classifier to generate the
classification results based on multimodal features or multiple independent
classification results, depending on the employed fusion scheme. In DL
networks, the Fully Connected (FC) layer [7, 175, 130, 77, 119, 60] is often
used as the final classifier. Other methods, such as SVM [114, 207], Random
Forest [47], and Score Merge [222, 82] can also be used as final classifiers.

###  3.5 Evaluation metrics

Evaluation metrics for multimodal fusion tasks are similar to those used in
unimodal classification tasks. Commonly used indicators for assessing the
performance of multimodal fusion methods and DL networks in the context of
medical classification tasks include True Positive (TP), True Negative (TN),
False Positive (FP), and False Negative (FN). These indicators can be used to
calculate several performance metrics, such as sensitivity, specificity,
accuracy, precision, and F1 score, among others. Additionally, AUC and Kappa
are commonly used metrics to evaluate medical classification tasks.

  * -

Accuracy (ACC) = $\frac{TP+TN}{TP+TN+FP+FN}$

  * -

Sensitivity (SEN) = $\frac{TP}{TP+FN}$

  * -

Specificity (SPEC) = $\frac{TN}{TN+FP}$

  * -

F1 Score = $\frac{2\times TP}{2\times TP+FP+FN}$

  * -

Positive Predictive Value (PPV) = $\frac{TP}{TP+FP}$

  * -

Negative Predictive Value (NPV) = $\frac{TN}{TN+FN}$

  * -

Area Under the receiver operating characteristic Curve (AUC)

  * -

Cohen’s Kappa (Kappa) = $\frac{p_{0}-p_{e}}{1-p_{e}}$

where $p_{0}$ is the accuracy and $p_{e}$ the sum of the products of the
actual and predicted numbers corresponding to each category, divided by the
square of the total number of samples.

##  4 Multimodal classification networks

###  4.1 Information fusion taxonomy for multimodal image classification

The positions of pre-processing and the final classifier are fixed during the
process of multimodal classification. Based on the number and sequence of DL
backbones and information fusion step, multimodal DL network architectures can
be categorized into five types: input fusion, single-level fusion,
hierarchical fusion, attention-based fusion, and output fusion, as shown in
Fig. 10. As explained hereafter, single-level, hierarchical, and attention-
based fusion are sub-categories of intermediate fusion. These categories
describe how the network processes and combines the input modalities to
produce classification results.

Fig. 10: Five types of multimodal fusion networks.

(1) Input fusion can also be referred to as input-level fusion, where the
information fusion phase precedes the DL backbone. Concatenation and Merge are
two methods of information fusion. For the concatenation method, data of
different modalities are used as different channels of the input. In the merge
approach, data is fused at the pixel or voxel level, and the merged images are
used as inputs for the DL classifier. The process diagram for input fusion is
shown in Fig. 11.

Fig. 11: Input fusion process diagram. Information fusion: Concatenation/Merge
(Inputs).

(2) Single-level fusion involves information fusion after the DL backbone but
before the final classifier. As part of a single-level fusion, the features
extracted by the DL backbone are fused only once at some point before the
classifier is applied. It can be divided into two types: Classic Fusion and
Network Fusion, depending on the network structure. In Classic Fusion, high-
dimensional features are extracted from different modalities using different
DL classifiers and then merged or concatenated. This is the most common
network structure in intermediate fusion, so we call it Classic. Fig.12
illustrates the process diagram of classic fusion. In network fusion, the
intermediate features of different modalities are first extracted using DL
classifiers, followed by the extraction of high-level features of the fused
modalities using additional DL backbones. Fig.13 shows the process diagram of
network single-level fusion.

Fig. 12: Single-level fusion process diagram. Information fusion:
Concatenation/Merge (Classic). Fig. 13: Single-level fusion process diagram.
Information fusion: Concatenation/Merge (Network).

(3) Hierarchical fusion is an improvement over single-level fusion. In this
approach, DL backbone extracts features from the data of different modalities,
while features from each level are then fused at the network level by
concatenation or merging. Additionally, further feature fusion is performed
following the DL backbone. This allows for more complex feature combinations
to be learned, improving classification accuracy. The process diagram for
output fusion is shown in Fig. 14.

Fig. 14: Hierarchical fusion process diagram. Information fusion 1:
Concatenation/Merge (Network). Information fusion 2: Concatenation/Merge
(Classic).

(4) The emergence of Transformers has led to the development of Attention-
based fusion as a new network architecture. Through its unique DL backbone,
this architecture is able to extract features and implement feature fusion
based on the attention relationship between different modalities. Fig. 15
illustrates the process of attention-based fusion. A more detailed analysis of
the network architecture is presented in Sect. 4.5.

Fig. 15: Attention-based fusion process diagram.

(5) Output fusion, also known as decision-level fusion or late fusion,
involves the use of DL backbones to extract high-dimensional features from
different modalities of data. The extracted features are then used to generate
separate classification results for each modality. These results are then
combined using a fusion technique, such as majority voting or averaging, to
produce a final classification result. The process diagram for output fusion
is depicted in Fig. 16.

Fig. 16: Output fusion process diagram. Information fusion: Merge (Outputs).

Recent years have seen a growing trend toward the use of deep learning
networks in multimodal fusion research. Fig. 1 illustrates the distribution of
five fusion strategies in the scope of the study. In contrast to traditional
methods, single-level fusion is the most commonly used method in DL multimodal
fusion, followed by input and output fusion. Hierarchical fusion and
attention-based fusion are also gaining attention and present great potential
for research. These more recent fusion methods offer more complex ways of
combining modalities, enabling deep learning networks to learn more powerful
representations of multimodal data.

###  4.2 Input fusion networks

Input fusion combines data from multiple modalities into a single feature
tensor fed into the deep neural network as an input. Input fusion typically
involves the fusion of modalities with similar structures, making
implementation relatively straightforward. Some modalities can be acquired
together at the time of clinical photography (e.g., CT and PET). In many
cases, these modalities have the same voxels and spacing after data
processing, making obtaining registered multimodal data easy. Furthermore, the
majority of input fusion tasks do not require re-modeling, only modifying the
input part of the unimodal model to achieve multimodality. Fusion can be
accomplished in three ways: concatenating or merging multimodal medical
images, extracting high-dimensional features from multimodal images, and then
fusing them.

(1) The registered multimodal data are fed into the DL classifier as input for
different channels to obtain classification results, which is the most common
input fusion approach. Fig. 17 illustrates this typical input fusion network
architecture used in the research of Aldoj et al. [10], Lin et al. [122], Zong
et al. [265], Zhou et al. [262]. Aldoj et al. [10] proposed a semi-automatic
method for the classification of prostate cancer without feature selection.
Several combinations of 3D volumes (e.g., ADC, DWI, and T2) are utilized as
inputs of the CNN network. Each sequence is considered an input channel; the
output is the classification of significant versus nonsignificant lesions. Lin
et al. [122] employed MRI and PET to diagnose Alzheimer’s disease. PET and MRI
are used as two channels for the input of the CNN classification network,
based on an ROI crop model to learn a classifier and fuse different features
from MRI and PET. Zong et al. [265] concatenated T2, ADC and DWI for tumor
foci classification using an end-to-end CNN network. In order to diagnose
triple-negative breast cancer, Zhou et al. [262] concatenated manually
segmented multiparametric MRI images (PEI, DWI) into a CNN network. Despite
the ease of implementing this fusion architecture, it has some limitations
with regard to the modal data requirements. For instance, the registration
performance of different modal data can influence the classification results.
Moreover, this approach is not suitable for fusing heterogeneous data, such as
3D medical images and 1D clinical records, which have different
characteristics and dimensions.

Fig. 17: Schematic diagram of the network architecture for input fusion.
Information fusion method: Concatenation (Inputs).

(2) The merging of images is another input fusion method in addition to
concatenation. Various image modalities are fused at the pixel or voxel level
in order to create a new fused image that is used for classification [202,
102, 183]. Song et al. [202] proposed an effective multimodal image fusion
method for Alzheimer’s disease diagnosis using MRI and PET. Through
registration and mask coding, they were able to fuse gray matter (GM) and
18-fluorodeoxyglucose positron emission tomography (FDG-PET) images to create
a new imaging modality called ”GM-PET”. In the resultant composite image, the
GM area is clearly highlighted, allowing AD diagnosis to be made while
maintaining both the contour and metabolic characteristics of the subject’s
brain tissue. They then fed the fused images to the CNN for classification.
The GM region cropped from the MRI image is mapped onto the PET image,
resulting in the fusion of PET and MRI data in Kong et al. [102] research. In
addition to providing anatomical and metabolic information about the brain,
the fusion modality also allows the viewer to focus on the main features of
the brain by reducing the visual noise. Rallabandi and Seetharaman [183]
employed a fusion approach integrating images from MRI and PET for the
diagnosis of Alzheimer’s disease. The fusion process involved applying two-
dimensional Fourier and discrete wavelet transform (DWT) to combine MRI and
PET images. Subsequently, the MR-PET fused image was reconstructed using
inverse Fourier and DWT methods. The benefit of fused images is that they
contain a wealth of medical information, but the process of generating them
often requires an extensive amount of prior medical knowledge.

(3) Some studies have performed input fusion after extracting features from
multimodal images instead of performing a direct fusion of medical images
[114, 131]. Li et al. [114] used PCA to extract features from MRI, PET, and
cerebrospinal fluid (CSF) and then concatenated these features into the
Restricted Boltzmann Machine (RBM) network for the diagnosis of Alzheimer’s
disease. Liu et al. [131] manually extracted features from MRI and PET and
then used stacked auto-encoder (SAE) to classify the concatenated multimodal
features in order to diagnose Alzheimer’s disease. The architecture of
extracting features and combining them can solve the problem of multimodal
heterogeneity. However, PCA-based or manual feature extraction requires prior
knowledge and does not fully utilize image information.

In input fusion, fused data is used in single-branch feature extraction, and
the network architecture design significantly reduces network parameters and
deployment difficulties. However, due to the fusion of the data at the input
level, the complementary information from the different modalities is not
utilized to the fullest extent possible.

###  4.3 Single-level fusion networks

The single-level fusion process uses different DL backbones to extract
features from different modalities separately, followed by an information
fusion process before making the final decision. Based on the position of
information fusion within the network architecture, it can be divided into
classic fusion structures and network fusion structures.

(1) The most common single-level fusion architecture is to extract features
from multimodal data by using different branches, then fuse these features and
feed them to the final classifier [208, 207, 206, 241, 266, 243, 92, 245, 171,
244, 180, 237, 175, 132, 101, 93, 190]. A schematic diagram of its network
architecture is shown in Fig. 18. After preprocessing the data, the
architecture [266] extracted low-level 3D features from fMRI and sMRI to
classify Attention Deficit Hyperactivity Disorder (ADHD) automatically. As
soon as the features are concatenated, softmax classifiers are used to
differentiate ADHD cases from typically developing children (TDC) cases. In
order to diagnose breast cancer, Joo et al. [92] fused MRI (T1, T2) and
clinical information. Two 3D ResNet-50 were used to extract features from
contrast-enhanced T1 subtraction MR images and T2 MR images, while the FC
layer provided clinical inputs. For the prediction of pathological complete
response, the outputs of each 3D ResNet-50 and FC layer were concatenated, and
the final FC layer with sigmoid activation function was used. Likewise, Yap et
al. [244] employed ResNet and FC layers to extract features from DSC, Clinical
Image, and metadata, then applied FC layers for skin lesion classification.
Aside from these methods of concatenating modal features, complex computations
can also be used to merge features. Xiong et al. [237] used visual field (VF)
and OCT for the diagnosis of glaucoma. VFNet and OCTNet were used to extract
features from the VF and OCT modes, respectively. A weighted average was used
to obtain an aggregated representation from bimodal features using an
attention module. Each modal feature was assigned a weight using a fully
connected layer, followed by a sigmoid function to calculate a scalar value
(0-1) indicating the feature’s relative contribution to the aggregate
representation. To aggregate all features, a global average pooling layer was
also used. The results of glaucoma diagnosis were predicted using three FC
layers and a softmax layer. For CT and PET modalities, Qin et al. [175]
extracted features using CNN networks, merged the features using gated
multimodal units (GMU), and classified lung cancer using FC layers. GMU,
unlike the widely used connection operation, allows for the learning of
intermediate representations of multimodality features using hidden structures
and gate controls, thus enabling the prediction layer to assign weights more
effectively to intrinsically associated features.

Fig. 18: Schematic diagram of the network architecture for classic single-
level fusion. Information fusion method: Concatenation (Classic).

(2) Two stages can be described as the single-level fusion architecture for
network fusion. The first stage involves extracting single-level features
separately from different modalities using DL backbones, followed by the
second stage of information fusion which involves utilizing an additional DL
backbone to extract high-level features from the fused features [196, 260, 38,
98, 181, 91, 113, 138]. Lastly, the extracted high-level features are used in
the final classification process. Fig. 19 illustrates a typical network fusion
architecture. Cheng and Liu [38] used cascaded CNN for the multimodal fusion
of MRI and PET to diagnose Alzheimer’s disease. They proposed a 2D CNN to
combine the multimodality features and make the final classification. After 3D
CNN output features are flattened to one dimension, the 1D feature vectors of
MRI and PET are combined to produce a two-dimensional feature map for 2D CNN
analysis. Kim and Lee [98] developed a multimodal architecture for combining
MRI, PET, and CSF features. Each modality’s individual representation of high-
level features is calculated using the stacked sparse extreme learning machine
auto-encoder (sELM‐AE). Another stacked sELM‐AE is used to get the joint
features from the high‐level MRI, PET, and CSF features. The kernel-based
extreme learning machine classifies the joint feature representation. With
multimodality neuroimaging and genetic data, Zhou et al. [261] proposed a
three-stage deep feature learning DNN framework for Alzheimer’s disease
classification. Each modality’s latent representation is learned in the first
stage, then each pair of modalities’ joint latent representations are learned
in the second stage, and in the third stage, each pair of modalities’ joint
latent representations are used to create the classification model. Rahaman et
al. [181] classified schizophrenia using sMRI, fMRI, and single nucleotide
polymorphisms. The latent representations for the static functional network
connectivity (sFNC), sMRI, and single nucleotide polymorphism (SNP) are
learned using an autoencoder, multi-layered perceptron, and bi-directional
long short-term memory (LSTM). The Multimodal Bottleneck Attention Module
performs the fusion of the embeddings and then sends the combined embeddings
to a variational autoencoder for encoding, followed by a Softmax layer for
classification.

Fig. 19: Schematic diagram of the network architecture for single-level
network fusion. Information fusion method: Merge (Network).

The single-level fusion method is currently used to merge multiple medical
modalities for classification tasks and can be applied to the fusion of
different medical modalities. The method does not require a specific format
for the data as it extracts features from modalities using different branches
and fuses data at a high-dimensional feature level. In this regard, single-
level fusion is a suitable solution for unregistered or different dimensional
data. Due to the fact that information fusion occurs only at the end of the
network architecture, single-level fusion is not capable of analyzing low-
dimensional features jointly.

###  4.4 Hierarchical fusion networks

Hierarchical fusion extends single-level fusion further in order to further
exploit the complementary information between multimodal data. The
hierarchical fusion process involves the fusion of different dimensional
features and the classification of these jointly represented features through
the process of fusion [140, 253, 258, 77, 119, 230, 157, 240, 215, 146, 238].
There are two ways to implement hierarchical fusion: by using additional
branches for multimodal feature fusion or by using fusion blocks for joining
features from different modalities.

(1) The common hierarchical fusion architecture involves extracting different
modalities via different branches and simultaneously combining multimodal
features of different dimensions via another parallel branch. Finally, the
high-dimensional features from the fusion branch and each modal branch are
combined for classification. Fig. 20 shows a typical network architecture for
hierarchical fusion, Zhou et al. [258], Zhang and Shi [253], Li et al. [119]
utilized this network architecture. Zhou et al. [258] utilized three sparse-
response Deep Belief Network (DBN) branches to extract features from PET/MRI
modalities, fuse them, and then employed an Extreme Learning Machine (ELM) to
classify the fused features for brain diseases. Zhang and Shi [253] used a
deep multi-modal fusion network (DMFNet) to fuse PET and MRI data for the
diagnosis of Alzheimer’s disease. Three branches are present in DMFNet, two of
which extract features from the MRI and PET scans, respectively. A channel
attention model is used to extract the features from each branch and merge the
reweighted feature maps. In the third branch, fused features are further
extracted. Li et al. [119] used a three-branch CNN network to combine 2D
fundus images with 3D OCT images in order to classify glaucoma and diabetic
retinopathy. The fusion of 2D and 3D data features on the fusion pointers was
achieved by changing the dimensionality of the data features using a
transformation layer.

Fig. 20: Schematic diagram of the network architecture for hierarchical
fusion. Information fusion method: Merge (Network) and Concatenation
(Classic).

(2) Hierarchical fusion can also be structured in another way by extracting
features using different branches and fusing them in different dimensions by
using fusion blocks, which are then returned to each modality branch for
further fusion. The design of such a network structure can reduce the number
of model parameters while fusing features at multiple levels. Fig. 21
illustrates a typical network architecture, Gao et al. [64], He et al. [77]
utilized this network architecture. To classify brain diseases, Gao et al.
[64] proposed a pathwise transfer deep convolution network that gradually
learned and combined the multi-level and multimodal features of MRI and PET.
The pathwise transfer blocks are designed to fully utilize complementary
information from different imaging modalities. Pathwise transfer blocks are
used to communicate information across PET and MRI, which helps to improve the
classification model’s performance. He et al. [77] proposed a multimodal MRI
hierarchical-order multimodal interaction fusion network (HOMIF) to diagnose
gliomas. There are two branch networks for each modality, several multimodal
interaction modules with different scales and orderings, diverse learning
constraints, and a predictive subnet in the framework. Each branch network has
three CNN blocks with multiscale inputs and an arm with diverse high-order
multimodal interaction (HOMI) modules to integrate and interact deeply with
the multiscale features.

Fig. 21: Schematic diagram of another network architecture for hierarchical
fusion. Information fusion method: Merge (Network) and Concatenation
(Classic).

The multi-level feature fusion allows hierarchical fusion to explore more
fully the complex and complementary information between modalities. Learning
the synergy of multimodal data while maintaining the features of the
modalities improves the model’s classification performance [253]. However, as
it involves the fusion of low-dimensional features, the registration of
multimodal data may affect the classification performance of hierarchical
fusion.

###  4.5 Attention-based fusion networks

As attentional mechanisms [218] have been proposed and developed, more and
more studies are beginning to incorporate attentional mechanisms into network
architectures. Some of the network architectures mentioned above also included
attention mechanisms in order to enhance the performance of the models. Zhang
and Shi [253] added attention modules to reweight the modal features. Xing et
al. [236] use a vision transformer (ViT) to extract the modal features and
fuse them. These studies, however, only operate on unimodal modalities and do
not utilize the attention mechanism for multimodal interactions. Recently,
some studies have used the attention mechanism to extract and combine features
[45, 176, 255, 125, 118, 46, 267, 34, 65, 26, 225]. This network architecture
is called attention-based fusion, which is not related to any of the previous
fusion architectures.

In the study of Dai et al. [45], they propose TransMed, which combines CNN and
transformer to capture high-level cross-modalities and low-level features.
First, TransMed sends the multimodal images to CNN, where they are processed
as sequences, then transformers learn the relationships between them and
predict the end result. TransMed is more efficient and accurate than existing
multimodal fusion methods because it effectively models the global features of
multimodal images.

Attention-based Hierarchical Multimodal Fusion (AHM-Fusion) is a novel fusion
module Qiu et al. [176] designed. The system includes both an early feature
guidance module and a late feature fusion module, capturing deep interaction
information between different multimodal features. In the early stage of
feature aggregation, the early feature guidance module is used to capture
multimodal interactions. To obtain classification results, late feature fusion
modules based on attention mechanisms are used. Through cascading double
attention layers in the late feature fusion module, the deep interaction
information is further captured. Then, they used a gating-based attention
mechanism to decrease the impact of insignificant features in each modality.

Zhang et al. [255] proposed a multimodal Medical Information Fusion (MMIF)
framework that combines the Category Constrained-Parallel ViT framework
(CCPViT) and the multimodal Representation Alignment Network (MRAN) as
backbones, enabling the modeling of images and texts as unimodal features, as
well as cross-modal features. CCPViT is proposed as a tool for learning key
features of different modalities and for solving unaligned multimodal tasks.
hen in MRAN, Cross-attention was used to cascade encoded images and decoded
texts to explore deep-level interactive representations of cross-modal data,
assisting with modal alignment and identifying abnormalities. MMIF is an
image-text foundation modeling that could contribute to a much higher-
precision classification model when compared with unimodal models.

Multimodal Mixing Transformer (3MT) was presented Liu et al. [125] as a novel
technique to classify diseases. Based on neuroimaging data, gender, age, and
the Mini-Mental State Examination (MMSE), They tested it for Alzheimer’s
Disease classification. Multimodal information is incorporated through a
Cascaded Modality Transformers architecture with cross-attention. Different
embedding layers are used to obtain Key (K) and Value (V) from imaging
features and clinical data. K and V are then placed into a cross-attention
layer with a latent code known as Query (Q). 3MT allows mixing an unlimited
number of modalities and formats and full data utilization.

Zuo et al. [267] has introduced a novel Swapping Bi-Attention Mechanism (SBM)
designed for the diagnosis of Alzheimer’s disease through the amalgamation of
structural-functional brain images. The proposed model capitalizes on the
transformer’s bi-attention mechanism to explore mutually beneficial
information inherent in both structural and functional images. Historically,
transformers have been investigated solely within the confines of single-
modality brain regions, neglecting the potential for leveraging complementary
information across modalities. SBM, however, implements token exchange between
the two modalities and adaptive fusion of intermediate features during the
application of the Transformer for feature extraction from diverse modalities.
This process facilitates the collaborative exchange of information embedded in
bimodal images, enhancing the transparency of feature alignment and fusion.
The incorporation of SBM results in a more lucid understanding of the
influence of different modalities on feature extraction and multimodal fusion.

Research is increasingly incorporating attention mechanisms, particularly
Transformer structures, into multimodal classification tasks. While performing
cross-modal attention computation, a multi-level fusion of multimodal features
is achieved. Furthermore, the Transformer structure is well-suited for joining
modalities of different dimensions. Nevertheless, Transformer research in
medical tasks is still in its infancy, and various studies tend to focus on
solving particular problems, making it difficult to conclude a general
multimodal classification architecture. A further important point to be noted
is that while the success of Transformer is accompanied by pre-training on
large datasets, the number of samples in medical datasets is often not
sufficient to achieve the good training effect of Transformer. As a result, it
is recommended that Transformer and CNN are used together in a hybrid fashion.

###  4.6 Output fusion networks

In output fusion, each modality uses a separate DL backbone to extract
features and make decisions, and the results are merged into one final
decision. Fig. 22 shows a typical network architecture for output fusion. The
final Classifier of decision fusion can be achieved by simple operations [149,
72] such as voting, weighting, and averaging, or by classifiers [5, 62, 248,
106, 177] such as SVM, extreme gradient boosting (XGBoost), adaptive boosting
(AdaBoost), Categorical Boosting (CatBoost), Decision Tree, and K-nearest
neighbor (KNN). Moon et al. [149] used unweighted average, weighted average,
weighted voting, and stacking to fusion the classification results from
different modalities of the US to identify breast tumors. Guo et al. [72]
applied a linear weighted module to assemble the predicted probabilities of
the pre-trained models based on the 4 MRI modalities for the classification of
gliomas. In order to achieve the diagnosis of early glottic cancer, Kwon et
al. [106] used decision trees to combine the classification results from the
sound data and the image data. Abdolmaleki and Abadeh [5] used SVM, KNN, and
linear discriminant analysis (LDA) to fuse the classification results of fMRI
and sMRI to diagnose ADHD.

Fig. 22: Schematic diagram of the network architecture for output fusion.
Information fusion method: Merge (Outputs).

The output fusion process involves combining unimodal results from different
modalities. As a result, it is relatively easy to implement and generally does
not require additional training. It is, however, difficult to exploit the
complementary information between different modalities because there is no
feature fusion. Furthermore, output fusion may not improve classification
performance if there are large differences in classification performance
between different modalities.

##  5 Discussion

###  5.1 Which fusion method is the best?

The choice of a fusion method is crucial when dealing with multimodal medical
classification problems. Fortunately, many fusion architectures have been
evaluated on the same dataset: ADNI. Based on quantitative results reported by
the authors, comparisons between fusion architectures is possible to some
extent. We consider studies performed on ADNI where MRI and PET are used to
diagnose Alzheimer’s disease. There are three stages in the progression of
Alzheimer’s disease: normal cognition (NC), mild cognitive impairment (MCI),
and Alzheimer’s disease (AD). In spite of the fact that MCI does not
significantly interfere with daily activities, a high risk of AD progression
has been consistently demonstrated in patients with MCI [58]. MCI subjects can
be classified into MCI converters (cMCI) and MCI non-converters (ncMCI) to
predict the transition risk of MCI. Tab. 5 reports the results of the
different fusion methods obtained by their authors for different
classification tasks. The experiments were not replicated: when comparing
these results, it should be noted that each paper relies on a different subset
of patients, although the number of subjects was similar.

Table 5: Comparison of the results of different fusion methods on ADNI dataset. In the multi-classification task, 3 classes is NC vs. MCI vs. AD and 4 classes is NC vs. ncMCI vs. cMCI vs. AD. Unit:%. Research | Year |  Fusion Methods |  Dataset | NC vs. AD | NC vs MCI |  Multi-classification
---|---|---|---|---|---|---
[131] | 2015 |  Input Fusion |  331 subjects: 77 NC, 102 ncMCI, 67 cMCI, 85 AD |  ACC: 91.40 SEN: 92.32 SPE: 90.42 |  ACC: 82.10 SEN: 60.00 SPE: 92.32 |  4 Classes ACC: 53.79 SEN: 52.14 SPE: 86.98
[202] | 2021 |  Input Fusion |  381 subjects: 126 NC, 160 MCI, 95 AD |  ACC: 94.11 SEN: 93.33 SPE: 94.27 |  ACC: 85.00 SEN: 84.69 SPE: 85.60 |  3 Classes ACC: 71.52 SEN: 55.67 SPE: 83.40
[102] | 2022 |  Input Fusion |  370 subjects: 130 NC, 129 MCI, 111 AD |  ACC: 93.21 SEN: 91.43 SPE: 95.42 |  ACC: 86.52 SEN: 94.34 SPE: 81.64 |  3 Classes ACC: 87.67
[207] | 2014 |  Single-level Fusion |  398 subjects: 101 NC, 128 ncMCI, 76 cMCI, 93 AD |  ACC: 95.35 SEN: 94.65 SPE: 95.22 |  ACC: 85.67 SEN: 95.37 SPE: 65.87 | -
[196] | 2017 |  Single-level Fusion |  202 subjects: 52 NC, 56 ncMCI, 43 cMCI, 51 AD |  ACC: 97.13 SEN: 95.93 SPE: 98.53 |  ACC: 87.24 SEN: 97.91 SPE: 67.04 |  4 Classes ACC: 57.00 SEN: 53.65 SPE: 85.05
[250] | 2019 |  Single-level Fusion |  392 subjects: 101 NC, 200 MCI, 91 AD |  ACC: 98.47 SEN: 96.58 SPE: 95.39 |  ACC: 85.74 SEN: 90.11 SPE: 91.82 | -
[3] | 2022 |  Single-level Fusion |  959 subjects: 264 NC, 273 ncMCI, 204 cMCI, 218 AD |  ACC: 98.24 SEN: 98.82 SPE: 97.52 |  ACC: 94.59 SEN: 90.26 SPE: 96.98 | -
[253] | 2020 |  Hierarchical Fusion |  500 subjects: 163 NC, 113 ncMCI, 105cMCI, 119 AD |  ACC: 95.21 SEN: 93.56 SPE: 97.48 | - |  4 Classes ACC: 86.15
[62] | 2020 |  Output Fusion |  398 subjects: 101 NC, 204 MCI, 93 AD |  ACC: 99.27 SEN: 95.89 SPE: 98.72 |  ACC: 90.35 SEN: 88.36 SPE: 92.56 | -

In general, we believe that deep multi-level fusion can better exploit the
synergy of multimodal data to produce better classification results. This is
further supported by the results in Tab. 5. Compared with the input fusion,
the single-level fusion has a more robust feature fusion, which improves the
overall ACC of the middle fusion. Hierarchical fusion utilizing multi-level
feature fusion did not significantly improve the performance of dichotomous
classification but performed well for four-class classifications. Generally, a
complex model does not improve performance much when applied to a simple
classification task. The more complex the network, the better it is at solving
complex classification problems. When the number of categories for multi-
category classification increases from two to four, the hierarchical fusion
classification accuracy improves significantly. Last but not least, we note
that the output fusion achieves excellent results on NC versus AD
classification, thanks to the pre-training of different modal branches. With
output fusion, DL backbones can be pre-trained on a large number of unimodal
datasets and then fine-tuned on the multimodal datasets. Similar results are
reported on other datasets. ABIDE data was combined with sMRI and fMRI to
diagnose autism spectrum disorders. It was found that the hierarchical fusion
[130] result was 87.2%, which was better than the input fusion [7] result of
65.5%. Rahaman et al. [181] used the COBRE dataset for the diagnosis of
schizophrenia, and the accuracy of input fusion, output fusion, and single-
level fusion was 70%, 78%, and 95%, respectively. Based on the GAMMA dataset,
Li et al. [119] achieved 63% accuracy in input fusion, 72% accuracy in single-
level fusion, and 80% accuracy in hierarchical fusion when they used the same
dataset for glaucoma diagnosis.

It is difficult to determine a unified solution for a wide variety of
multimodal fusion medical image fusion tasks. In spite of this, we can draw
some preliminary conclusions from the above analysis. For medical modalities
with similar structures, modal registration is easier, so input fusion,
single-level fusion, and hierarchical fusion are all network structures worth
investigating. Generally, single-level fusion and hierarchical fusion fuse
deeper features, which will improve the classification performance. When data
have a wide range of structures or dimensions, single-level fusion and
attention-based fusion are preferable solutions, as they are capable of
handling a wide range of modal feature fusion scenarios. Lastly, if we have a
large number of unimodal datasets for each modality in multimodal data, output
fusion will perform well.

In addition to using a single multimodal fusion method, multiple fusion
methods can be combined [212, 120, 82, 222]. Tang et al. [212] achieved the
classification of skin lesions using a combination of single-level fusion and
output fusion. For the classification of Diabetic Retinopathy, Li et al. [120]
utilized different configurations of OCT Angiography data. Their approach
combined hierarchical fusion for registered modalities with late fusion for
unregistered modalities. In order to improve the diagnosis of breast cancer,
Hu et al. [82] fused multi-parametric MRI data at three levels: input, feature
(intermediate), and decision (output). Combining different fusion methods can
cumulate their advantages, allowing data from various perspectives to be fused
and improving classification performance to some extent. It is one of the
promising strategies that can be used when performing multimodal medical
classification.

###  5.2 How to find the best architecture?

During the investigation of multimodal approaches, we have found that
researchers need not only many tests to compare various fusion methods but
also a large number of hyperparametric tests to determine the best network
architecture for each fusion method. It takes a great deal of time and labor
to conduct these extensive tests. There have been many recent studies that
have applied Neural Architecture Search (NAS) to multimodal networks that can
integrate various fusion techniques to determine the best architecture for a
given dataset. The use of these methods is widespread in the fields of
diagnosing dementia [33], gliomas segmentation [221], multimodal action
recognition [165], visual question answering[249], multimodal damage
identification [199], multimodal gesture recognition [246], etc. We believe
that this approach has the potential to be explored in the future for
multimodal classification in medicine.

###  5.3 How to manage incomplete multimodal data?

The problem of modality incompleteness is one of the most pressing challenges
in multimodality medical research. The high cost and potentially harmful
effects of medical images may lead many patients to refuse being scanned with
multiple imaging modalities for clinical diagnosis [159]. In the ADNI dataset,
all subjects had MRI data; however, only about half of the subjects had PET
scans [159]. The most common approach to solving the modality incomplete
problem is to discard the modality incomplete subjects [131, 31, 207, 196],
but this approach reduces the number of trainable subjects for the deep
learning model, resulting in reduced classification performance. There is also
the option of estimating the features of the missing subjects [57, 204] based
on, for example, the mean or median of the subjects with complete modal data,
although this requires some prior knowledge and may introduce errors.

Generative Adversarial Networks (GAN) [69] is a type of generative model used
to produce data of a modality from another modality [70]. With the development
of GAN, more and more fields are using this technology to generate images. The
modal incompleteness problem has recently been solved through the use of GAN
in many studies [122, 251, 160, 159, 161, 91, 65, 215]. The GAN is used to
generate the missing data and then the generated data is used for multimodal
classification. It provides a significant increase in the number of subjects
in the dataset, improves the model’s classification performance, and is an
effective solution when dealing with multimodal incompleteness. GAN-based
solutions are currently the most promising.

Recent research endeavors have sought to confront the difficulties associated
with addressing unbalanced datasets and incomplete modalities through the
implementation of a distinctive fusion network design and training strategy.
This approach aims to optimize the utilization of available dataset
information while mitigating bias introduced by independently generating
missing modalities. Gravina et al. [71] introduced a Multi-Input–Multi-Output
3D CNN designed for the assessment of dementia severity, specifically tailored
for scenarios involving incomplete multimodal brain MRI and PET data. In
alignment with our hierarchical fusion architecture, they incorporated a
fusion branch named PAIRED-NET during feature extraction, employing distinct
CNN branches for MRI and PET modalities, each capable of producing independent
outputs. Simultaneous parameter updates for all three branches occurred during
training when the patient sample encompassed the full modality set. In
instances where a modality was missing, only the parameters of the branch
corresponding to the available modality were updated. This methodology
facilitates comprehensive network training using the entire dataset, enabling
classification of instances with a single missing modality during testing. Liu
et al. [125] introduced a cascaded Multi-Modal Mixing Transformer (3MT)
designed for the classification of Alzheimer’s Disease with incomplete data.
The architecture of 3MT comprises a sequence of Cascaded Modality Transformers
(CMTs), each incorporating features from a specific modality. At the
conclusion of this sequence, a more informed class prediction is obtained by
aggregating the extracted multi-modal features. In scenarios involving missing
data, the CMTs corresponding to the absent modalities receive zero embeddings,
indicating ”not available” to the model. This training approach equips the
model with prior knowledge for handling diverse missing data scenarios.

###  5.4 Does multimodal fusion always performs better?

Multimodal data not only contain complementary information but may also
contain a great deal of redundant information. One study found that multimodal
fusion did not enhance classification performance [97]. In one sense, this
relates to the design of the network, and in another sense, multimodal fusion
may not improve the performance of classification if the information in the
modalities is relatively similar or if a particular modality does not
accurately define the target class. Before starting a multimodal fusion
project and gathering multimodal data, it is advisable to conduct a redundancy
analysis [188].

Alternately, Narazani et al. [154] questioned the multimodal diagnostic
objectives. Clinical studies primarily aim to determine the type of dementia,
whereas studies on DL focused on only one type, AD. multimodal fusion studies
have performed well in terms of classifying NC versus AD, but the clinical
goal is the classification of AD at multiple levels. As a result of their
tests, multimodal fusion networks did not improve the multi-level
classification. Therefore, multimodal fusion classification studies should be
conducted in conjunction with clinical needs.

###  5.5 Can we advantageously combine multimodal image classification with
other tasks?

We should point out that input fusion [164, 89, 90, 43, 95], intermediate
fusion [56, 55, 35, 11, 115], and output fusion [94, 14] methods can also be
applied to medical image segmentation, medical image fusion, and similar
tasks. A notable trend is to incorporate the fusion network into the feature
extraction process, thereby enabling the creation of multi-task multimodal
networks. Cheng et al. [39] proposed an end-to-end multi-task learning network
for simultaneous glioma segmentation and IDH genotyping based on the sharing
of spatial and global feature representations extracted from the hybrid CNN-
Transformer encoder. The performance of both classification and segmentation
can be enhanced through the use of a joint network.

###  5.6 Can we advantageously combine images with contextual data in a
classification pipeline?

In this survey, our main focus was on integrating data from various imaging
modalities. However, out of 114 reviewed papers, 27 incorporate non-image
contextual information in various forms, often structured demographic and
clinical metadata extracted from electronic health records [85, 257, 242, 219,
168]. Clinical metadata includes the result of physical tests, like visual
acuity or refraction for chronic central serous chorioretinopathy diagnosis
[248], the result of chemical tests, like Pap or HPV for cervical dysplasia
diagnosis [241], and the result of cognitive tests, like executive functioning
(ADNI-EF) and memory (ADNI-MEM) test for Alzheimer’s disease diagnosis [110].
Sometimes, images are combined with more complex contextual data like voice
signals [106], free-form text [255], or genomic data [180, 219, 181, 176].
Image and contextual metadata have no geometrical relationship, so input and
hierarchical fusion are not relevant in this scenario. Earlier solutions thus
relied on single-level intermediate fusion or output fusion. Recent studies
tend to use attention-based intermediate fusion [255, 176, 34]. One challenge
with contextual data is that they are often incomplete: the reader is referred
to section 5.3 for this challenge. Nevertheless, all these studies report
increased classification performance when using contextual data.

###  5.7 Trends for the future

As the network structure evolves and hardware devices become increasingly
available, there has been a growing interest in multimodality research. From
Table 5, it is evident that the rapid development of multimodal fusion has led
to significant improvements in classification results. In fact, multimodal
networks based on deep learning exhibit greater potential for development than
unimodal networks.

Transformer is one of the most popular network architectures, and multimodal
fusion based on Transformer has developed rapidly in the past two years. In
particular, for visual-language tasks, Transformer can handle the fusion of
images, languages, and text very effectively. Based on the research conducted
in different fields, we classify Transformer-based multimodal fusion networks
into self-attention Transformers [6, 153, 195, 117, 163, 13, 203, 231] and
cross-attention Transformers [136, 36, 211, 263, 185, 182, 36, 37, 121], as
shown in Fig. 23. Following the extraction of features using encoders, self-
attention Transformers concatenate features from different modalities and
compute the attention relationship between the fused features using
Transformer blocks. Alternatively, cross-attentional Transformers compute the
attentional relationships among different modalities in order to achieve
information fusion. Nowadays, these two architectures are the most popular
multimodal fusion networks.

Fig. 23: Two architectures of Transformer-based fusion.

Compared to CNNs, Transformers have the advantage of efficiently identifying
long-range relationships between sequences. In medical images, most visual
representations are ordered due to the similarity of human organs. Medical
images contain more information regarding sequence relationships than natural
images [45]. This indicates that Transformer-based multimodal medical image
fusion is a promising approach, and the above two network architectures are
worth exploring. While recent medical research has employed analogous
structures [65, 267, 26], it is imperative to undertake broader investigations
and validations to extend the applicability of these findings.

In addition to these developments, the field is also witnessing an emerging
trend in the exploration of representation learning for multimodal data using
techniques such as pretext tasks or contrastive learning. These methods aim to
learn robust and transferable representations that can be applied to
downstream tasks of a classification nature or directly in parallel with the
classification task [224, 148, 30, 73, 235, 210, 74]. This field is new and a
lot of research questions are emerging. How to learn an aligned representation
across modalities? Can we learn one representation space for the different
modalities? The research community is actively moving in this direction to
address challenges associated with representation learning.

In response to the challenge posed by the limited scale of medical datasets
and the laborious nature of manual labeling, certain investigations have
resorted to harnessing information from image-text pairs available on the web.
This approach facilitates the construction of transfer learning multimodal
models, which have demonstrated notable efficacy in subsequent fine-tuning
tasks and zero-shot classification endeavors. Zhang et al. [256] introduced
ConVIRT, an unsupervised methodology for acquiring medical visual
representations through the analysis of naturally paired images and text.
Their approach involves evaluating image representations in conjunction with
paired text representations through a bidirectional objective, surpassing
alternative methods in performance across various downstream medical
classification tasks. Drawing inspiration from the Contrastive Language–Image
Pre-training (CLIP) approach [179], Huang et al. [88] undertook fine-tuning
specifically tailored for medical applications, resulting in the formulation
of Pathology Language–Image Pretraining (PLIP). Exploiting numerous de-
identified images and abundant textual data disseminated by clinicians on
public platforms like Twitter, PLIP introduces a multimodal, unsupervised,
Transformer-based transfer learning model. This model effectively classifies
new pathological images across four external datasets, exhibiting state-of-
the-art performance. These initiatives highlight the immense potential of
publicly shared medical information as a valuable resource for developing
multimodal medical AI systems, thereby enhancing the landscape of medical
diagnosis.

In support to these advancements and needs, the TorchMultimodal
library151515https://github.com/facebookresearch/multimodal has been created
as a framework for training state-of-the-art multimodal multi-task models at
scale by Meta Research using PyTorch framework. This library is the result of
concerted community efforts, reflecting the growing focus on multimodal tasks
and methodologies, as well as representation learning for multimodal data.
Furthermore, in alignment with the increasing enthusiasm for multimodal
learning, libraries such as Transformers
161616https://huggingface.co/docs/transformers/index by Hugging Face offer
robust resources for constructing and training such models. These Transformers
present pretrained models that can handle various modalities, such as text and
images, simplifying tasks like image categorization or answering questions
with multimodal data. This emphasis on multimodal functionalities underscores
the growing demand for models that possess the ability to comprehend and
analyze diverse types of data. Although Transformers are not exclusively
designed for the medical domain, they can serve as a beneficial resource for
scholars aiming to create and distribute code for tasks related to multimodal
classification of medical images.

As illustrated in Tab. LABEL:tab_paper, very few multimodal image
classification papers in the medical field are associated with public code. We
hope TorchMultimodal, Transformers or similar libraries will facilitate code
sharing.

##  6 Conclusion

In this paper, we conducted a comprehensive review of the development of deep
learning-based multimodal medical classification tasks over the past few
years. We examined the complementary relationships among several common
clinical modalities and delved into five types of architectures for deep
learning multimodal classification networks: input fusion, single-level
fusion, hierarchical fusion, attention-based fusion, and output fusion. Our
study covered a wide range of multimodal fusion scenarios in medical
classification and the application domains for which different network
architectures are most suitable.

Additionally, we discussed emerging trends and challenges in the field,
including the exploration of representation learning techniques and the
development of dedicated frameworks like TorchMultimodal. These advancements
provide efficient tools for training state-of-the-art multimodal multi-task
models at scale. In particular, we highlighted the advantages of using
Transformer-based multimodal fusion architectures, particularly in medical
imaging applications, where sequence relationships are more prevalent. This
demonstrates the potential of these architectures in advancing the field of
multimodal medical classification tasks.

Looking forward, we encourage the research community to continue investigating
novel fusion techniques, optimization methods, and network architectures to
further enhance the performance of multimodal classification tasks. Developing
interpretable models, addressing data imbalance and scarcity, and exploring
unsupervised and semi-supervised learning approaches are other areas worth
investigating. Additionally, we recommend future research focus on the
application of multimodal fusion in emerging areas such as genomics,
proteomics, and patient-centered care, where the integration of diverse data
types can potentially lead to significant improvements in diagnostic and
therapeutic outcomes.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgments

The work takes place in the framework of the ANR RHU project Evired. This work
benefits from State aid managed by the French National Research Agency under
the “Investissement d’Avenir” program bearing the reference ANR-18-RHUS-0008.
It was also funded in part by the ANR under the LabCom program
(ANR-19-LCV2-0005 - ADMIRE project).

Table 6: List of Terms. Term | Description
---|---
ADC | Apparent Diffusion Coefficient
AD | Alzheimer’s Disease
AdaBoost | Adaptive Boosting
ADHD | Attention Deficit Hyperactivity Disorder
AE | Auto-Encoder
AGDAF | Attention Guided Discriminative and Adaptive Fusion
ASD | Autism Spectrum Disorder
CatBoost | Categorical Boosting
CFP | Color Fundus Photographs
CLIMAT | Clinically-Inspired Multi-Agent Transformers
CMR | Cardiac Magnetic Resonance
CNN | Convolutional Neural Network
CSF | CerebroSpinal Fluid
CT | Computed Tomography
DBM | Deep Boltzmann Machine
DBN | Deep Belief Network
DCE | Dynamic Contrast-Enhanced
DL | Deep Learning
DNN | Deep Neural Network
Dsc | Dermatoscopic Image
DWI | Diffusion-Weighted Imaging
DXA | Dual-energy X-ray Absorptiometry
EHR | Electronic Health Records
EMR | Electronic Medical Record
FA | Fractional Anisotropy
FC Layer | Fully Connected Layer
FCNN | Fully Connected Neural Network
FHR | Fetal Heart Rate
FLAIR | Fluid Attenuated Inversion Recovery
FFN | Feed-Forward Network
fMRI | functional Magnetic Resonance Imaging
GAN | Generative Adversarial Network
GRU | Gated Recurrent Unit
HoFN | High-order Factorization Network
HPV | Human Papillomavirus
KELM | Kernel‐based Extreme Learning Machine
KNN | K-Nearest Neighbor
LDA | Linear Discriminant Analysis
LR | Logistic Regression
LSTM | Long-Short Term Memory
MD | Mean Diffusivity
mDSNet | multimodality Disease-Image-Specific Network
MGF | Multiheaded Gating Fusion
MLP | Multilayer Perceptron
MMNet | Multimodal 3D medical image classification Network
MM-SDPN | Multimodal Stacked Deep Polynomial Networks
MRI | Magnetic Resonance Imaging
NBI | Narrow Band Imaging
PCA | Principal Component Analysis
PEI | Positive Enhancement Integral
PET | Positron Emission Tomography
RBM | Restricted Boltzmann Machine
ROI | Regions-Of-Interest
SAE | Stacked Auto-Encoder
SBM | Swapping Bi-Attention Mechanism
sELM‐AE | stacked sparse Extreme Learning Machine Auto‐Encoder
sMRI | structural Magnetic Resonance Imaging
SNP | Single Nucleotide Polymorphism
SVM | Support Vector Machine
T1 | T1-Weighted
T1C | T1-Contrast
T1-Gd | T1-weighted Gadolinium Contrasted
T2 | T2-Weighted
T2WI | T2-Weighted MP-MRI Image
TSI | Tumor Shape Image
TWIST | Time-resolved Angiography with Interleaved Stochastic Trajectories
OCT | Optical Coherence Tomography
UC | Uterine Contraction
US | Ultrasound
VF | Visual Field
ViT | Vision Transformer
WSI | Whole Slide Image
XGBoost | Extreme Gradient Boosting
Table 7: List of publications for different fusion networks. IF: Input Fusion; SLF: Single-level Fusion; HF: Hierarchical Fusion; ABF: Attention-based Fusion; OF: Output Fusion.  Research work |  Multimodal combination |  Fusion Methods |  Information Fusion Technique |  DL Backbone |  Final Classifier |  Body Organ |  Dataset |  Patients |  Class |  Code
---|---|---|---|---|---|---|---|---|---|---
[114] |  MRI, PET, CSF |  IF |  Concatenation (Inputs) |  RBM |  SVM |  Brain |  ADNI |  202 |  2 |  N/A
[131] |  MRI, PET |  IF |  Concatenation (Inputs) |  Manual |  Softmax |  Brain |  ADNI |  331 |  2,4 |  N/A
[7] |  sMRI, fMRI |  IF |  Concatenation (Inputs) |  DBN |  FC Layer |  Brain |  ABIDE I, ABIDE II |  185 |  2 |  N/A
[10] |  MRI (ADC, DWI, T2) |  IF |  Concatenation (Inputs) |  CNN |  FC Layer |  Prostate |  TCIA (PROSTATEx) |  200 |  2 |  N/A
[174] |  US (US B-mode, US color Doppler) |  IF |  Concatenation (Inputs) |  CNN |  FC Layer |  Breast |  Private Data |  59959 |  4 |  N/A
[265] |  MRI (ADC, DWI, T2) |  IF |  Concatenation (Inputs) |  CNN |  FC Layer |  Prostate |  TCIA (PROSTATEx) |  201 |  2 |  N/A
[189] |  MRI (T2, ADC, High-b) |  IF |  Merge (Inputs) |  Resnet |  FC Layer |  Prostate |  Private Data, TCIA (PROSTATEx) |  687 |  2 |  N/A
[122] |  MRI, PET |  IF |  Concatenation (Inputs) |  CNN |  FC Layer |  Brain |  ADNI |  1086 |  2 |  N/A
[202] |  MRI, PET |  IF |  Merge (Inputs) |  CNN |  FC Layer |  Brain |  ADNI |  381 |  2,3 |  N/A
[50] |  MRI (T1, T1C, T2M FLAIR) |  IF |  Merge (Inputs) |  CNN |  FC Layer |  Brain |  TCIA (GBM, LGG), BraTS 2019 |  628 |  2 |  N/A
[102] |  MRI, PET |  IF |  Merge (Inputs) |  CNN |  FC Layer |  Brain |  ADNI |  370 |  2,3 |  N/A
[251] |  MRI, PET |  IF |  Concatenation (Inputs) |  Resnet |  FC Layer |  Brain |  ADNI |  873 |  2,3 |  N/A
[262] |  MRI (PEI, DWI) |  IF |  Concatenation (Inputs) |  CNN |  FC Layer |  Breast |  Private Data |  210 |  2 |  N/A
[183] |  MRI, PET |  IF |  Merge (Inputs) |  CNN |  SoftMax |  Brain |  OASIS-3 |  1098 |  2 |  N/A
[156] |  MRI, PET |  IF |  Concatenation (Inputs) |  ResNet |  FC Layer |  Brain |  ADNI |  412 |  2 |  N/A
[208] |  MRI, PET, CSF |  SLF |  Concatenation (Classic) |  SAE |  SVM |  Brain |  ADNI |  202 |  2 |  N/A
[207] |  MRI, PET |  SLF |  Concatenation (Classic) |  DBM |  SVM |  Brain |  ADNI |  398 |  2 |  N/A
[206] |  MRI, PET |  SLF |  Concatenation (Classic) |  SAE |  SVM |  Brain |  ADNI |  202 |  2 |  N/A
[241] |  Photograph of the cervix, Pap tests, HPV tests |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Cervix |  TCIA (Guanacaste Project) |  690 |  2 |  N/A
[143] |  MRI (ADC, DWI, DCE) |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Prostate |  TCIA (PROSTATEx) |  201 |  2 |  N/A
[220] |  MRI, PET |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  317 |  2 |  N/A
[266] |  sMRI, fMRI |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADHD-200 |  730 |  2 |  N/A
[243] |  MRI (ADC, T2) |  SLF |  Merge (Classic) |  CNN |  FC Layer |  Prostate |  Private Data |  160 |  2 |  161616https://github.com/Andysis/co-trained-CADx
[196] |  MRI, PET |  SLF |  Concatenation (Network) |  MM-SDPN |  FC Layer |  Brain |  ADNI |  202 |  2,4 |  N/A
[260] |  MRI, PET |  SLF |  Concatenation (Network) |  DNN |  Score Merge |  Brain |  ADNI |  805 |  3,4 |  N/A
[245] |  MRI (TI, T2, T1C, FLAIR) |  SLF |  Merge (Classic) |  CNN |  FC Layer |  Brain |  BraTS 2015 |  274 |  2 |  N/A
[38] |  MRI, PET |  SLF |  Concatenation (Network) |  CNN |  FC Layer |  Brain |  ADNI |  193 |  2 |  N/A
[109] |  MRI (ADC, T2WI) |  SLF |  Merge (Classic) |  CNN |  SVM |  Prostate |  TCIA, Private Data |  364 |  2 |  N/A
[160] |  MRI, PET |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  1457 |  2 |  N/A
[66] |  MRI (TI, T2, FLAIR) |  SLF |  Merge (Classic) |  CNN |  FC Layer |  Brain |  BraTS 2017, TCIA (LGG) |  444 |  2 |  N/A
[126] |  MRI, PET |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  397 |  2 |  N/A
[223] |  MRI (ADC, T2) |  SLF |  Concatenation (Classic) |  CNN |  Softmax |  Prostate |  Private Data, TCIA (PROSTATEx) |  360 |  2 |  N/A
[98] |  MRI, PET, CSF |  SLF |  Concatenation (Network) |  sELM‐AE |  KELM |  Brain |  ADNI |  202 |  2 |  N/A
[96] |  Dsc, Clinical Image, Metadata |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Skin |  SPC |  1011 |  2 |  N/A
[135] |  MRI, PET |  SLF |  Concatenation (Network) |  DNN |  Score Merge |  Brain |  ADNI |  626 |  2 |  N/A
[261] |  MRI, PET, SNP |  SLF |  Concatenation (Network) |  DNN |  Score Merge |  Brain |  ADNI |  360 |  2 |  N/A
[63] |  MRI, PET |  SLF |  Concatenation (Classic) |  CNN, LSTM |  Softmax |  Brain |  ADNI |  397 |  2 |  N/A
[47] |  MRI (ADC, T2, TWIST) |  SLF |  Concatenation (Classic) |  CNN |  Random Forest |  Breast |  Private Data |  576 |  2 |  N/A
[87] |  MRI, PET |  SLF |  Concatenation (Classic) |  VGG |  FC Layer |  Brain |  ADNI |  1512 |  2 |  N/A
[110] |  MRI, CSF, Demographic Information, Cognitive Performance |  SLF |  Concatenation (Classic) |  GRU |  LR |  Brain |  ADNI |  1618 |  2 |  N/A
[171] |  MRI, PET |  SLF |  Merge (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  723 |  2 |  N/A
[250] |  MRI, PET |  SLF |  Concatenation (Classic) |  CNN |  Softmax |  Brain |  ADNI |  392 |  2 |  N/A
[175] |  PET, CT |  SLF |  Merge (Classic) |  CNN |  FC Layer |  Lung |  Private Data |  397 |  3 |  N/A
[159] |  MRI, PET |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  2355 |  2 |  N/A
[61] |  MRI, PET, Clinical Datas |  SLF |  Concatenation (Classic) |  CNN, LSTM |  FC Layer |  Brain |  ADNI |  1536 |  4 |  N/A
[219] |  MRI, EHR, SNP |  SLF |  Concatenation (Classic) |  CNN, SAE |  FC Layer |  Brain |  ADNI |  808 |  3 |  N/A
[173] |  US (US B-mode, US color Doppler, US elastography images) |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Breast |  Private Data |  775 |  2 |  N/A
[142] |  MRI (T1, FA, MD) |  SLF |  Concatenation (Classic) |  ResNet |  FC Layer |  Brain |  OASIS-3 |  1098 |  3 |  N/A
[2] |  MRI, PET, SNP |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  805 |  2 |  N/A
[257] |  CT, EMR |  SLF |  Concatenation (Classic) |  CNN, HoFN |  FC Layer |  Lung |  Private Data |  733 |  4 |  N/A
[161] |  MRI, PET |  SLF |  Concatenation (Classic) |  mDSNet |  FC Layer |  Brain |  ADNI |  1455 |  2 |  N/A
[242] |  EMR, Pathological images |  SLF |  Concatenation (Classic) |  VGG, AE |  FC Layer |  Breast |  Private Data |  185 |  2 |  N/A
[254] |  MRI, PET |  SLF |  Concatenation (Network) |  CNN |  Score Merge |  Brain |  ADNI |  875 |  2 |  N/A
[92] |  MRI (T1, T2), Clinical Information |  SLF |  Concatenation (Classic) |  ResNet |  FC Layer |  Breast |  Private Data |  536 |  2 |  N/A
[180] |  sMRI, fMRI, SNP |  SLF |  Concatenation (Classic) |  DNN |  FC Layer |  Brain |  COBRE |  437 |  2 |  N/A
[3] |  MRI, PET |  SLF |  Concatenation (Network) |  CNN |  FC Layer |  Brain |  ADNI |  959 |  2 |  N/A
[172] |  Echocardio-graphy, CMR |  SLF |  Merge (Network) |  CNN |  SVM |  Heart |  Private Data |  50 |  2 |  N/A
[8] |  DXA, CFP |  SLF |  Concatenation (Network) |  CNN |  FC Layer |  Heart |  Private Data |  483 |  2 |  N/A
[181] |  sMRI, fMRI, Genomic Sequence |  SLF |  Merge (Network) |  AE, MLP, LSTM |  Softmax |  Brain |  COBRE |  437 |  2 |  N/A
[228] |  CFP, OCT |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Eye |  GAMMA |  300 |  3 |  N/A
[91] |  MRI, PET |  SLF |  Concatenation (Network) |  ResNet |  FC Layer |  Brain |  ADNI |  360 |  2 |  N/A
[237] |  VF, OCT |  SLF |  Merge (Classic) |  CNN |  FC Layer |  Eye |  Private Data |  1083 |  2 |  N/A
[86] |  VF, CFP |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Eye |  Private Data |  1027 |  2 |  N/A
[236] |  PET-AV45, PET-FDG |  SLF |  Concatenation (Classic) |  ViT |  FC Layer |  Brain |  ADNI |  381 |  2 |  N/A
[54] |  sMRI, fMRI, SNP |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  788 |  2 |  N/A
[216] |  MRI, Metadata |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  1461 |  2 |  N/A
[113] |  MRI, PET |  SLF |  Merge (Network) |  CNN |  FC Layer |  Brain |  ADNI |  536 |  2 |  N/A
[132] |  MRI, Metadata |  SLF |  Concatenation (Classic) |  VGG |  CatBoost |  Brain |  ADNI |  242 |  2 |  N/A
[101] |  MRI (T1, T2, T1C, FLAIR) |  SLF |  Concatenation (Classic) |  CNN, RNN |  FC Layer |  Brain |  BraTS 2021 |  585 |  2 |  N/A
[93] |  PET, MRI, CT |  SLF |  Concatenation (Classic) |  ViT, CNN |  FC Layer |  Brain |  OASIS-1, OASIS-3, ADNI |  N/A |  2,3,4 |  N/A
[1] |  sMRI, fMRI |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ABIDE I |  855 |  2 |  N/A
[138] |  MRI, SNP, Clinical data |  SLF |  Merge (Network) |  CNN |  FC Layer |  Brain |  ADNI |  577 |  2 |  N/A
[190] |  sMRI, fMRI |  SLF |  Concatenation (Classic) |  DNN |  FC Layer |  Brain |  ABIDE I, ABIDE II |  1383 |  2 |  N/A
[71] |  MRI, PET |  SLF |  Concatenation (Classic) |  CNN |  FC Layer |  Brain |  OASIS-3 |  1025 |  2,3 |  N/A
[140] |  White Light RGB, NBI |  HF |  Merge (Network), Concatenation (Classic) |  CNN |  FC Layer |  Digestive tract |  ISIT-UMR |  76 |  3 |  N/A
[253] |  MRI, PET |  HF |  Merge (Network), Merge (Classic) |  CNN |  Softmax |  Brain |  ADNI |  500 |  2,4 |  N/A
[258] |  MRI, PET |  HF |  Merge (Network),Merge (Classic) |  RBM |  Softmax |  Brain |  ADNI |  340 |  2 |  N/A
[77] |  MRI (T1C, FLAIR) |  HF |  Merge (Network), Merge (Classic) |  CNN |  FC Layer |  Brain |  TCIA, BraTS 2017 |  499 |  2 |  N/A
[64] |  MRI, PET |  HF |  Concatenation (Network), Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  977 |  2 |  N/A
[119] |  CFP, OCT |  HF |  Merge (Network), Concatenation (Classic) |  CNN |  FC Layer |  Eye |  GAMMA, Private Data |  264 |  2,3 |  N/A
[130] |  sMRI, fMRI |  HF |  Merge (Network), Concatenation (Classic) |  MGF |  FC Layer |  Brain |  ABIDE, ADHD-200, COBRE |  2120 |  2 |  N/A
[240] |  MRI, PET |  HF |  Merge (Network),Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  579 |  2 |  N/A
[230] |  MRI (T1, T2, T1C, FLAIR) |  HF |  Merge (Network), Concatenation (Classic) |  CNN |  FC Layer |  Brain |  BraTS 2018, BraTS 2019 |  326 |  2 |  N/A
[157] |  Dsc, Clinical Image, Metadata |  HF |  Merge (Network), Concatenation (Classic) |  CNN |  FC Layer |  Skin |  SPC |  1011 |  2,3,5 |  N/A
[146] |  MRI, PET |  HF |  Merge (Network), Merge (Classic) |  CNN, ViT |  FC Layer |  Brain |  ADNI |  720 |  2 |  N/A
[238] |  MRI (T1, T2, T1C, FLAIR) |  HF |  Concatenation (Network), Concatenation (Classic) |  CNN |  SoftMax |  Brain |  BraTS 2018, BraTS 2019 |  620 |  2 |  N/A
[215] |  MRI, PET |  HF |  Merge (Network), Concatenation (Classic) |  CNN |  FC Layer |  Brain |  ADNI |  821 |  2 |  N/A
[45] |  MRI (T1,T2) |  ABF |  Attention Fusion |  TransMed |  FC Layer |  Parotid, Knee |  MRNet |  344 |  2 |  N/A
[80] |  Visual Data, Clinical Data |  ABF |  Attention Fusion |  CLIMAT |  FFN |  Brain, Knee |  ADNI, OAI [155] |  10260 |  5 |  171717https://github.com/oulu-imeds/climatv2
[125] |  Neuro-imaging Data, Clinical Data |  ABF |  Attention Fusion |  3MT |  FC Layer |  Brain |  ADNI |  816 |  2 |  N/A
[255] |  Image, Text |  ABF |  Attention Fusion |  MMIF |  FC Layer |  Uterus |  CTU-UHB |  160 |  2 |  N/A
[118] |  Multi-parametric MRI |  ABF |  Attention Fusion |  AGDAF |  FC Layer |  Liver |  Private Data |  112 |  2 |  N/A
[46] |  MRI (T1,T2) |  ABF |  Attention Fusion |  MMNet |  FC Layer |  Parotid, Prostate |  MRNet, TCIA (PROSTATEx) |  2089 |  2 |  N/A
[176] |  Genomic Data, Pathology Data |  ABF |  Attention Fusion |  AHM-Fusion |  FC Layer |  Brain, Lung |  TCGA (LUAD/ LUSC, GBM/ LGG) |  1670 |  2,3 |  N/A
[267] |  sMRI, fMRI |  ABF |  Attention Fusion |  SBM |  FC Layer |  Brain |  ADNI |  268 |  2 |  N/A
[34] |  MRI, PET, Metadata |  ABF |  Attention Fusion |  CBAM |  FC Layer |  Brain |  ADNI |  227 |  2,3 |  N/A
[65] |  MRI, PET |  ABF |  Attention Fusion, Concatenation (Classic) |  CNN, ViT |  FC Layer |  Brain |  ADNI, OASIS-3 |  1700 |  2 |  N/A
[26] |  sMRI, fMRI |  ABF |  Attention Fusion |  ViT, CNN |  FC Layer |  Brain |  COBRE |  827 |  2,3,5 |  N/A
[225] |  Dsc, Clinical Image |  ABF |  Attention Fusion, Concatenation (Classic) |  CNN |  FC Layer |  Skin |  SPC |  1011 |  2,3,5 |  N/A
[5] |  sMRI, fMRI |  OF |  Merge (Outputs) |  CNN |  SVM, KNN, LDA |  Brain |  ADHD-200 |  730 |  2 |  N/A
[232] |  MRI (T1C, T2), Clinical Data |  OF |  Merge (Outputs) |  ResNet |  Bagging |  Kidney |  Private Data |  1162 |  2 |  N/A
[149] |  US (ROI, Tumor image, TSI, Fused image) |  OF |  Merge (Outputs) |  CNN |  Score Merge |  Breast |  BUSI [9] |  697 |  2 |  N/A
[62] |  MRI, PET |  OF |  Merge (Outputs) |  CNN |  AdaBoost |  Brain |  ADNI |  398 |  2 |  N/A
[85] |  CT, EMR |  OF |  Merge (Outputs) |  DNN, CNN |  Score Merge |  Lung |  Private Data |  1794 |  2 |  N/A
[247] |  MRI, SNP |  OF |  Merge (Outputs) |  CNN, MLP |  Ensemble Gate |  Brain |  ADNI |  100 |  2 |  N/A
[248] |  CFP, Clinical Data |  OF |  Merge (Outputs) |  ResNet |  XGBoost |  Eye |  Private Data |  166 |  2 |  N/A
[168] |  MRI, EHR |  OF |  Merge (Outputs) |  CNN, AE |  Score Merge |  Brain |  ADNI |  3996 |  2,3 |  N/A
[72] |  MRI (TI, T2, T1C, FLAIR) |  OF |  Merge (Outputs) |  CNN |  Score Merge |  Brain |  CPM-RadPath 2020 |  221 |  3 |  N/A
[106] |  Laryngeal image, Voice |  OF |  Merge (Outputs) |  CNN |  Decision Tree |  Larynx |  Private Data |  431 |  2 |  N/A
[177] |  MRI, Metadata |  OF |  Merge (Outputs) |  CNN |  CatBoost |  Brain |  OASIS-3 |  666 |  2 |  181818https://github.com/vkola-lab/ncomms2022
[82] |  MRI (DCE, T2) |  IF, SLF, OF |  Merge (Input), Merge (Classic), Merge (Outputs) |  VGG |  Score Merge |  Breast |  Private Data |  616 |  2 |  N/A
[222] |  WSI, MRI (T1, T1-Gd, T2, FLAIR) |  IF, OF |  Concatenation (Input), Merge (Outputs) |  CNN |  Score Merge |  Brain |  CPM-RadPath 2020 |  378 |  3 |  N/A
[212] |  Dsc, Clinical Image, Metadata |  SLF, OF |  Concatenation (Classic), Merge (Outputs) |  CNN |  Score Merge |  Skin |  SPC |  1011 |  2,3,5 |  N/A
[152] |  CT, MRI |  IF, OF |  Merge (Input), Merge (Outputs) |  CNN |  Score Merge |  Brain |  OASIS-3 |  1377 |  4 |  N/A

## References

  * Abbas et al. [2023] Abbas, S.Q., Chi, L., Chen, Y.P.P., 2023.  Deepmnf: Deep multimodal neuroimaging framework for diagnosing autism spectrum disorder.  Artificial Intelligence in Medicine 136, 102475.
  * Abdelaziz et al. [2021] Abdelaziz, M., Wang, T., Elazab, A., 2021.  Alzheimer’s disease diagnosis framework from incomplete multimodal data using convolutional neural networks.  Journal of Biomedical Informatics 121, 103863.
  * Abdelaziz et al. [2022] Abdelaziz, M., Wang, T., Elazab, A., 2022.  Fusing multimodal and anatomical volumes of interest features using convolutional auto-encoder and convolutional neural networks for alzheimer’s disease diagnosis.  Frontiers in Aging Neuroscience 14.
  * Abdelgawad et al. [2020] Abdelgawad, E.A., Abu-samra, M.F., Abdelhay, N.M., Abdel-Azeem, H.M., 2020.  B-mode ultrasound, color doppler, and sonoelastography in differentiation between benign and malignant cervical lymph nodes with special emphasis on sonoelastography.  Egyptian Journal of Radiology and Nuclear Medicine 51, 1–10.
  * Abdolmaleki and Abadeh [2020] Abdolmaleki, S., Abadeh, M.S., 2020.  Brain mr image classification for adhd diagnosis using deep neural networks, in: 2020 international conference on machine vision and image processing (MVIP), IEEE. pp. 1–5.
  * Akbari et al. [2021] Akbari, H., Yuan, L., Qian, R., Chuang, W.H., Chang, S.F., Cui, Y., Gong, B., 2021.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  Advances in Neural Information Processing Systems 34, 24206–24221.
  * Akhavan Aghdam et al. [2018] Akhavan Aghdam, M., Sharifi, A., Pedram, M.M., 2018.  Combination of rs-fmri and smri data to discriminate autism spectrum disorders in young children using deep belief network.  Journal of digital imaging 31, 895–903.
  * Al-Absi et al. [2022] Al-Absi, H.R., Islam, M.T., Refaee, M.A., Chowdhury, M.E., Alam, T., 2022.  Cardiovascular disease diagnosis from dxa scan and retinal images using deep learning.  Sensors 22, 4310.
  * Al-Dhabyani et al. [2019] Al-Dhabyani, W., Gomaa, M., Khaled, H., Fahmy, A., 2019.  Dataset of breast ultrasound images. data brief 28, 104863 (2020).
  * Aldoj et al. [2020] Aldoj, N., Lukas, S., Dewey, M., Penzkofer, T., 2020.  Semi-automatic classification of prostate cancer on multi-parametric mr imaging using a multi-channel 3d convolutional neural network.  European radiology 30, 1243–1253.
  * Andrade-Miranda et al. [2022] Andrade-Miranda, G., Jaouen, V., Bourbonne, V., Lucia, F., Visvikis, D., Conze, P.H., 2022.  Pure versus hybrid transformers for multi-modal brain tumor segmentation: a comparative study, in: 2022 IEEE International Conference on Image Processing (ICIP), IEEE. pp. 1336–1340.
  * Andrearczyk et al. [2022] Andrearczyk, V., Oreiller, V., Boughdad, S., Rest, C.C.L., Elhalawani, H., Jreige, M., Prior, J.O., Vallières, M., Visvikis, D., Hatt, M., et al., 2022.  Overview of the hecktor challenge at miccai 2021: automatic head and neck tumor segmentation and outcome prediction in pet/ct images, in: Head and Neck Tumor Segmentation and Outcome Prediction: Second Challenge, HECKTOR 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27, 2021, Proceedings. Springer, pp. 1–37.
  * Appalaraju et al. [2021] Appalaraju, S., Jasani, B., Kota, B.U., Xie, Y., Manmatha, R., 2021.  Docformer: End-to-end transformer for document understanding, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 993–1003.
  * Aygün et al. [2018] Aygün, M., Şahin, Y.H., Ünal, G., 2018.  Multi modal convolutional neural networks for brain tumor segmentation.  arXiv preprint arXiv:1809.06191 .
  * Azam et al. [2021] Azam, M.A., Khan, K.B., Ahmad, M., Mazzara, M., 2021.  Multimodal medical image registration and fusion for quality enhancement.  Computers, Materials & Continua 68, 821–840.
  * Azam et al. [2022] Azam, M.A., Khan, K.B., Salahuddin, S., Rehman, E., Khan, S.A., Khan, M.A., Kadry, S., Gandomi, A.H., 2022.  A review on multimodal medical image fusion: Compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics.  Computers in biology and medicine 144, 105253.
  * Bailey et al. [2005] Bailey, D.L., Maisey, M.N., Townsend, D.W., Valk, P.E., 2005.  Positron emission tomography. volume 2.  Springer.
  * Ballard [1987] Ballard, D.H., 1987.  Modular learning in neural networks., in: Aaai, pp. 279–284.
  * Baltrušaitis et al. [2018] Baltrušaitis, T., Ahuja, C., Morency, L.P., 2018.  Multimodal machine learning: A survey and taxonomy.  IEEE transactions on pattern analysis and machine intelligence 41, 423–443.
  * Bandettini [2012] Bandettini, P.A., 2012.  Twenty years of functional mri: the science and the stories.  Neuroimage 62, 575–588.
  * Bashir et al. [2019] Bashir, R., Junejo, R., Qadri, N.N., Fleury, M., Qadri, M.Y., 2019.  Swt and pca image fusion methods for multi-modal imagery.  Multimedia Tools and Applications 78, 1235–1263.
  * Beckett et al. [2015] Beckett, L.A., Donohue, M.C., Wang, C., Aisen, P., Harvey, D.J., Saito, N., Initiative, A.D.N., et al., 2015.  The alzheimer’s disease neuroimaging initiative phase 2: Increasing the length, breadth, and depth of our understanding.  Alzheimer’s & Dementia 11, 823–831.
  * Besenczi et al. [2016] Besenczi, R., Tóth, J., Hajdu, A., 2016.  A review on automatic analysis techniques for color fundus photographs.  Computational and structural biotechnology journal 14, 371–384.
  * Bhat and Koundal [2021] Bhat, S., Koundal, D., 2021.  Multi-focus image fusion techniques: a survey.  Artificial Intelligence Review 54, 5735–5787.
  * Bhatnagar et al. [2013] Bhatnagar, G., Wu, Q.J., Liu, Z., 2013.  Directive contrast based multimodal medical image fusion in nsct domain.  IEEE transactions on multimedia 15, 1014–1024.
  * Bi et al. [2023] Bi, Y., Abrol, A., Fu, Z., Calhoun, V., 2023.  A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data.  bioRxiv , 2023–07.
  * Bien et al. [2018] Bien, N., Rajpurkar, P., Ball, R.L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B.N., Yeom, K.W., Shpanskaya, K., et al., 2018.  Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of mrnet.  PLoS medicine 15, e1002699.
  * Boulahia et al. [2021] Boulahia, S.Y., Amamra, A., Madi, M.R., Daikh, S., 2021.  Early, intermediate and late fusion strategies for robust deep learning-based multimodal action recognition.  Machine Vision and Applications 32, 121.
  * Buzug [2011] Buzug, T.M., 2011.  Computed tomography.  Springer.
  * Cai et al. [2022] Cai, Z., Lin, L., He, H., Tang, X., 2022.  Corolla: An efficient multi-modality fusion framework with supervised contrastive learning for glaucoma grading, in: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–4.  doi:10.1109/ISBI52829.2022.9761712.
  * Calhoun and Sui [2016] Calhoun, V.D., Sui, J., 2016.  Multimodal fusion of brain imaging data: a key to finding the missing link (s) in complex mental illness.  Biological psychiatry: cognitive neuroscience and neuroimaging 1, 230–244.
  * Calhoun et al. [2012] Calhoun, V.D., Sui, J., Kiehl, K., Turner, J., Allen, E., Pearlson, G., 2012.  Exploring the psychosis functional connectome: aberrant intrinsic networks in schizophrenia and bipolar disorder.  Frontiers in psychiatry 2, 75.
  * Chatzianastasis et al. [2023] Chatzianastasis, M., Ilias, L., Askounis, D., Vazirgiannis, M., 2023.  Neural architecture search with multimodal fusion methods for diagnosing dementia.  arXiv preprint arXiv:2302.05894 .
  * Chen et al. [2023] Chen, H., Guo, H., Xing, L., Chen, D., Yuan, T., Zhang, Y., Zhang, X., 2023.  Multimodal predictive classification of alzheimer’s disease based on attention-combined fusion network: Integrated neuroimaging modalities and medical examination data.  IET Image Processing 17, 3153–3164.
  * Chen et al. [2018] Chen, L., Wu, Y., DSouza, A.M., Abidin, A.Z., Wismüller, A., Xu, C., 2018.  Mri tumor segmentation with densely connected 3d cnn, in: Medical Imaging 2018: Image Processing, SPIE. pp. 357–364.
  * Chen et al. [2021a] Chen, R.J., Lu, M.Y., Weng, W.H., Chen, T.Y., Williamson, D.F., Manz, T., Shady, M., Mahmood, F., 2021a.  Multimodal co-attention transformer for survival prediction in gigapixel whole slide images, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015–4025.
  * Chen et al. [2021b] Chen, S., Guhur, P.L., Schmid, C., Laptev, I., 2021b.  History aware multimodal transformer for vision-and-language navigation.  Advances in neural information processing systems 34, 5834–5847.
  * Cheng and Liu [2017] Cheng, D., Liu, M., 2017.  Cnns based multi-modality classification for ad diagnosis, in: 2017 10th international congress on image and signal processing, biomedical engineering and informatics (CISP-BMEI), IEEE. pp. 1–5.
  * Cheng et al. [2022] Cheng, J., Liu, J., Kuang, H., Wang, J., 2022.  A fully automated multimodal mri-based multi-task learning for glioma segmentation and idh genotyping.  IEEE Transactions on Medical Imaging 41, 1520–1532.
  * Chudacek et al. [2014] Chudacek, V., Spilka, J., Bursa, M., Janku, P., Hruban, L., Huptych, M., Lhotska, L., 2014.  Open access intrapartum ctg database.  BMC pregnancy and childbirth 14, 1–12.
  * Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al., 2013.  The cancer imaging archive (tcia): maintaining and operating a public information repository.  Journal of digital imaging 26, 1045–1057.
  * consortium [2012] consortium, A.., 2012.  The adhd-200 consortium: a model to advance the translational potential of neuroimaging in clinical neuroscience.  Frontiers in systems neuroscience 6, 62.
  * Cui et al. [2018] Cui, S., Mao, L., Jiang, J., Liu, C., Xiong, S., 2018.  Automatic semantic segmentation of brain gliomas from mri images using a deep cascaded neural network.  Journal of healthcare engineering 2018.
  * Cuingnet et al. [2011] Cuingnet, R., Gerardin, E., Tessieras, J., Auzias, G., Lehéricy, S., Habert, M.O., Chupin, M., Benali, H., Colliot, O., ADNI, et al., 2011.  Automatic classification of patients with alzheimer’s disease from structural mri: a comparison of ten methods using the adni database.  neuroimage 56, 766–781.
  * Dai et al. [2021] Dai, Y., Gao, Y., Liu, F., 2021.  Transmed: Transformers advance multi-modal medical image classification.  Diagnostics 11, 1384.
  * Dai et al. [2022] Dai, Y., Gao, Y., Liu, F., Fu, J., 2022.  Mutual attention-based hybrid dimensional network for multimodal imaging computer-aided diagnosis.  arXiv preprint arXiv:2201.09421 .
  * Dalmis et al. [2019] Dalmis, M.U., Gubern-Mérida, A., Vreemann, S., Bult, P., Karssemeijer, N., Mann, R., Teuwen, J., 2019.  Artificial intelligence–based classification of breast lesions imaged with a multiparametric breast mri protocol with ultrafast dce-mri, t2, and dwi.  Investigative radiology 54, 325–332.
  * Das and Kundu [2013] Das, S., Kundu, M.K., 2013.  A neuro-fuzzy approach for medical image fusion.  IEEE transactions on biomedical engineering 60, 3347–3353.
  * Davatzikos et al. [2011] Davatzikos, C., Bhatt, P., Shaw, L.M., Batmanghelich, K.N., Trojanowski, J.Q., 2011.  Prediction of mci to ad conversion, via mri, csf biomarkers, and pattern classification.  Neurobiology of aging 32, 2322–e19.
  * Decuyper et al. [2021] Decuyper, M., Bonte, S., Deblaere, K., Van Holen, R., 2021.  Automated mri based pipeline for segmentation and prediction of grade, idh mutation and 1p19q co-deletion in glioma.  Computerized Medical Imaging and Graphics 88, 101831.
  * Deng et al. [2009] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009.  Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee. pp. 248–255.
  * Di Martino et al. [2017] Di Martino, A., O’connor, D., Chen, B., Alaerts, K., Anderson, J.S., Assaf, M., Balsters, J.H., Baxter, L., Beggiato, A., Bernaerts, S., et al., 2017.  Enhancing studies of the connectome in autism using the autism brain imaging data exchange ii.  Scientific data 4, 1–15.
  * Di Martino et al. [2014] Di Martino, A., Yan, C.G., Li, Q., Denio, E., Castellanos, F.X., Alaerts, K., Anderson, J.S., Assaf, M., Bookheimer, S.Y., Dapretto, M., et al., 2014.  The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism.  Molecular psychiatry 19, 659–667.
  * Dolci et al. [2022] Dolci, G., Rahaman, M.A., Chen, J., Duan, K., Fu, Z., Abrol, A., Menegaz, G., Calhoun, V.D., 2022.  A deep generative multimodal imaging genomics framework for alzheimer’s disease prediction, in: 2022 IEEE 22nd International Conference on Bioinformatics and Bioengineering (BIBE), IEEE. pp. 41–44.
  * Dolz et al. [2019] Dolz, J., Desrosiers, C., Ben Ayed, I., 2019.  Ivd-net: Intervertebral disc localization and segmentation in mri with a multi-modal unet, in: Computational Methods and Clinical Applications for Spine Imaging: 5th International Workshop and Challenge, CSI 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Springer. pp. 130–143.
  * Dolz et al. [2018] Dolz, J., Gopinath, K., Yuan, J., Lombaert, H., Desrosiers, C., Ayed, I.B., 2018.  Hyperdense-net: a hyper-densely connected cnn for multi-modal image segmentation.  IEEE transactions on medical imaging 38, 1116–1126.
  * Donders et al. [2006] Donders, A.R.T., Van Der Heijden, G.J., Stijnen, T., Moons, K.G., 2006.  A gentle introduction to imputation of missing values.  Journal of clinical epidemiology 59, 1087–1091.
  * Dubois et al. [2007] Dubois, B., Feldman, H.H., Jacova, C., DeKosky, S.T., Barberger-Gateau, P., Cummings, J., Delacourte, A., Galasko, D., Gauthier, S., Jicha, G., et al., 2007.  Research criteria for the diagnosis of alzheimer’s disease: revising the nincds–adrda criteria.  The Lancet Neurology 6, 734–746.
  * El-Gamal et al. [2016] El-Gamal, F.E.Z.A., Elmogy, M., Atwan, A., 2016.  Current trends in medical image registration and fusion.  Egyptian Informatics Journal 17, 99–124.
  * El Habib Daho et al. [2023] El Habib Daho, M., Li, Y., Zeghlache, R., Atse, Y.C., Le Boité, H., Bonnin, S., Cosette, D., Deman, P., Borderie, L., Lepicard, C., Tadayoni, R., Cochener, B., Conze, P.H., Lamard, M., Quellec, G., 2023.  Improved automatic diabetic retinopathy severity classification using deep multimodal fusion of uwf-cfp and octa images, in: Antony, B., Chen, H., Fang, H., Fu, H., Lee, C.S., Zheng, Y. (Eds.), Ophthalmic Medical Image Analysis, Springer Nature Switzerland, Cham. pp. 11–20.
  * El-Sappagh et al. [2020] El-Sappagh, S., Abuhmed, T., Islam, S.R., Kwak, K.S., 2020.  Multimodal multitask deep learning model for alzheimer’s disease progression detection based on time series data.  Neurocomputing 412, 197–215.
  * Fang et al. [2020] Fang, X., Liu, Z., Xu, M., 2020.  Ensemble of deep convolutional neural networks based multi-modality images for alzheimer’s disease diagnosis.  IET Image Processing 14, 318–326.
  * Feng et al. [2019] Feng, C., Elazab, A., Yang, P., Wang, T., Zhou, F., Hu, H., Xiao, X., Lei, B., 2019.  Deep learning framework for alzheimer’s disease diagnosis via 3d-cnn and fsbi-lstm.  IEEE Access 7, 63605–63618.
  * Gao et al. [2021] Gao, X., Shi, F., Shen, D., Liu, M., 2021.  Task-induced pyramid and attention gan for multimodal brain image imputation and classification in alzheimer’s disease.  IEEE journal of biomedical and health informatics 26, 36–43.
  * Gao et al. [2023] Gao, X., Shi, F., Shen, D., Liu, M., 2023.  Multimodal transformer network for incomplete image generation and diagnosis of alzheimer’s disease.  Computerized Medical Imaging and Graphics 110, 102303.
  * Ge et al. [2018] Ge, C., Gu, I.Y.H., Jakola, A.S., Yang, J., 2018.  Deep learning and multi-sensor fusion for glioma classification using multistream 2d convolutional networks, in: 2018 40th Annual international conference of the IEEE engineering in medicine and biology society (EMBC), IEEE. pp. 5894–5897.
  * Ge et al. [2017] Ge, Z., Demyanov, S., Chakravorty, R., Bowling, A., Garnavi, R., 2017.  Skin disease recognition using deep saliency features and multimodal learning of dermoscopy and clinical images, in: Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III 20, Springer. pp. 250–258.
  * Goodfellow et al. [2016] Goodfellow, I., Bengio, Y., Courville, A., 2016.  Deep learning.  MIT press.
  * Goodfellow et al. [2020] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2020.  Generative adversarial networks.  Communications of the ACM 63, 139–144.
  * Goodfellow et al. [2014] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2014.  Generative adversarial networks.  URL: https://arxiv.org/abs/1406.2661, doi:10.48550/ARXIV.1406.2661.
  * Gravina et al. [2024] Gravina, M., García-Pedrero, A., Gonzalo-Martín, C., Sansone, C., Soda, P., 2024.  Multi input–multi output 3d cnn for dementia severity assessment with incomplete multimodal data.  Artificial Intelligence in Medicine 149, 102774.