# Multimodal Explainability via Latent Shift applied to COVID-19
stratification

Valerio Guarrasi
Unit of Computer Systems and Bioinformatics, Department of Engineering
University Campus Bio-Medico of Rome
Department of Computer, Control, and Management Engineering
Sapienza University of Rome
<EMAIL_ADDRESS>
&Lorenzo Tronchin
Unit of Computer Systems and Bioinformatics, Department of Engineering
University Campus Bio-Medico of Rome
<EMAIL_ADDRESS>
&Domenico Albano
IRCCS Istituto Ortopedico Galeazzi
&Eliodoro Faiella
Department of Radiology, Sant’Anna Hospital
&Deborah Fazzini
Department of Diagnostic Imaging and Stereotactic Radiosurgey, Centro
Diagnostico Italiano S.p.A.
&Domiziana Santucci
Department of Radiology, Sant’Anna Hospital
University Campus Bio-Medico of Rome
&Paolo Soda
Unit of Computer Systems and Bioinformatics, Department of Engineering
University Campus Bio-Medico of Rome
Department of Radiation Sciences, Radiation Physics, Biomedical Engineering
Umeå University

###### Abstract

We are witnessing a widespread adoption of artificial intelligence in
healthcare. However, most of the advancements in deep learning (DL) in this
area consider only unimodal data, neglecting other modalities. Their
multimodal interpretation necessary for supporting diagnosis, prognosis and
treatment decisions. In this work we present a deep architecture, explainable
by design, which jointly learns modality reconstructions and sample
classifications using tabular and imaging data. The explanation of the
decision taken is computed by applying a latent shift that, simulates a
counterfactual prediction revealing the features of each modality that
contribute the most to the decision and a quantitative score indicating the
modality importance. We validate our approach in the context of COVID-19
pandemic using the AIforCOVID dataset, which contains multimodal data for the
early identification of patients at risk of severe outcome. The results show
that the proposed method provides meaningful explanations without degrading
the classification performance.

_K_ eywords XAI $\cdot$ Multimodal deep learning $\cdot$ Joint fusion $\cdot$
Classification $\cdot$ COVID-19

##  1 Introduction

In the last decade, the practice of modern medicine has started to heavily
rely on the utilization of data coming from multiple sources [1]. At the same
time, artificial intelligence (AI) has achieved state-of-the-art results in
various domains [2], healthcare included. Nevertheless, most of the deep
neural networks applied to medical tasks consider only one data modality,
neglecting information available in other sources. However, analyzing medical
findings is multimodal by its very nature: a characteristic that, in turn,
asks for developing AI approaches able to process data of different modalities
[3]. This has fostered the rise of multimodal deep learning (MDL) [3, 4],
which aims to develop learning models able to process and link information
gathered from different modalities. MDL is a topic where researchers have
investigated several methods to learn together multimodal information via
early, late, and joint fusion [4], as it will be presented in section 2.

With the goal of reaching high performance, many complex AI models developed
so far have a black-box nature [5], neglecting trust and transparency [6], two
features that are of particular importance in biomedicine [7]. Indeed, a lack
in explainability limits the application of AI models into the clinical
practice. To overcome this limitation, in the last years large research
efforts have been directed towards explainable AI (XAI), which aims to explain
how a black-box model produces its outcomes.

In healthcare, the need of multimodal models and of explainability make
multimodal explanations vital to develop robust and trustworthy AI models.
This happens because multimodal models extract more comprehensive information
than the unimodal models, so that their explanations could offer more insights
into the available medical data. The multimodal setting of XAI explores the
complementary and explanatory strengths of the different modalities, with the
goal of obtaining better explanations that localize the relevant features and
modalities [8]. Despite this relevance, to the best of our knowledge, the
biomedical literature lacks of explainable deep multimodal models. For this
reason we present a novel end-to-end multimodal architecture, explainable by
design, that jointly learns modality reconstructions and multimodal
classification using imaging and tabular modalities. With this architecture we
extract deep multimodal representations of the data; then, we apply a latent
shift to simulate a counterfactual prediction, thus obtaining an intrinsic
explanation that reveals how and why the model arrived at a particular
decision.

We test our approach in the context of the COVID-19 pandemic using the
AIforCOVID public dataset [9] for three reasons. First, the development of AI-
based tools supporting COVID-19 prognosis exploiting multimodal data is still
an open research issue addressed by few work in the literature [9, 10, 11, 12,
13, 14, 15], which will be discussed in section 2. Second, while there is a
lack of multimodal XAI (MXAI) approaches in general, no one has proposed
multimodal explanations to gain trust and transparency in COVID-19 prognosis.
Third, the AIforCOVID dataset is the largest publicly available repository
containing chest X-ray (CXR) images and clinical data collected at the time of
hospitalization, further to clinical outcomes stratifying patients into those
with and without risk of a severe disease progression [16]. It is worth noting
that, in general, images and clinical data are two important sources of
information in medicine. Indeed, the former allows radiologists to focus on
visual evidence for both diagnostic and prognostic purposes, whereas the
latter which are usually stores as tabular data, offer to clinicians a concise
and multi-dimensional assessment of patients’ health status. Hence, having
both the modalities should be important to test MDL and MXAI approaches that
would support the medical decision process.

The rest of the manuscript is organized as follows. Section 2 introduces the
state-of-the art of both MDL and MXAI. Then, section 3 presents our novel
architecture and training procedure, and it explains the MXAI method
extracting multimodal explanations. In section 4 we describe the dataset used
to validate the methods, the pre-processing phase on the data, the
implementation setup and the validation strategy adopted. Section 5 presents
and discusses the obtained results, whilst section 6 provides concluding
remarks.

##  2 Background

In this section we first present the state-of-the-art of MDL in the context of
COVID-19 prognosis prediction by using images and tabular clinical data,
whereas the interested readers can deepen MDL in healthcare in recent surveys
[1, 17]. Indeed the literature is quite large and its review is out of the
scope of this work. Second, given the lack of MXAI approaches in COVID-19
prognosis, we will summarize MXAI research in healthcare, whilst the readers
can refer to [18, 19, 20] for XAI applications on unimodal data in COVID-19
imaging. We will conclude this section summarizing the motivations of this
work.

###  2.1 MDL

There is a general consensus that medical images complemented by clinical data
can help physicians, and radiologist in particular, better understanding the
patient’s state, thus advancing to a more informative decision making process
[21, 22]. In this respect, research in the field of MDL has been increasing
[3] since multimodal data give the opportunity to train models that can learn
the complex dynamics behind a disease.

The level at which the fusion of input modalities occurs in the network is
usually distinguished into early, intermediate or late fusion [4]. Early
fusion combines raw data or extracted features by the different modalities
which are fed to a simple learner, whilst late fusion combines at the decision
level the outputs of networks independently trained on each modality.
Intermediate fusion learns a joint representation of different modalities at a
shared representation layer, by propagating the loss back to the feature
extractor network in an end-to-end manner.

The application of AI in COVID-19 using medical imaging and clinical data has
mainly focused on discriminating patients suffering from COVID-19 pneumonia
from those which are healthy or affected by different types of pneumonia [23].
Nevertheless, only four work investigated patients’ stratification into mild
and severe outcomes using such multimodal data [9, 13, 14, 15], which can be
further divided in three using computed tomography (CT) scans [13, 14, 15] and
one using CXR images [9].

In [13] the authors proposed a deep network using CT scans and $53$ clinical
features to detect the potential malignant progression of mild patients. Via a
multi-layer perceptron (MLP) processing the clinical features only, their
method gets an embedding that is then concatenated in an early fusion approach
with the flattened CT scan. This new vector then fed a Long short-term memory
(LSTM) network followed by a fully-connected (FC) network, which produces the
output. On a private cohort of 199 patients, they obtained an accuracy of
$79.20\%$. In [14] the authors used $130$ clinical features and CT scans to
discriminate between negative, mild and severe COVID-19 cases via an early
fusion of a VGG-16 and a 7-layer FC network. They used a private dataset
containing 1521 patients, achieving an accuracy equal to $81.10\%$. Fang et al
applied joint fusion combining CT scans and $61$ clinical features, which fed
a deep network to predict COVID-19 malignant progression [15]. The approach
extracts the abstract representations of CT images using a 3D ResNet, and of
the clinical data via a FC network. These two embeddings are concatenated and
given to an LSTM followed by another FC network. The whole architecture is
training in an end-to-end modality. On a private dataset containing 1040
patients they achieved an accuracy equal to $87.70\%$.

It is worth noting that CXR helps indicating abnormal formations of a large
variety of chest diseases by using a very small amount of radiation, whilst CT
delivers a much higher detail level of the lungs’ structures. Furthermore,
X-ray equipment is much smaller, less complex, and with lower costs than CT
scans; it also prevents other three limitations of CT imaging, i.e., the lack
of available machines’ slots, the difficulty of moving bedridden patients, and
the long sanitation times. For these reasons, several authors indicated that
CXR imaging fit well with the needs of COVID-19 pandemic [24]. Let us now
focus on the only work that uses CXR images and clinical data for the COVID-19
prognosis [9]. There, the authors presented three different multimodal AI
approaches, offering also baseline performance on the AIforCOVID dataset,
which we will present in section 4 together with the best results attained.
The first method, named as handcrafted approach, computed first-order and
texture features from the images, which are then stacked in an early fusion
fashion with the clinical features, then feeding different learners among
which the Support Vector Machine resulted to be the best. The second approach,
referred to as hybrid, combines automatic features extracted by a
convolutional neural network with the clinical ones; then it runs a feature
selection stage whose output is given to the learner. The hybrid approach
achieved the best results using the GoogleNet and the Support Vector Machine
classifier. The third approach, named as end-to-end, performs intermediate
fusion of the two modalities by defining a multi-input network concatenating
hidden vectors of the two modalities. This architecture contains three main
branches: two process independently CXR scans and clinical features to get a
small number of relevant and abstract features, while the third one
concatenates such embeddings that is given to a FC network, which outputs the
prognosis.

###  2.2 MXAI

High performing deep models are often black-boxes, which hide their decision-
making process, making it hard to understand why a certain result is obtained.
This has boosted the growth of XAI, and many unimodal methods have been
proposed to extract explanations on how the model has interpreted the data [6]
with applications to different fields. In particular, many authors agree that
explanations are strongly recommended in medical applications [25], because
this would help mapping explainability with causability that, in turns, would
allow practitioners to understand why a model came up with a result.

The only available review on MXAI [26] surveys its applications in computer
vision and natural language processing, showing that MXAI lacks in the medical
field. This is confirmed by the position paper [27], which states that in
radiology there is a lack of integrative methods that, combining imaging and
tabular data, provide explanations on the decisions taken. This confirms the
need of multimodal explanations to capture the complexity of all the factors
underlying a disease. Indeed, for a medical task to have a comprehensive
global view of the data and of the system [28], an ideal MXAI method should be
able to identify the importance of each modality and the importance of each
unimodal feature.

The MXAI review [26], even if it does not focus on the medical field, is also
interesting because it groups XAI algorithms adopting three different
criteria. First, it focuses on the stage at which the XAI can be applied,
identifying pre-modeling, during modelling and post-hoc modeling explanations
[26]. As their names explain, the pre-modeling methods’ explainability is
included before the model development, during modelling include the models
which are usually explainable by design and employ intrinsic methods, and
post-hoc modelling is applied after the model is developed by extracting
explanations via perturbations or backpropagation methods [29]. Second, with
reference to the scope of the explanation, XAI models can be either local or
global [26], depending if the explanation regards a single instance or the
model as a whole. The third criterion deal with the dependency of the XAI
algorithm, so that it distinguishes model specific and model agnostic
explanations [26]. While the interested readers can deepen [26] to have more
details, on the basis of this survey we observe that there is a lack in
multimodal intrinsic explainability, i.e., methods able to return multimodal
local explanations.

As mentioned in the forewords of this section, the analysis of the literature
that uses multimodal data for COVID-19 prognosis shows that, to the best of
our knowledge, none has investigated MXAI in this field yet.

###  2.3 Motivations

Multimodal settings have improved the predictive power of models in many
applications thanks to the interaction of different modalities, via a richer
representation with task-relevant features [8]. Nevertheless, this
availability of information from different modality makes explainability a key
necessity to reduce the opacity of the multimodal deep architectures [7]. This
has recently fostered the raise of MXAI, which has mainly focused on computer
vision and natural language processing. Indeed, the literature on XAI in
medical applications has concentrated more on unimodal attribution methods,
struggling in having explanations of neural networks working on multiple data
sources. Therefore, developing multimodal methods for explainability is an
urgent and open issue, also because the development of multimodal deep
architectures in different healthcare applications asks for novel approaches
to open such black boxes. In turn, this can help physicians, patients and
regulators to trust the decisions taken. Among the several fields where MDL
and MXAI can be applied, we test our methodology to the early identification
of COVID-19 patients at risk of severe outcome using imaging and tabular data,
because the survey of the literature presented hereinbefore shows that few
work has addressed this challenge, despite the disruptive impact of this
disease worldwide.

##  3 Methods

In this section we present a novel architecture that exploits joint learning,
for which we design an intrinsic counterfactual MXAI approach to extract
explanations of a classification task. Such multimodal explanations will
permit users to understand not only the importance of each modality for each
classification, but also the features which contributed the most to the
decision for every single modality.

We first present the architecture of the multimodal model; second we focus on
the training approach and, third, we detail the intrinsic MXAI method.

###  3.1 Notation

The notation used henceforth makes us of the following symbols:

  * •

$T$ and $I$ are the tabular and the imaging modalities, respectively;

  * •

$\boldsymbol{x}_{T}$ and $\boldsymbol{x}_{I}$ are the inputs for the tabular
and imaging modality, respectively;

  * •

$AE$ and $CAE$ are the autoencoder and convolution autoencoder which receive
as input $\boldsymbol{x}_{T}$ and $\boldsymbol{x}_{I}$, respectively. Both are
composed of an encoder $E_{AE}$, $E_{CAE}$ and a decoder $D_{AE}$, $D_{CAE}$,
respectively;

  * •

$\boldsymbol{h}_{T}\in\mathbb{R}^{n}$ and
$\boldsymbol{h}_{I}\in\mathbb{R}^{m}$ are the latent vectors of the $AE$ and
the $CAE$, respectively. Their concatenation produces the multimodal embedding
$\boldsymbol{h}\in\mathbb{R}^{n+m}$;

  * •

$\hat{\boldsymbol{x}}_{T}$ and $\hat{\boldsymbol{x}}_{I}$ are the outputs
produced by the $AE$ and the $CAE$, respectively, representing the
reconstruction of the inputs $\boldsymbol{x}_{T}$ and $\boldsymbol{x}_{I}$;

  * •

$C_{MLP}$ is the multi-layer perceptron receiving the vector $\boldsymbol{h}$
to perform the classification;

  * •

$\boldsymbol{y}\in\mathbb{R}^{c}$ is the output vector of $C_{MLP}$, which
expresses the predicted posterior probability, with $c$ being equal to the
number of classes;

  * •

$L_{T}$, $L_{I}$, $L_{C}$ are the loss functions of the $AE$, $CAE$ and
$C_{MLP}$, respectively, whose linear combination results in $L$, weighted by
the corresponding scalar parameters $\gamma_{T}\in\mathbb{R}$,
$\gamma_{I}\in\mathbb{R}$, $\gamma_{C}\in\mathbb{R}$;

  * •

$\boldsymbol{h}_{T}^{\lambda}\in\mathbb{R}^{n}$,
$\boldsymbol{h}_{I}^{\lambda}\in\mathbb{R}^{m}$ and
$\boldsymbol{h}^{\lambda}\in\mathbb{R}^{n+m}$ are the modified vector
embeddings of $\boldsymbol{h}_{T}$, $\boldsymbol{h}_{I}$ and $\boldsymbol{h}$,
respectively, regulated by scalar parameter $\lambda\in\mathbb{R}$;

  * •

$\hat{\boldsymbol{x}}_{T}^{\lambda}$, $\hat{\boldsymbol{x}}_{I}^{\lambda}$ and
$\boldsymbol{y}^{\lambda}\in\mathbb{R}^{c}$ are the outputs produced by
$D_{AE}$, $D_{CAE}$ and $C_{MLP}$, respectively, when the input is
$\boldsymbol{h}_{T}^{\lambda}$, $\boldsymbol{h}_{I}^{\lambda}$ and
$\boldsymbol{h}^{\lambda}$, respectively;

  * •

$\Delta_{T}\in\mathbb{R}$ and $\Delta_{I}\in\mathbb{R}$ express the resulting
modality importance comparing $\boldsymbol{h}_{T}$ with
$\boldsymbol{h}_{T}^{\lambda}$ and $\boldsymbol{h}_{I}$ with
$\boldsymbol{h}_{I}^{\lambda}$, respectively;

  * •

$\hat{\boldsymbol{\Delta}}_{T}\in\mathbb{R}^{n}$,
$\hat{\boldsymbol{\Delta}}_{I}\in\mathbb{R}^{m}$ express the resulting
unimodal feature importance comparing $\hat{\boldsymbol{x}}_{T}$ with
$\hat{\boldsymbol{x}}_{T}^{\lambda}$ and $\hat{\boldsymbol{x}}_{I}$ with
$\hat{\boldsymbol{x}}_{I}^{\lambda}$, respectively.

###  3.2 Architecture

Here we present the structure of the designed classification model that works
with $T$ and $I$. The proposed multimodal architecture consists of three
blocks: an autoencoder ($AE$), a convolutional autoencoder ($CAE$), and a
multi-layer perceptron classifier ($C_{MLP}$). As shown in Figure 1, the
network has two inputs and three outputs. The tabular modality
$\boldsymbol{x}_{T}$ feeds the $AE$, whereas the imaging modality
$\boldsymbol{x}_{I}$ is given to the CAE. Both are composed of an encoder and
a decoder, returning the reconstruction of the respective modality,
$\hat{\boldsymbol{x}}_{T}$ and $\hat{\boldsymbol{x}}_{I}$. By concatenating
the two embeddings $\boldsymbol{h}_{T}$ and $\boldsymbol{h}_{I}$ we get
$\boldsymbol{h}$, which is given to the $C_{MLP}$ classifier that returns the
classification vector $\boldsymbol{y}$. The entire architecture is trained in
an end-to-end manner, via a linear combination of three loss functions, two
for reconstruction ($L_{T}$ and $L_{I}$) and one for the classification
($L_{C}$).

This overview reveals that our framework jointly learns deep representations
of imaging and tabular data to perform a classification task. Indeed, it
learns a feature space with local modality structure able to be used for
reconstruction, and it manipulates the combined feature space by incorporating
a classification oriented loss.

Figure 1: Schematic view of the multimodal deep architecture.

#### Autoencoders

An $AE$ and a $CAE$ are artificial neural networks which learn an
approximation to the identity function, with the goal of minimizing the
distance between the outputs $\hat{\boldsymbol{x}}_{T}$,
$\hat{\boldsymbol{x}}_{I}$ and the inputs $\boldsymbol{x}_{T}$,
$\boldsymbol{x}_{I}$, respectively. The encoders $E_{AE}$ and $E_{CAE}$
compress the corresponding inputs $\boldsymbol{x}_{T}$, $\boldsymbol{x}_{I}$
to a latent space representation $\boldsymbol{h}_{T}$ and
$\boldsymbol{h}_{I}$, using fully connected layers in the $AE$ and
convolutional layers in the $CAE$, respectively. The decoders $D_{AE}$ and
$D_{CAE}$ use the bottleneck latent space representation $\boldsymbol{h}_{T}$
and $\boldsymbol{h}_{I}$ to reconstruct the inputs $\boldsymbol{x}_{T}$,
$\boldsymbol{x}_{I}$ in $\hat{\boldsymbol{x}}_{T}$,
$\hat{\boldsymbol{x}}_{I}$, respectively. Therefore:

$\displaystyle\boldsymbol{h}_{T}=E_{AE}(\boldsymbol{x}_{T})$ (1)
$\displaystyle\boldsymbol{h}_{I}=E_{CAE}(\boldsymbol{x}_{I})$ (2)
$\displaystyle\hat{\boldsymbol{x}}_{T}=D_{AE}(\boldsymbol{h}_{T})$ (3)
$\displaystyle\hat{\boldsymbol{x}}_{I}=D_{CAE}(\boldsymbol{h}_{I})$ (4)

When training the $AE$ and the $CAE$ we aim to minimize the distance between
its inputs and outputs over all samples, using as reconstruction loss
functions $L_{T}$ and $L_{I}$ for the tabular and imaging modalities,
respectively. We constrain the dimension of latent spaces $\boldsymbol{h}_{T}$
and $\boldsymbol{h}_{I}$ to be lower than input data $\boldsymbol{x}_{T}$ and
$\boldsymbol{x}_{I}$, respectively, forcing both the $AE$ and the $CAE$ to
capture the most salient features of the data. This is a well-known approach
to avoid identity mapping [30].

#### Classifier

The two embeddings $\boldsymbol{h}_{T}$ and $\boldsymbol{h}_{I}$ are
concatenated in $\boldsymbol{h}$ and used as input to the fully connected
$C_{MLP}$, which performs the classification task returning $\boldsymbol{y}$,
So that:

$\boldsymbol{y}=C_{MLP}(\boldsymbol{h})$ (5)

The goal of this block is to minimize the classification error with a
classification loss $L_{C}$. Note that the final layer of the $C_{MLP}$ uses
the Softmax activation function, such that

$\sum_{i=1}^{c}\boldsymbol{y}_{i}=1$ (6)

This implies that $\boldsymbol{y}$ can be considered as an estimate of the
posterior probability.

#### End-to-end training

In this way, the network’s training can be back-propagated in an end-to-end
manner, via a linear combination of the three loss functions:

$L=\gamma_{T}L_{T}+\gamma_{I}L_{I}+\gamma_{C}L_{C}$ (7)

where $\gamma_{T}$, $\gamma_{I}$, and $\gamma_{C}$ are parameters, which
regulate the importance of each loss function.

This approach has the beneficial effect of being able to learn embedded
features in an end-to-end way, which are jointly used to perform data
reconstruction and classification, minimizing the reconstruction loss of $AE$
and the $CAE$ and the classification loss of the $C_{MLP}$. Our key idea is
that the co-learning of the $AE$, the $CAE$ and the $C_{MLP}$ is beneficial to
learn features from the tabular and imaging modality to obtain a
classification and a good reconstruction useful for the explainability,
presented in section 3.4.

###  3.3 Three-stage training

Given the complex structure of the architecture proposed, we train the network
with a three-stage procedure, which adapts the $\gamma_{T}$, $\gamma_{I}$,
$\gamma_{C}$ parameters in way to concentrate the training on different parts
of the network. The three stages are:

  1. 1.

Setting $\gamma_{T}=1$, $\gamma_{I}=0$ and $\gamma_{C}=0$ to train only the
weights of the $AE$;

  2. 2.

Setting $\gamma_{T}=0$, $\gamma_{I}=1$ and $\gamma_{C}=0$ to train only the
weights of the $CAE$;

  3. 3.

Setting $\gamma_{T}=1$, $\gamma_{I}=1$ and $\gamma_{C}=1$ to train all the
weights of network.

The main idea is to help the training of the $C_{MLP}$ classifier, giving
initialization weights that constrict an optimal modality embedding for
reconstruction, expressing a good summary of the data. Notice also that, given
the architecture of the network, it is irrelevant if we invert stage 1 and 2
since the $AE$ and $CAE$ have no weights in common. In step 3 we decided to
set all the parameters to $1$ so that all the tasks would have equal weight.

###  3.4 MXAI

We use a gradient update, also referred to as latent shift, that can transform
the latent representation of the inputs to exaggerate or curtail the features
used for prediction. Via the latent shift explanations we obtain both modality
importance and feature importance for each prediction.

#### Latent shift

Figure 2: Schematic view of the MXAI framework.

The only requisite to apply latent shift to a network is of having all the
network components, which receive the latent vector $\boldsymbol{h}$, to be
differentiable. With the $AE$, a $CAE$ and a $C_{MLP}$ we satisfy this
requisite and, in addition, they are simple to implement and train. Once these
components are trained, we extract the explanation as shown in Figure 2.
Multimodal input instances $\boldsymbol{x}_{T}$ and $\boldsymbol{x}_{I}$ are
encoded producing the multimodal latent representations $\boldsymbol{h}_{T}$
and $\boldsymbol{h}_{I}$, which are combined into $\boldsymbol{h}$, as already
described. Perturbations of this latent embedding are computed via

$\boldsymbol{h}^{\lambda}=\boldsymbol{h}-\lambda\pdv{C_{MLP}(\boldsymbol{h})}{\boldsymbol{h}}$
(8)

where $\lambda\in\mathbb{R}$ is a parameter establishing how much the original
embedding is modified. With $\lambda>0$, we expect that
$C_{MLP}(\boldsymbol{h}^{\lambda})$ would provide a prediction
$\boldsymbol{y}^{\lambda}$ so that

$\max({\boldsymbol{y}})\geq{\boldsymbol{y}^{\lambda}(\operatorname*{arg\,max}(\boldsymbol{y}))}$
(9)

This implies that, as $\lambda$ increases, we expect a flip of the predicted
class. In other words, guided by the direction of variation of the output in
the latent space determined by the gradient of network output, we are
interested in determining the $\lambda$ value for which a classification label
flip occurs. With too small values of $\lambda$, for the smoothness principle,
the difference between the original modality input and the reconstruction will
not be large enough to change the prediction of the model. On the contrary,
too large values of $\lambda$ would distort the reconstruction so much that it
will not be useful for explainability. Thus, to find the value of $\lambda$
where the class flip occurs, we use an iterative search that, starting from
$\lambda=0$, and using a fixed step heuristically set to $10$, increases
$\lambda$ until $\boldsymbol{y}^{\lambda}\neq\boldsymbol{y}$.

We produce $\lambda$-shifted counterfactual multimodal reconstructions
$\boldsymbol{x}_{T}^{\lambda}$, $\boldsymbol{x}_{I}^{\lambda}$ and output
probabilities $\boldsymbol{y}^{\lambda}$ defined as

$\displaystyle\hat{\boldsymbol{x}}_{T}^{\lambda}=D_{AE}(\boldsymbol{h}_{T}^{\lambda})$
(10)
$\displaystyle\hat{\boldsymbol{x}}_{I}^{\lambda}=D_{CAE}(\boldsymbol{h}_{I}^{\lambda})$
(11) $\displaystyle\boldsymbol{y}^{\lambda}=C_{MLP}(\boldsymbol{h}^{\lambda})$
(12)

where $\boldsymbol{h}_{T}^{\lambda}$ and $\boldsymbol{h}_{I}^{\lambda}$ are
given by:

$\displaystyle\boldsymbol{h}_{T}^{\lambda}=\boldsymbol{h}_{T}-\lambda\pdv{C_{MLP}(\boldsymbol{h})}{\boldsymbol{h}_{T}}$
(13)
$\displaystyle\boldsymbol{h}_{I}^{\lambda}=\boldsymbol{h}_{I}-\lambda\pdv{C_{MLP}(\boldsymbol{h})}{\boldsymbol{h}_{I}}$
(14)

so that $\boldsymbol{h}^{\lambda}$ is the concatenation of
$\boldsymbol{h}_{T}^{\lambda}$ and $\boldsymbol{h}_{I}^{\lambda}$.

It is worth noting that finding an informative latent space relies on the
quality of the $AE$ and $CAE$. This justifies even more the use of the three-
stage training, facilitating the training of the $AE$ and $CAE$.

#### Modality importance

Since the multimodal embedding $\boldsymbol{h}$ is composed by the
concatenation of the unimodal embeddings $\boldsymbol{h}_{T}$ and
$\boldsymbol{h}_{I}$, we know to which modality each element of
$\boldsymbol{h}$ is associated to. Once calculated
$\boldsymbol{h}_{T}^{\lambda}$ and $\boldsymbol{h}_{I}^{\lambda}$, we compute
the modality normalized absolute differences, to understand how much each
element has been shifted:

$\displaystyle\Delta_{T}=\frac{{||\boldsymbol{h}_{T}-\boldsymbol{h}_{T}^{\lambda}||_{1}}}{n}$
(15)
$\displaystyle\Delta_{I}=\frac{{||\boldsymbol{h}_{I}-\boldsymbol{h}_{I}^{\lambda}||_{1}}}{m}$
(16)

where $n$ and $m$ denote the number of elements in each vector, and
$||.||_{1}$ denotes the $l_{1}$-$norm$. Hence, $\Delta_{T}$ and $\Delta_{I}$
express the importance of each modality: the more a modality embedding has
changed, the more important it is for the classification of a given sample.

#### Features importance

Similar to the modality importance, we now focus on an approach to reveal
which features per modality are more important for the classification of a
certain instance. Using the shifted reconstructions
$\hat{\boldsymbol{x}}_{T}^{\lambda}$ and $\hat{\boldsymbol{x}}_{I}^{\lambda}$,
we compute the absolute differences with the original reconstructions
$\hat{\boldsymbol{x}}_{T}$ and $\hat{\boldsymbol{x}}_{I}$

$\displaystyle\hat{\boldsymbol{\Delta}}_{T}=|\hat{\boldsymbol{x}}_{T}-\hat{\boldsymbol{x}}_{T}^{\lambda}|$
(17)
$\displaystyle\hat{\boldsymbol{\Delta}}_{I}=|\hat{\boldsymbol{x}}_{I}-\hat{\boldsymbol{x}}_{I}^{\lambda}|$
(18)

Note that $\hat{\boldsymbol{\Delta}}_{T}$ and $\hat{\boldsymbol{\Delta}}_{I}$
make us understand for each feature how much it has changed for the
classification shift. The more a feature changes, the more important it is for
the classification. This works for both modalities, resulting
$\hat{\boldsymbol{\Delta}_{T}}$ to be an importance vector for the tabular
modality and $\hat{\boldsymbol{\Delta}_{I}}$ to be an importance matrix for
the imaging modality.

#### Putting our method in the taxonomy

Following the taxonomy introduced in section 2 and originally presented in
[26], our proposal is an hybrid between during and post-hoc modelling as it
exploits a specific network architecture using both perturbations and
backpropagation methods to extract the explanations. In particular our method
uses counterfactual explanations, specifying the minimal desired changes
required to flip the decision, mapping the class-specific and discriminative
features of each modality. In addition, our MXAI method is local, since we are
interested in explaining how the model functions at instance level. Finally,
our method is model-specific since its architecture is constructed in a way to
output the explanations.

##  4 Experimental configuration

In this section we introduce the used dataset and how the two modalities are
pre-processed to train the network. Then, we deepen the validation phase of
the proposed MXAI method where we conducted a reader study with four COVID-19
expert radiologists.

###  4.1 Dataset

For the last two years the world has been struck by the COVID-19 pandemic
causing millions of cases and deaths. During this period, many researchers
practitioners and companies have developed novel AI methods and tools to
combat the rising of the pandemic by deepening the virus’s understanding. Many
studies have focused their attention on unimodal data using CXR, CT or
clinical examinations to replace or to supplement the reverse transcriptase-
polymerase chain reaction tests. But given the multimodal nature of medicine,
both imaging data and clinical information can help radiologists and
practitioners on determining the source of symptoms, stratifying the disease
severity, and establishing the best treatment plan for the patient’s specific
needs.

We use the AIforCOVID imaging archive [9] because it is the only publicly
available multimodal dataset on COVID-19 stratification, as shown in the
survey [16]. The archive includes clinical data (tabular modality) and CXR
scans (imaging modality) of 820 patients recorded from six different Italian
hospitals. The interested readers can refer to [9, 31, 32, 33] for further
details.

The patients’ data were collected at the time of hospitalization if the TR-PCR
test resulted positive to the SARS-CoV-2 infection. All the patients were
assigned to the mild or severe class, on the basis of the clinical outcome.
The mild group consists of the patients that were sent back to domiciliary
isolation or were hospitalized without any ventilatory support, whilst the
severe group is composed of patients who required non-invasive ventilation
support, intensive care unit or patients who deceased.

###  4.2 Pre-processing

We applied the same pre-processing procedure and validation approach presented
in [9] to avoid any performance bias, which are now briefly summarized for the
sake of presentation.

#### Tabular data

We use the 34 clinical descriptors indicated in [9] which are not direct
indicators of the prognosis. Missing data were imputed using the mean and the
mode for continuous and categorical variables, respectively. A min-max scaler
was applied along the variables to have the features all in the same range
$[0,1]$.

#### Imaging data

This modality consists of CXR scans, which were processed by extracting the
segmentation mask of lungs, using a U-Net [34] trained on two non-COVID-19
lung datasets [35, 36]. The mask was used to extrapolate the minimum squared
bounding box containing both lungs. The extracted box was then resized to
$224\times 224$ matrix, and normalized with a min-max scaler bringing the
pixel values in the range $[0,1]$.

###  4.3 Implementation setup

Here we describe the architectures of the three blocks of the model, as well
as the parameters and settings used during training.

The AE’s input and output layers consist of 34 (one for each feature) and 2
(one for each class) neurons, respectively. Its encoder and decoder are
composed of fully connected hidden layers activated by ReLU functions. We
opted to use such architectures since these feed-forward networks are able to
learn a low-dimensional representation before being fused with the other
modality [37]. Both $E_{AE}$ and $D_{AE}$ have 6 hidden layers and $n=8$. The
loss function $L_{T}$ is the mean squared error (MSE).

The $CAE$ is a 2D ResNet101 [38], which we selected because of the skip
connections that mitigate the problem of the vanishing gradient, ensuring high
fidelity image reconstruction. Both the input and the output of the network
have a $224\times 224$ dimension, so that the dimension of the embedding is
$m=4608$. The dimensions the embeddings of both the $AE$ and the $CAE$ were
chosen small enough to prevent the curse of dimensionality. To facilitate the
reconstruction training, this model was pre-trained trained on 4 different CXR
datasets: NIH [39], PC [40], RSNA [41], and MIMIC [42] that in total account
for a total of 674525 scans. For consistency, the corresponding loss function
$L_{I}$ is the MSE.

Let us now focus on the $C_{MLP}$ classifier: its input and output layers
consist of 4616 and 2 neurons (one for each class), respectively. It is
composed of 7 fully connected hidden layers (with 512, 256, 128, 64, 32, 16, 8
neurons, respectively) activated by ReLU functions, with a Softmax activation
in the output layer. We design this structure to gradually learn the
classification from the multimodal embedding. The loss function $L_{C}$ is the
cross-entropy.

For all the three stages of the training, introduced in section 3.3, we adopt
the same training procedure of [9], now summarized. To prevent overfitting of
the $CAE$, we applied the following image random transformations: horizontal
or vertical shift ($-20\leq$ pixels $\leq 20$), random zoom ($0.9\leq$ factor
$\leq 1.1$), vertical flip, random rotation ($-15^{\circ}\leq$ angle $\leq
15^{\circ}$), and elastic transform ($20\leq\alpha\leq 40$, $\sigma=7$). No
augmentation was applied on the tabular data. The loss functions $L_{T}$,
$L_{I}$, $L_{C}$ and $L$ are regulated by an Adam optimizer with an initial
learning rate of 0.001, which is scheduled to reduce by an order of magnitude
every time the minimum validation loss does not change for 10 consecutive
epochs. To prevent overtraining and overfitting we fixed the number of maximum
epochs to 300, with an early stopping of 25 epochs on the validation loss.

###  4.4 Validation approach

To understand the robustness of the model we trained the network in 10-fold
stratified cross-validation (CV), and leave-one-center-out CV (LOCO),
following the same experimental procedure described in [9], thus ensuring a
fair competition between the approaches. In CV, the fold distribution of the
training, validation and testing sets is 70%-20%-10%, respectively. In LOCO
validation we study how the models generalize to different data sources, since
in each fold the test set contains all the samples belonging to only one of
the six hospitals that, of course, are not in the training and validation
sets.

All the experiments were performed using a batch size of 16 on a NVIDIA TESLA
A100 GPU with 32 GB of memory, using PyTorch as the coding library for deep
learning.

###  4.5 Sanity check

To study the validity of the proposed MXAI method, we conducted a reader study
with four radiologists assessing the prognosis of 96 patients randomly
extracted. Each radiologist has more than 10 years of experience. The
radiologists $R_{1}$, $R_{2}$, $R_{3}$, $R_{4}$ were presented with a survey
that has two aims. The first is to compare our method’s classification
performance with the one of human experts. The second is to understand if the
importance metrics $\Delta_{T}$, $\Delta_{I}$, $\hat{\boldsymbol{\Delta}}_{T}$
and $\hat{\boldsymbol{\Delta}}_{I}$, returned by our method, are coherent with
the ones selected by the radiologists, which we denote as
$\Delta_{T}^{R_{i}}$, $\Delta_{I}^{R_{i}}$,
$\hat{\boldsymbol{\Delta}}_{T}^{R_{i}}$ and
$\hat{\boldsymbol{\Delta}}_{I}^{R_{i}}$. In particular, $\Delta_{T}^{R_{i}}$,
$\Delta_{I}^{R_{i}}$ are the modality importance for the $i^{th}$ radiologist,
and $\hat{\boldsymbol{\Delta}}_{T}^{R_{i}}$,
$\hat{\boldsymbol{\Delta}}_{I}^{R_{i}}$ are the unimodal feature importance
vector and matrix for the $i^{th}$ radiologist. The survey was executed in
double-blind, where no interaction between the radiologists was permitted.

In the survey, each radiologist observed both data modalities at the same time
for each patient and performs the prognosis task. Afterwards, the radiologists
have to attribute an importance score, on a scale from 1 to 5, indicating how
much significant each modality was for the prognosis task. The grading of the
scores are: 1 insignificant, 2 a bit significant, 3 neutral, 4 significant, 5
important. A Softmax activation is applied constraining such values on the
range $[0,1]$, where 0 means that the considered modality has no importance
and 1 attributes the maximum importance. As mentioned before, these modality
importance are denoted as $\Delta_{T}^{R_{i}}$, $\Delta_{I}^{R_{i}}$. The
radiologist has the possibility to attribute the same importance to each
modality if he/she believes that, for that patient, both modalities had the
same impact in the decision.

Then, to understand the most important features for each modality, we asked
the radiologist to select the clinical variables and to segment the areas of
interest in the X-ray image most useful to stratify the patient, collecting
$\hat{\boldsymbol{\Delta}}_{T}^{R_{i}}$ and
$\hat{\boldsymbol{\Delta}}_{I}^{R_{i}}$, which are boolean, where elements
equal to $1$ correspond to important features. On both
$\hat{\boldsymbol{\Delta}}_{T}^{R_{i}}$ and
$\hat{\boldsymbol{\Delta}}_{I}^{R_{i}}$ a min-max normalization is applied on
each instance, putting the elements on the range $[0,1]$, where 0 means that
the considered feature has no importance and 1 attributes the max importance.

In the case of modality importance, we would expect a high intersection
between the information reported by the radiologists $\Delta_{T}^{R_{i}}$ and
$\Delta_{I}^{R_{i}}$ with the output of our method $\Delta_{T}$ and
$\Delta_{I}$, respectively; the same holds in the case of unimodal feature
importance, when comparing $\hat{\boldsymbol{\Delta}}_{T}^{R_{i}}$ and
$\hat{\boldsymbol{\Delta}}_{I}^{R_{i}}$ with $\hat{\boldsymbol{\Delta}}_{T}$
and $\hat{\boldsymbol{\Delta}}_{I}$, respectively.

The surveys were executed on Google Forms, and the tool utilized to show and
to segment the CXR scans was ITK-SNAP [43].

###  4.6 Statistical analysis

The accuracy, the sensitivity and the specificity are the evaluation metrics
used to assess the classification performance, as in [9]. To asses if there
exists a difference between the performance of our model and the baseline
model we apply the one-way ANOVA and, to interpret the statistical
significance, we used the pairwise Tukey test with a Bonferroni $p$-value
correction at $\alpha=0.05$.

As described at the end of the previous section, we validate the MXAI modality
importance $\Delta_{T}$, $\Delta_{I}$ and the feature importance
$\hat{\boldsymbol{\Delta}}_{T}$, $\hat{\boldsymbol{\Delta}}_{I}$, comparing
them with the importance proposed by the radiologists $\Delta_{T}^{R_{i}}$,
$\Delta_{I}^{R_{i}}$ and $\hat{\boldsymbol{\Delta}}_{T}^{R_{i}}$,
$\hat{\boldsymbol{\Delta}}_{I}^{R_{i}}$, respectively.

For the modality importance, we calculate the Pearson correlation $\rho$, and
the paired sample t-test between the vector of importance modality
($\Delta_{T}$ and $\Delta_{I}$) over the instances given by our method, with
the corresponding importance ($\Delta_{T}^{R_{i}}$ and $\Delta_{I}^{R_{i}}$)
vector reported by the radiologists. With the resulting statistics we can
comprehend the measure of dependency between our method and the radiologists.
The higher $\rho$, the more our explanations are coherent with the clinical
findings, if the resulting t-test is not statistically significant ($p$-value
$>0.05$).

Turning our attention to the feature importance, we compute the intersection
over union (IoU) between the feature importance proposed by the radiologists
($\hat{\boldsymbol{\Delta}}_{T}^{R_{i}}$ and
$\hat{\boldsymbol{\Delta}}_{I}^{R_{i}}$) and the importance resulting from the
latent shift ($\hat{\boldsymbol{\Delta}}_{T}$ and
$\hat{\boldsymbol{\Delta}}_{I}$), respectively. For the tabular modality
$IoU_{T}$, we take the important features presented by the radiologists
$\hat{\boldsymbol{\Delta}}_{T}^{R_{i}}$ and the binarized feature vector
$\hat{\boldsymbol{\Delta}}_{T}^{b}$ (such that the values of
$\hat{\boldsymbol{\Delta}}_{T}$ which are $<0.5$ are set to $0$ and the ones
$\geq 0.5$ are set to $1$), and compute

$IoU_{T}=\frac{\hat{\boldsymbol{\Delta}}_{T}^{R_{i}}\cap\hat{\boldsymbol{\Delta}}_{T}^{b}}{\hat{\boldsymbol{\Delta}}_{T}^{R_{i}}\cup\hat{\boldsymbol{\Delta}}_{T}^{b}}$
(19)

The higher the metric, the more concurrences there are between our method and
the human annotation. Similarly, when analyzing the imaging modality $IoU_{I}$
we take the segmented mask returned by the radiologists
$\hat{\boldsymbol{\Delta}}_{I}^{R_{i}}$ and the binarized attribution map
$\hat{\boldsymbol{\Delta}}_{I}^{b}$ (such that the values of
$\hat{\boldsymbol{\Delta}}_{I}$ which are $<0.5$ are set to $0$ and the ones
$\geq 0.5$ are set to $1$), and compute

$IoU_{I}=\frac{\hat{\boldsymbol{\boldsymbol{\Delta}}}_{I}^{R_{i}}\cap\hat{\boldsymbol{\boldsymbol{\Delta}}}_{I}^{b}}{\hat{\boldsymbol{\boldsymbol{\Delta}}}_{I}^{R_{i}}\cup\hat{\boldsymbol{\boldsymbol{\Delta}}}_{I}^{b}}$
(20)

As before, the higher the metric, the more concurrences there are between our
method and the human annotations.

##  5 Results and discussion

In this section we present the results obtained, dividing the discussion into
five subsections that, in order, deal with the classification and
reconstruction performance, the three-stage training assessment, MXAI
performance, an ablation study and forthcoming clinical applications.

Table 1: Classification performance. Model | Validation | Accuracy | Sensitivity | Specificity
---|---|---|---|---
Our proposal (three-stage training) | CV | 76.75$\pm$5.32 | 78.58$\pm$6.48 | 74.55$\pm$5.86
LOCO | 74.21$\pm$6.08 | 76.73$\pm$18.88 | 68.40$\pm$15.46
Survey | 76.77 | 78.54 | 74.57
AIforCOVID [9] | CV | 76.90$\pm$5.40 | 78.80$\pm$6.40 | 74.70$\pm$5.90
LOCO | 74.30$\pm$6.10 | 76.90$\pm$18.90 | 68.50$\pm$15.50
$R_{1}$ | Survey | 68.75 | 43.75 | 93.75
$R_{2}$ | Survey | 72.92 | 70.83 | 75.00
$R_{3}$ | Survey | 76.04 | 70.83 | 81.25
$R_{4}$ | Survey | 72.92 | 62.50 | 83.33
Our proposal (one-stage training) | CV | 70.38$\pm$1.78 | 72.57$\pm$1.72 | 68.62$\pm$1.12
LOCO | 68.35$\pm$1.17 | 70.92$\pm$1.08 | 62.16$\pm$1.89
Survey | 70.48 | 72.51 | 68.67

###  5.1 Classification and reconstruction performance

Table 1 shows the classification performance and its columns specify: the
learning model (human radiologists included), the validation approach (CV,
LOCO or the reduced set of images used for the survey), the evaluation metrics
employed, i.e., accuracy, sensitivity and specificity, for which we show the
mean and the standard deviation of the metric in CV and in LOCO. The table is
organized into four horizontal sections: the first, the second and the fourth
report the performance attained by learning models, whereas the third shows
the performance of the four radiologists. In particular, the first section
shows results of our proposal and the second includes the best baseline model
presented by [9], which were attained by the hybrid approach.

Given that the goal of our method is to increase the explainability of the
model and not to necessarily increase the performance, we first verify that
with the co-learning we do not have a drop in performance with respect to the
baseline [9]. The results show that our model, even if it co-learns two tasks
at once, obtains a performance similar to [9]. In fact, the differences
between our model and the baseline on all metrics, in both CV and LOCO, are
not statistically significant ($p$-value $>0.05$).

We now compare our results with those of the radiologists: our proposal
provides larger accuracy and sensitivity, while the specificity is lower. This
happens because predicting the prognosis of a patient affected by COVID-19 is
a difficult task, giving a hint that AI could aid practitioners in the
decision-making process.

Table 2: Reconstruction performance. Model | Validation | Modality | MSE
---|---|---|---
Our proposal (three-stage training) | CV | $T$ | 0.04$\pm$0.01
LOCO | $T$ | 0.05$\pm$0.02
Survey | $T$ | 0.04
CV | $I$ | 0.03$\pm$0.01
LOCO | $I$ | 0.04$\pm$0.02
Survey | $I$ | 0.03
Our proposal (one-stage training) | CV | $T$ | 0.09$\pm$0.04
LOCO | $T$ | 0.11$\pm$0.05
Survey | $T$ | 0.09
CV | $I$ | 0.07$\pm$0.02
LOCO | $I$ | 0.09$\pm$0.03
Survey | $I$ | 0.07

Let us recall that our method is jointly trained to classify and reconstruct
the inputs via the autoencoders: for this reason in Table 2 we show the MSE of
$AE$ and $CAE$, i.e., the two autoencoders working with the $T$ and $I$
modalities, respectively. As in Table 1, here we have a similar row-column
organization. The value of the MSE suggests that the reconstruction for both
modalities is of high quality, guaranteeing good interpretability for the MXAI
method.

###  5.2 Three-stage training assessment

To validate the three-stage training introduced in section 3.3, we compare the
classification and reconstruction performance with those attained adopting a
one-stage training, which consists of skipping phases 1 and 2 of our method
and directly train, in an end-to-end manner, the entire network with the
combined loss $L$, without any pre-training. The corresponding results are
shown in the last section of Tables 1 and 2. In the case of classification
performance (Table 1), we observe that the one-stage training provides lower
performance than our three-stage proposal, whatever the performance metric and
whatever the validation approach. Furthermore, such performance differences
are all statistically significant ($p$-value $<0.01$). Similar considerations
hold in the case of reconstruction performance (Table 2). These results
confirm the usefulness of the three-stage training procedure, which aids the
multimodal joint model in converging to a better solution.

Table 3: $\rho$ (on the lower triangular) and the corresponding t-test $p$-value (upper triangular) of the modality importance, computed for each pair between our model and the radiologists. | Our proposal | $R_{1}$ | $R_{2}$ | $R_{3}$ | $R_{4}$
---|---|---|---|---|---
Our proposal | - | 0.26 | 0.50 | 0.37 | 0.42
$R_{1}$ | 0.78 | - | 0.28 | 0.32 | 0.45
$R_{2}$ | 0.84 | 0.90 | - | 0.35 | 0.50
$R_{3}$ | 0.77 | 0.82 | 0.78 | - | 0.41
$R_{4}$ | 0.79 | 0.77 | 0.83 | 0.85 | -

###  5.3 MXAI performance

We now focus our attention on the validation of the explanations provided by
our proposal. In this respect, using the patients included in the survey, we
now compare the modality and the feature explanations of our model with
respect to the importance reported by the radiologists, as presented in
section 4.5. According to section 4.6, Table 3 shows the Pearson correlation
$\rho$ and the corresponding t-test $p$-values computed between the importance
vector of a modality reported by our method and the importance vector reported
by each radiologist. These results reveal a high measure of dependency between
our method and the radiologists and among the radiologists, suggesting that
our model gives reasonable modality importance while producing the prognosis.

Table 4: $IoU_{T}$ (lower triangular) and the $IoU_{I}$ (upper triangular) of the feature importance, computed for each pair between our model and the radiologists. | Our proposal | $R_{1}$ | $R_{2}$ | $R_{3}$ | $R_{4}$
---|---|---|---|---|---
Our proposal | - | 59.62$\pm$3.13 | 62.97$\pm$2.61 | 63.77$\pm$2.25 | 64.63$\pm$2.76
$R_{1}$ | 52.37$\pm$3.12 | - | 60.56$\pm$2.78 | 63.42$\pm$3.39 | 61.81$\pm$3.28
$R_{2}$ | 54.72$\pm$2.86 | 52.37$\pm$2.78 | - | 60.98$\pm$3.76 | 62.44$\pm$2.99
$R_{3}$ | 53.52$\pm$3.21 | 54.69$\pm$3.42 | 51.23$\pm$2.98 | - | 63.36$\pm$3.64
$R_{4}$ | 51.31$\pm$2.69 | 52.73$\pm$3.31 | 54.66$\pm$2.79 | 55.43$\pm$3.05 | -

Let us now consider the unimodal feature importance: in this case Table 4
shows the $IoU_{T}$ and $IoU_{I}$ for each possible pair between our model and
the radiologists. As mentioned in section 4.6, these metrics permit us
understand the coherence between the returned feature importance. These scores
not only show that the radiologists have a fairly high degree of intersection
of important features among each other, but also that the degree of
intersection is of the same magnitude even with our proposal. This implies
that our model concentrates on the relevant features of each modality when
making the decision on the prognosis.

Table 5: Classification performance in the ablation study Model | Validation | Modality | Accuracy | Sensitivity | Specificity
---|---|---|---|---|---
Our Proposal | CV | $T$ | 75.78$\pm$0.75 | 76.63$\pm$0.71 | 74.74$\pm$1.02
LOCO | $T$ | 73.48$\pm$3.20 | 69.87$\pm$3.11 | 79.56$\pm$8.60
Survey | $T$ | 75.69 | 76.65 | 74.71
CV | $I$ | 74.14$\pm$1.03 | 74.57$\pm$1.78 | 73.96$\pm$1.21
LOCO | $I$ | 70.46$\pm$1.06 | 72.03$\pm$1.02 | 69.55$\pm$1.61
Survey | $I$ | 74.32 | 74.42 | 73.88
AIforCOVID [9] | CV | $T$ | 75.70$\pm$0.80 | 76.00$\pm$0.70 | 75.40$\pm$1.10
LOCO | $T$ | 73.40$\pm$4.40 | 69.90$\pm$15.80 | 79.50$\pm$13.60
CV | $I$ | 74.20$\pm$1.00 | 74.80$\pm$1.90 | 73.80$\pm$1.30
LOCO | $I$ | 70.50$\pm$1.00 | 72.00$\pm$1.10 | 69.60$\pm$1.50
Table 6: Reconstruction performance of the ablation study. Model | Data | Modality | MSE
---|---|---|---
Our Proposal | CV | $T$ | 0.03$\pm$0.01
LOCO | $T$ | 0.04$\pm$0.02
Survey | $T$ | 0.03
CV | $I$ | 0.02$\pm$0.01
LOCO | $I$ | 0.03$\pm$0.02
Survey | $I$ | 0.02
Table 7: $IoU_{T}$ (lower triangular) and the $IoU_{I}$ (upper triangular) of the feature importance for models trained in ablation, computed for each pair between our model and the radiologists.. | Our proposal | $R_{1}$ | $R_{2}$ | $R_{3}$ | $R_{4}$
---|---|---|---|---|---
Our proposal | - | 60.31$\pm$2.98 | 62.45$\pm$2.31 | 63.57$\pm$2.49 | 64.22$\pm$3.58
$R_{1}$ | 53.63$\pm$2.45 | - | 60.56$\pm$2.20 | 62.10$\pm$3.10 | 61.90$\pm$2.98
$R_{2}$ | 54.65$\pm$2.95 | 52.45$\pm$3.21 | - | 61.07$\pm$3.30 | 62.90$\pm$3.20
$R_{3}$ | 54.49$\pm$3.03 | 53.99$\pm$3.32 | 50.89$\pm$3.00 | - | 62.88$\pm$3.50
$R_{4}$ | 50.40$\pm$3.01 | 52.41$\pm$3.40 | 52.54$\pm$3.11 | 55.30$\pm$2.87 | -

###  5.4 Ablation study

We now discuss what happens when only one modality is available. To this end,
we ran two experiments: in the first we removed the $AE$ and we trained again
the other part of the model, i.e., we worked only with the imaging modality
disregarding the tabular clinical data. In the second we flipped the ablation,
removing the $CAE$ and, thus, we did not consider the images. Table 5 shows
the results we achieved in the case of experiments ran in CV, LOCO and using
the survey images. Furthermore, the third columns specifies the modality used.
As before, we also show the hybrid baseline model presented in [9], which is
trained only on one modality. The results of our proposal shows that using
only the tabular clinical data or only the imaging data provides similar
results, which do not statistically differ from each other, whatever the
performance score considered ($p$-value $>0.05$). Furthermore, in comparison
with the performance of the full multimodal approach (first section of Table
1), we notice that both unimodal models report a statistically significant
drop in performance ($p$-value $<0.01$), whatever the validation approach or
the performance score. As before, we notice that our model, even if it co-
learns two tasks at once, obtains similar performance to the hybrid model,
i.e., the best baseline model of [9]. In fact, the difference between our
model and the baseline model on all metrics, in both CV and LOCO, is not
statistically significant ($p$-value $>0.05$).

For the sake of completeness, Table 6 shows the reconstruction results in
terms of MSE for each modality. In comparison with Table 2, as it would be
expected, we notice that the reconstruction error has significantly decreased
($p$-value $>0.05$) since the model can concentrate at one modality at a time,
making it easier to learn an efficient embedding mapping.

As before, we also investigate the results in terms of explainability.
Straightforwardly, in this case the modality importance does not make sense,
so that Table 7 reports the $IoU_{T}$ and the $IoU_{I}$ of new unimodal
feature importance. These results show that, even if we have a drop in
classification and reconstruction performance, the explanations are consistent
between the radiologists and between our method and the radiologists,
suggesting that the MXAI method is robust to a missing modality.

Figure 3: Example of feature importance for the tabular and the imaging
modality.

###  5.5 Plausible clinical applications

In a clinical practice scenario, we believe that our AI system can be the
precursor of subsequent multimodal research to predict the evolution of
COVID-19 because its classifications and explanations can aid the radiologists
in performing the prognosis task.

Indeed, on the one side, in [44] the authors showed that the rise of X-ray
severity over the course of COVID-19 infection increases the sensitivity of
COVID-19 detection using CXR over time ($55\%$ at $\leq 2$ days to $79\%$ at
$>11$ days), whilst it decreases the specificity ($83\%$ at $\leq 2$ days to
$70\%$ at $>11$ days). On the other side, as Table 1 shows, our proposal
provides a larger sensitivity than the radiologists, suggesting that it can
anticipate the evolution of positive COVID-19 cases, that is in an initial
phase of the disease when the patient accesses the emergency department, our
approach achieves a sensitivity ($78.56\%$) equal to that which the X-ray
alone shows after several days.

Furthermore, the proposed architecture has the beneficial feature to offer
transparent decisions since, for each patient, the radiologists can observe at
the same time the original data (both the clinical features and the X-ray
image), the modality importance $\Delta_{T}$ and $\Delta_{T}$, and the
unimodal feature importance $\hat{\boldsymbol{\Delta}}_{T}$ and
$\hat{\boldsymbol{\Delta}}_{I}$. With $\Delta_{T}$ and $\Delta_{T}$, the
radiologists would be guided to understand on which modality to concentrate
more on. Instead, with $\hat{\boldsymbol{\Delta}}_{T}$ and
$\hat{\boldsymbol{\Delta}}_{I}$ we guide the radiologist to concentrate on
certain clinical characteristics and on specific areas of the X-ray scan. In
the interface provided to the users, the tabular clinical data importance
$\hat{\boldsymbol{\Delta}}_{T}$ is represented as a bar-plot (Figure 3 left
panel), so that the longer the bar, the more important the clinical variable
is. The X-ray image importance map $\hat{\boldsymbol{\Delta}}_{I}$ is shown as
a heatmap on the original scan, which represents the relevance of each pixel
in the image for the prognosis task on a color scale ranging from blue (low
importance) to red (high importance), shown in the right panel of Figure 3.

##  6 Conclusion

In this work we presented an end-to-end multimodal architecture that jointly
learns modality reconstructions and multimodal classification using tabular
clinical and imaging data. With respect to the literature using such
modalities for medical tasks, we deem that our method is the only one which
offers intrinsic model-specific local multimodal explanations. In particular,
multimodal explanations are computed by exploiting the latent space learnt by
jointly training the end-to-end architecture and using a latent shift-based
counterfactual method. We tested our approach in the context of the COVID-19
pandemic using the AIforCOVID public dataset, which includes both X-ray and
clinical data. The extensive quantitative experimentation shows that the
latent space retains features useful to succeed both in a reconstruction and
classification task and, thus, resulting in an informative space for the
latent-shift method. Moreover, the sanity check, although very time-consuming,
was very useful since it showed a high intersection between the explanations
provided by the method and those of the radiologists, both for the modality
and the feature importance. Future work is directed to test our architecture
and MXAI method on other datasets.

## Acknowledgements

The computations of this work were enabled by resources provided by the
Swedish National Infrastructure for Computing (SNIC), partially funded by the
Swedish Research Council through grant agreement no. 2018-05973. This work was
partially supported by “University-Industry Educational Centre in Advanced
Biomedical 411 and Medical Informatics (CEBMI)” (Grant agreement no.
612462-EPP-1-2019-1-SK-EPPKA2-KA, 412 Educational, Audiovisual and Culture
Executive Agency of the European Union).

## References

  * [1] Shih-Cheng Huang et al.  Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines.  NPJ digital medicine, 3(1):1–9, 2020.
  * [2] Maithra Raghu et al.  A survey of deep learning for scientific discovery.  arXiv preprint arXiv:2003.11755, 2020.
  * [3] Tadas Baltrušaitis et al.  Multimodal machine learning: A survey and taxonomy.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423–443, 2018.
  * [4] Jiquan Ngiam et al.  Multimodal deep learning.  In International Conference on Machine Learning, 2011.
  * [5] Erico Tjoa et al.  A survey on explainable artificial intelligence (xai): Toward medical xai.  IEEE Transactions on Neural Networks and Learning Systems, 32(11):4793–4813, 2020.
  * [6] Alejandro Barredo Arrieta et al.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  Information Fusion, 58:82–115, 2020.
  * [7] Dhanesh Ramachandram et al.  Deep multimodal learning: A survey on recent advances and trends.  IEEE Signal Processing Magazine, 34(6):96–108, 2017.
  * [8] Dong Huk Park et al.  Multimodal explanations: Justifying decisions and pointing to the evidence.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8779–8788, 2018.
  * [9] Paolo Soda et al.  AIforCOVID: predicting the clinical outcomes in patients with COVID-19 applying AI to chest-X-rays: an italian multicentre study.  Medical Image Analysis, 74:102216, 2021.
  * [10] Alberto Signoroni et al.  BS-Net: Learning COVID-19 pneumonia severity on a large chest X-ray dataset.  Medical Image Analysis, 71:102046, 2021.
  * [11] Jocelyn S Zhu et al.  Deep-learning artificial intelligence analysis of clinical variables predicts mortality in COVID-19 patients.  Journal of the American College of Emergency Physicians Open, 1(6):1364–1373, 2020.
  * [12] Hazem Al-Najjar et al.  A classifier prediction model to predict the status of Coronavirus COVID-19 patients in South Korea.  European Review for Medical and Pharmacological Sciences, 2020.
  * [13] Xiang Bai et al.  Predicting COVID-19 malignant progression with AI techniques.  2020\.
  * [14] Wanshan Ning et al.  Open resource of clinical data from patients with pneumonia for the prediction of COVID-19 outcomes via deep learning.  Nature Biomedical Engineering, 4(12):1197–1207, 2020.
  * [15] Cong Fang et al.  Deep learning for predicting COVID-19 malignant progression.  Medical Image Analysis, 72:102096, 2021.
  * [16] Beatriz Garcia Santa Cruz et al.  Public COVID-19 x-ray datasets and their impact on model bias–a systematic review of a significant problem.  Medical Image Analysis, 74:102225, 2021.
  * [17] Sören Richard Stahlschmidt et al.  Multimodal deep learning for biomedical data fusion: a review.  Briefings in Bioinformatics, 23(2):bbab569, 2022.
  * [18] Weronika Hryniewska et al.  Checklist for responsible deep learning modeling of medical images based on COVID-19 detection studies.  Pattern Recognition, 118:108035, 2021.
  * [19] Lin Zou et al.  Ensemble image explainable AI (XAI) algorithm for severe community-acquired pneumonia and COVID-19 respiratory infections.  IEEE Transactions on Artificial Intelligence, 2022.
  * [20] Kazeem Moses Abiodun et al.  Explainable AI for fighting COVID-19 pandemic: opportunities, challenges, and future prospects.  Computational Intelligence for COVID-19 and Future Pandemics, pages 315–332, 2022.
  * [21] Mervyn D Cohen.  Accuracy of information on imaging requisitions: does it matter?  Journal of the American College of Radiology, 4(9):617–621, 2007\.
  * [22] William W Boonn et al.  Radiologist use of and perceived need for patient data access.  Journal of Digital Imaging, 22(4):357–362, 2009.
  * [23] Laure Wynants et al.  Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal.  BMJ, 369, 2020.
  * [24] Michael Roberts et al.  Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans.  Nature Machine Intelligence, 3(3):199–217, 2021.
  * [25] Riccardo Guidotti et al.  A survey of methods for explaining black box models.  ACM Computing Surveys (CSUR), 51(5):1–42, 2018.
  * [26] Gargi Joshi et al.  A review on explainability in multimodal deep neural nets.  IEEE Access, 9:59800–59821, 2021.
  * [27] Andreas Holzinger.  Explainable AI and multi-modal causability in medicine.  i-com, 19(3):171–179, 2020.
  * [28] Katharina Weitz et al.  “let me explain!”: exploring the potential of virtual agents in explainable AI interaction design.  Journal on Multimodal User Interfaces, 15(2):87–98, 2021.
  * [29] Ramprasaath R Selvaraju et al.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 618–626, 2017.
  * [30] Nitish Bahadur et al.  Dimension estimation using autoencoders and application.  In Deep Learning Applications, Volume 3, pages 95–121. Springer, 2022.
  * [31] Valerio Guarrasi et al.  Pareto optimization of deep networks for COVID-19 diagnosis from chest X-rays.  Pattern Recognition, 121:108242, 2022.
  * [32] Valerio Guarrasi et al.  Optimized fusion of CNNs to diagnose pulmonary diseases on chest X-Rays.  In International Conference on Image Analysis and Processing, pages 197–209. Springer, 2022.
  * [33] Valerio Guarrasi et al.  A multi-expert system to detect COVID-19 cases in x-ray images.  In 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS), pages 395–400. IEEE, 2021.
  * [34] Olaf Ronneberger et al.  U-Net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer, 2015.
  * [35] Junji Shiraishi et al.  Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules.  American Journal of Roentgenology, 174(1):71–74, 2000.
  * [36] Stefan Jaeger et al.  Two public chest X-ray datasets for computer-aided screening of pulmonary diseases.  Quantitative Imaging in Medicine and Surgery, 4(6):475, 2014.
  * [37] Xavier Glorot et al.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * [38] Kaiming He et al.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [39] Xiaosong Wang et al.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2097–2106, 2017.
  * [40] Aurelia Bustos et al.  Padchest: A large chest x-ray image dataset with multi-label annotated reports.  Medical Image Analysis, 66:101797, 2020.
  * [41] George Shih et al.  Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia.  Radiology. Artificial Intelligence, 1(1), 2019.
  * [42] Alistair EW Johnson et al.  MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs.  arXiv preprint arXiv:1901.07042, 2019.
  * [43] Paul A Yushkevich et al.  ITK-SNAP: An interactive tool for semi-automatic segmentation of multi-modality biomedical images.  In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 3342–3345. IEEE, 2016\.
  * [44] Stephanie Stephanie et al.  Determinants of Chest X-ray sensitivity for COVID-19: a multi-institutional study in the United States.  Radiology: Cardiothoracic Imaging.