11institutetext:  Faculty of EEMCS, University of Twente, 7500 AE Enschede,
The Netherlands
11email<EMAIL_ADDRESS>Hospital Group
Twente (ZGT), 7609 PP Almelo, The Netherlands
22email<EMAIL_ADDRESS>University of Marburg,
35037 Marburg, Germany
33email<EMAIL_ADDRESS>
# Feature importance to explain multimodal prediction models. A clinical use
case

Jorn-Jan van de Beld 1122 0000-0001-6220-0508 Shreyasi Pathak 11
0000-0002-6984-8208 Jeroen Geerdink 22 0000-0001-6718-6653 Johannes H. Hegeman
1122 0000-0003-2188-2738 Christin Seifert 33 0000-0002-6776-3868

###### Abstract

Surgery to treat elderly hip fracture patients may cause complications that
can lead to early mortality. An early warning system for complications could
provoke clinicians to monitor high-risk patients more carefully and address
potential complications early, or inform the patient. In this work, we develop
a multimodal deep-learning model for post-operative mortality prediction using
pre-operative and per-operative data from elderly hip fracture patients.
Specifically, we include static patient data, hip and chest images before
surgery in pre-operative data, vital signals, and medications administered
during surgery in per-operative data. We extract features from image
modalities using ResNet and from vital signals using LSTM. Explainable model
outcomes are essential for clinical applicability, therefore we compute
Shapley values to explain the predictions of our multimodal black box model.
We find that i) Shapley values can be used to estimate the relative
contribution of each modality both locally and globally, and ii) a modified
version of the chain rule can be used to propagate Shapley values through a
sequence of models supporting interpretable local explanations. Our findings
imply that a multimodal combination of black box models can be explained by
propagating Shapley values through the model sequence.

###### Keywords:

clinical decision support mortality prediction multimodal machine learning hip
fractures

##  1 Introduction

The absolute number of hip fractures in the older Dutch population ($\geq$ 65
years) almost doubled between 1981 and 2008 from 7,614 to 16,049 [25]. In
1997, it was estimated that in 2050 4.5 million people worldwide will suffer
from a hip fracture [11]. Mortality is high during the early postoperative
period with rates reported of up to 13.3% in the first 30 days after surgery
[13]. An accurate risk score computed using data known before surgery _(pre-
operative data)_ can lead to better information about the patient and be of
help in the treatment selection [14]. A risk score computed after surgery
additionally using data from surgery _(per-operative data)_ could provide an
early warning for complications, allowing for swift measures to mitigate the
consequences.

In recent years, machine learning (ML) has proven to be promising for clinical
practice to assist doctors in clinical decision-making and reduce workload
[3]. ML and deep neural network (DNN) models have been used to predict risks
of certain complications after surgery [33] and mortality using pre-operative
and per-operative data [19, 9], reporting ROC-AUC scores up to 0.91.
Multimodal models use multiple data sources to predict outcomes, while
unimodal models only consider a single data source. Clinicians use multiple
data sources in their decision-making, therefore multimodal models should in
theory be better at making these complex predictions [32].

It is crucial that decisions made by an ML model can be understood by
clinicians [31]. Some model types like decision trees are intrinsically
explainable, however, more complex models trade performance at the cost of
explainability [23]. In this paper, we investigate to what extent
explainability is hampered by using multimodal black box models that take
multiple data sources (modalities) as input. We apply model agnostic
explainability methods to understand the contribution of i) each modality and
ii) individual features.

The contributions of our paper are as follows:

  1. 1.

We present a neural model for predicting 30-day mortality of hip fracture
patients combining three different data modalities (images, time-series, and
structured data.

  2. 2.

We apply XAI techniques, specifically Shapley values, to validate the
mortality risk score given by our multimodal model.

  3. 3.

We present the application of a novel method, specifically Shapley value
propagation, to provide local explanations for a complex multimodal clinical
prediction model.

##  2 Related Work

Table 1: Overview of related work on short-term complication prediction.
Abbreviations: convolutional neural network (CNN), auto-encoder (AE), random
forest (RF), k-nearest neighbor (KNN), logistic regression (LR), support
vector machine (SVM), gradient-boosted trees (GBT), decision tree (DT), naive
bayes (NB), deep neural network (DNN), fully connected network (FC), long
short-term memory (LSTM) network, XGBoost (XGB).

Paper |  Studypopulation |  Predictiontarget(s) |  Data |  ML model(s)*
---|---|---|---|---
[24] |  Septic patients |  Short-term mortality |  Pre-operative static |  CNN, AE, RF, KNN, SVM
[10] |  Total shoulder arthroplasty |  Post-operative complications |  Pre- and per-operative static |  LR, GBT, RF, KNN, DT, NB
[27] |  Spinal metastasis surgery |  Short-term outcomes including mortality |  Pre-operative static |  LR
[19] |  Any surgery |  Post-operative mortality |  Pre- and per-operative static |  LR, DNN
[9] |  Surgery with tracheal intubation |  Post-operative short-term mortality |  Pre- and per-operative and temporal |  FC+LSTM+CNN
[5] |  Adult1 hip fracture patients |  Post-operative short-term mortality |  Pre-operative static |  LR, CNN
[17] |  Adult2 hip fracture patients |  Post-operative short-term mortality |  Pre-operative static |  LR
[22] |  Elderly3 hip fracture patients |  Post-operative short-term mortality |  Pre-operative static |  LR
[34] |  Elderly3 hip fracture patients |  Post-operative short-term mortality |  Pre-operative static and images |  LR, XGB, RF, SVM

1 Patients were at least 18 years old
2 Patients were at least 23 years old
3 Patients were at least 71 years old
* Models separated with a comma were compared and models connected with a plus sign were combined

In this section, we discuss the literature on three aspects; short-term
complication prediction after surgery, multimodal prediction model, and
explainability of ML models.

###  2.1 Short-term Complication Prediction

ML models have been applied for short-term complication prediction after
surgery, e.g. in septic [24] and hip fracture patients [5]. Table 1 summarises
a sample of studies in literature featuring short-term complication
prediction, showing pre-operative data as the most common input modality and
logistic regression (LR) models as the most common ML model used in existing
work.

Logistic regression models have been developed to predict early mortality
($<$30 days) after surgery in hip fracture patients using pre-operative static
patient data, where reported AUC scores range from 0.76 to 0.82 [22, 5, 17].
In addition to pre-operative data, Yenidogan et al. [34] also included pre-
operative hip and chest images in their multimodal model. They extracted
features from the images using convolutional neural networks (CNN) and trained
a random forest (RF) to predict early mortality, where they reported an AUC of
0.79. However, the effect of per-operative data on risk score prediction after
hip fracture surgery has not been investigated before.

Cao et al. developed a model for the prediction of 30-day mortality of adult
patients after hip fracture surgery using static data from 134,915 patients
[5]. They compared the performance of a CNN with LR and reported an AUC of
0.76 for both models. Yenidogan et al. addressed the 30-day mortality
prediction after hip fracture surgery specifically for elderly patients [34].
The authors exploited structured and image data available before surgery and
showed significant improvement compared to their baseline the Almelo hip
fracture score (AHFS) developed by [22]. They trained two CNNs to extract
features from hip and chest X-ray images, which were fed to a RF classifier
together with the structured data features. Their multimodal model (AUC=0.79)
outperformed the AHFS baseline (AUC of 0.72). Thus the authors concluded, that
the additional information from multiple modalities is beneficial for model
performance. In this paper, we add more modalities to further improve the
prediction of postoperative mortality.

###  2.2 Multimodal Prediction Models

Clinical models that combine multiple modalities outperform models restricted
to a single modality [8]. Multimodal models are commonly used for video
classification tasks, where audio and image data are processed concurrently
[21]. The approaches differ in the way they share information between
modalities within the neural network. Two common approaches are late and early
fusion. Late fusion combines the predictions of the unimodal models with no
further cross-modal information flow, while early fusion combines modalities
very early in the pipeline [21].

While early fusion allows for full information flow between modalities, it has
a high computational cost, due to the high number of neuron connections. Late
fusion has a low computational cost but does not enable the model to learn
cross-modal interactions.

###  2.3 Explainability

Models are required to be explainable to gain the trust of clinicians, where
knowing what features are most important to the model for its prediction is
crucial [31]. Furthermore, clinicians need to be able to justify their
decision-making towards patients and colleagues. ML models can either be
explained locally or globally [7]. Local explanations in a clinical setting
focus on justifying the prediction for a single patient, while global
explanations provide insight into general prediction tendencies for a larger
population.

Multiple methods are available to compute the feature importance in deep
learning models, most notably: LIME [26], deepLIFT [30], layer-wise relevance
propagation [2] and Shapley values [20]. In this paper, we use SHAP (SHapley
Additive exPlanations) to estimate the importance of input features and
modalities. Theoretically, all possible input feature subsets are considered
during the computation of Shapley values, which makes them less sensitive to
multicollinearity and thereby suitable for medical data.

Feature attribution methods, like SHAP, were not designed to explain complex
multimodal clinical models [15]. However, feature attributions can be
propagated through a series models according to a modified version of the
chain rule, which has been shown specifically for SHAP [6]. A multimodal model
can be viewed as a sequence of unimodal models, which might make it suitable
to be explained using the proposed chain rule. In this study, we apply this
method to provide local explanations for our multimodal clinical prediction
model.

##  3 Materials and Methods

In this section, we describe our dataset, our unimodal and multimodal models,
our training procedure, and our evaluation.

###  3.1 Dataset

Our private data set contains anonymized data from 1669 patients who underwent
hip fracture surgery at a public hospital between January 2013 and July 2021.
Patients were at least 71 years old at the time of surgery. The data set
contains 1189 (71.2%) females and 480 males (28.8%). Our goal is to predict
30-day mortality after hip fracture surgery, which occurs at a rate of 7.8%
(131/1669) in our dataset. We collected data from five modalities, which we
divided into two groups - pre-operative and per-operative data.

Pre-operative data encompasses information known before surgery, specifically:
static patient data (Static) modality, an axial hip X-ray image (HipImg)
modality and an anterior-posterior chest X-ray image (ChestImg) modality. The
static patient data has 76 features, which we further subdivided into seven
categories: demographics, daily living conditions, nutrition, surgery
information, lab results, medication, and comorbidities, for details see
Appendix 0.B.

Per-operative data was collected during surgery containing vital signs
(Vitals) modality and medication data (Med) modality. The vital signs are
heart rate, pulse, oxygen saturation, and blood pressure. We split blood
pressure into diastolic, systolic, and mean blood pressure resulting in a
total of six time series. The medication data includes 17 medication groups,
which are commonly administered during hip fracture surgery, see Appendix 0.B.
If a patient has missing medication data, then all values were set to zero,
thus assuming the patient did not receive any medication during surgery at
all.

Figure 1: Overview of how the unimodal models are fused to form the multimodal
models. Dimensions at the input and feature extraction layers are shown in
brackets, where the sequence length (seq_len) for the vitals varies between
patients. No feature transformation was performed for the per-operative
medication data. Data types: tabular/structured (S), image (I), time-series
(T).

####  3.1.1 Data preprocessing:

The pre-operative static patient data contain missing values, specifically 10
features had a missing percentage over 10%. These included some the Charlson
comorbidity index (66.6%) and living situation (30%). We imputed the data
iteratively with a K-Nearest Neighbor Regression model111We used the scikit-
learn implementation https://scikit-
learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html
($k=10$). We processed the vitals data, such that elements were spaced 15
seconds apart, where each element represents a single time step containing six
vital signs. We filled up gaps of up to 5 minutes (20 elements) in each vital
sign using linear interpolation. Furthermore, given the close similarity of
heart rate and pulse, we interchangeably replaced missing values if only one
value within the pair is missing. So, if the heart rate was missing at a
certain time step, then we took the pulse at that time step if available, and
vice versa for missing pulse values. We z-normalized the vital signs for each
patient separately because this made fewer assumptions about the data
population [16]. If after interpolation an element (time step) still contains
any missing data, it is skipped during training and inference.

###  3.2 Machine Learning Models

We built the multimodal model (cf. Figure 1), by developing a model for each
modality separately and fusing the representations of the single modalities.
We trained the unimodal models to predict mortality. Then, we transferred the
learned weights from unimodal models to the multimodal models to further train
the latter. Table 2 provides an overview of the models (abbreviation and
description) that we consider in this paper. The remainder of this section
details the unimodal models and the fusion approach to create the multimodal
model.

Table 2: Overview of models and their abbreviations. The per-operative vital signs have variable sequence length (_len_). Data types: tabular/structured (S), image (I), time-series (T). Model |  Pre-orper-operative |  Modality | Architecture |  Inputsize |  Outputsize
---|---|---|---|---|---
Pre-Static${}^{\text{S}}$ |  pre |  static patient data (S) | 1 FC |  76$\times$1 |  16$\times$1
Pre-Hip${}^{\text{I}}$ |  pre |  hip (I) | ResNet50 |  224$\times$224 |  16$\times$1
Pre-Chest${}^{\text{I}}$ |  pre |  chest (I) | ResNet50 |  224$\times$224 |  16$\times$1
Pre${}^{\text{S+I}}$ |  pre |  static (S), hip (I), chest (I) | 2 FC |  48$\times$1 |  1
Per-Vitals${}^{\text{T}}$ |  per |  vital (T) | Bi-LSTM |  6$\times$_len_ |  16$\times$1
Per${}^{\text{S+T}}$ |  per |  vital (T), medication (S) | 2 FC |  33$\times$1 |  1
All${}^{\text{S+I+T}}$ |  both |  static (S), hip (I), chest (I), vital (T), medication (S) | 2 FC |  81$\times$1 |  1

####  3.2.1 Pre-operative Models

We developed three pre-operative unimodal models: the Pre-
Static${}^{\text{S}}$ model for pre-operative static data and the Pre-
Hip${}^{\text{I}}$ and Pre-Chest${}^{\text{I}}$ models for hip and chest
images, respectively. For Pre-Static${}^{\text{S}}$, the main task was
dimensionality reduction, such that the pre-static feature dimension is
reduced to the same number as the other modalities, before passing as input to
the multimodal prediction model. Therefore, we used a single fully connected
hidden layer with 16 neurons with the leaky-ReLu activation function222 In the
early stages of development, we observed that our models suffered from the
“dying ReLu” problem, therefore we chose to employ leaky-ReLu instead of the
regular ReLu activation function for all our fully connected layers [1].. For
Pre-Hip${}^{\text{I}}$ and Pre-Chest${}^{\text{I}}$, we used CNNs, which have
emerged as the de-facto standard for medical image classification [4]. A wide
range of CNN architectures are available, but given our small data set size,
we chose ResNet50 [12], a standard architecture with relatively few parameters
for which a pre-trained model is available. We added two fully connected
layers with 256 and 16 neurons between ResNet50’s global average pooling layer
and the classification layer. We used the same architecture for Pre-
Hip${}^{\text{I}}$ and Pre-Chest${}^{\text{I}}$.

####  3.2.2 Per-operative Models

The Per-Vitals${}^{\text{T}}$ model takes 6 temporal features with variable
sequence length as input, where the sequence lengths vary between 120 and 1337
elements, or 30 minutes to 5 hours, respectively. We used bidirectional long
short-term memory (Bi-LSTM) units [28] to extract information from the vital
signs. Our Per-Vitals${}^{\text{T}}$ model contains a single Bi-LSTM layer
with 2x128 units, followed by two hidden layers with 128 and 16 neurons.
Timesteps still containing missing data after imputation were skipped during
training and testing. Further, we encode the per-operative medication data
with binary encoding.333In preliminary experiments we investigated ordinal and
temporal encoding for the per-operative medication data but did not find a
difference in performance. This data contained only 17 features, so contrary
to the Pre-Static${}^{\text{S}}$ model we did not need to transform the input
features to a lower-dimensional feature space.

####  3.2.3 Multimodal Models

We applied early fusion to generate three multimodal models:
Pre${}^{\text{S+I}}$, Per${}^{\text{S+T}}$, and All${}^{\text{S+I+T}}$ (see
Table 2). For each of these multimodal models, only the relevant modalities
shown in Figure 1 are included. We concatenated the pre-classification layer
outputs from all unimodal models, where each modality supplies 16 features,
except for the per-operative medication data, which yielded 17 features. The
concatenation layer is followed by a fully connected layer with 64 neurons and
a classification layer with a single neuron with sigmoid activation for
mortality prediction.

###  3.3 Training Procedure and Evaluation

We randomly split our data into a training (50%), validation (25%), and test
(25%) set. We used repeated (N=5) cross-validation (k=5) to measure
variability between training runs, furthermore, we used stratification to
ensure a similar number of positive cases in each set. Models were optimized
for maximum validation AUC, which was computed after each training epoch. We
set the maximum number of epochs to 100 and used early stopping with a
patience of 10. We halved the learning rate if there was no improvement for 5
epochs. We used the Adam optimizer and a batch size of 32. We tuned the
learning rate of each model, by experimenting with a predefined range starting
at $10^{-2}$ down to $10^{-5}$ with decrements of $10^{-1}$. The image models
were trained with a relatively small learning rate of $10^{-5}$ because higher
values led to low precision ($<$0.01). The specific learning rates for each
model can be found in Appendix 0.A. To prevent overfitting, we added a dropout
of 0.3 between fully connected layers and set the weight for L2 regularization
to $10^{-3}$.

During training, our models were tasked with minimizing the weighted binary
cross-entropy loss. The weight $w_{c_{i}}$ for class $c_{i}$ was computed
according to Equation 1 where $N_{total}$ is the total number of cases,
$N_{c}=2$ is the number of classes and $N_{c_{i}}$ is the number of cases with
class $c_{i}$ [18].

$\frac{N_{total}}{N_{c}\cdot N_{c_{i}}}$ (1)

During the training of the image models, we augmented training images with
random shift ($\pm$0.2), shear ($\pm$0.2), zoom ($\pm$0.2), and rotation
($\pm$20∘) to mimic a more diverse training set. We used bicubic interpolation
to scale images to the 224x224 for ResNet50.

The multimodal models were initialized with the weights of the unimodal
models, which we kept frozen while we trained the weights of the post-
concatenation layers. Afterwards, we finetuned the model by unfreezing all
layers, except for the layers in the image models. The learning rates were
$5\cdot 10^{-2}$ and $5\cdot 10^{-3}$, for the first and second steps,
respectively.

We computed recall, precision, F1-score, and AUC for the mortality prediction
task. Each model architecture was trained 5 times with different randomly
initialized weights, to measure variability between training runs. We report
the mean and standard deviation of these runs.

The data are not publicly available due to privacy and ethical considerations,
but all relevant code is available on
GitHub444https://github.com/jornjan/mmshap-XAI2024.

###  3.4 Explanation

We applied post-hoc model explanations to one version of the final model
(All${}^{\text{S+I+T}}$) to better understand its predictions, thereby
increasing clinical relevancy. To estimate the importance of features we used
Shapley values555SHAP library https://github.com/slundberg/shap, which
resemble the contribution of each extracted feature towards a predicted
outcome [20]. The contribution can be negative, indicating the value of the
feature lowered the predicted value, or the contribution can be positive
indicating the feature value increased the predicted value.

Given the multimodal nature of our model, we split the explanation into two
steps. In the first step, we compute the contribution of each modality towards
the predicted outcome. In the second step, we compute the contribution of
individual features from a given modality, specifically, we report the
importance of individual pre-operative static features in the multimodal
prediction, because we found that these contributed during our experiments. In
the first step, at the modality level, we provide local (single test case) and
global (all test cases) explanations, while we limit ourselves to local
explanations in the second step.

We define $H$ as the concatenation of the extracted hidden features from each
modality ($H_{static}$, $H_{hip}$, $H_{chest}$ and $H_{vitals}$) and the per-
operative medicine input values, $X_{med}$. We denote $f_{mm}$ as the function
that maps the multimodal features in the concatenation layer to a 30-day
mortality prediction ($\hat{y}$). The estimated Shapley values
($\hat{\phi}_{h_{i}}$) for each feature $h_{i}\in H$ have the property that
they sum up to the predicted outcome [20]. $\hat{\phi}_{0}$ equals the average
prediction based on the reference dataset, for which we took the training set.

$H=H_{static}\oplus H_{hip}\oplus H_{chest}\oplus H_{vitals}\oplus X_{med}.$
$\hat{y}=f_{mm}(H)=\hat{\phi}_{0}+\sum_{i=1}^{|H|}\hat{\phi}_{h_{i}}$ (2)

Global explanations follow from taking the average absolute contribution of
each feature across all cases. We report global explanation for each modality,
$m$ by calculating the absolute ($AC_{m}$) and relative ($RC_{m}$)
contributions as follows:

$\displaystyle AC_{m}$
$\displaystyle=\sum_{j=1}^{|H_{m}|}\hat{\phi}_{h_{j}},\mspace{20.0mu}h_{j}\in
H_{m}$ (3) $\displaystyle RC_{m}$
$\displaystyle=\frac{AC_{m}}{\sum_{i=1}^{|H|}\hat{\phi}_{h_{i}}},\mspace{25.0mu}h_{i}\in
H$ (4)

We perform the second step, where we compute the contribution of individual
static features for the multimodal prediction. Shapley values can be
propagated through a sequence of models using a modified version of the chain
rule [6]. Equation 5 shows how we computed the contribution of individual
static features for a single case. We define $f_{static}$ as the function that
extracts features from the static input features ($x_{i}$) and $AC_{static}$
as the absolute contribution of the extracted static features.

$\hat{\phi}_{x_{i}}=\hat{\phi}(f_{static},x_{i})\mspace{5.0mu}(\hat{\phi}(f_{mm},H_{static})\mspace{5.0mu}\oslash\mspace{5.0mu}AC_{static}),\mspace{15.0mu}x_{i}\in
X_{static}$ (5)

##  4 Results

###  4.1 Model Performance

Table 3 presents the average mortality prediction performance on the test set
for all unimodal and multimodal models. The Pre${}^{\text{S+I}}$, Pre-
Static${}^{\text{S}}$, and All${}^{\text{S+I+T}}$ models all scored the
highest average test set AUC of 0.78, which suggests adding modalities to the
pre-operative static patient data does not improve performance. Specifically,
per-operative features seem to have low predictive power given the poor test
set performances of the Per-Vitals${}^{\text{T}}$ and Per${}^{\text{S+T}}$
models. All models tend to have a higher recall than precision, meaning the
difficulty lies within correctly identifying non-risk patients. _To summarize,
pre-operative data has more predictive power than per-operative data.
Specifically, the pre-operative static modality is the most important among
all pre-operative modalities. This suggests that risk of mortality can be
estimated pre-operatively by looking at static patient data and treatment
options can be discussed prospectively._

Table 3: Average performance of all models along with standard deviation over 25 runs. The best performance in each column is indicated in bold. | AUC | Recall | Precision | F1-score
---|---|---|---|---
Unimodal |  |  |  |
| Pre-Static${}^{\text{S}}$ | $\textbf{0.78}\pm 0.04$ | $\textbf{0.82}\pm 0.09$ | $0.17\pm 0.03$ | $0.28\pm 0.04$
| Pre-Hip${}^{\text{I}}$ | $0.50\pm 0.07$ | $0.55\pm 0.32$ | $0.09\pm 0.04$ | $0.14\pm 0.05$
| Pre-Chest${}^{\text{I}}$ | $0.50\pm 0.07$ | $0.55\pm 0.29$ | $0.10\pm 0.03$ | $0.15\pm 0.03$
| Per-Vitals${}^{\text{T}}$ | $0.49\pm 0.05$ | $0.49\pm 0.30$ | $0.09\pm 0.01$ | $0.15\pm 0.02$
Multimodal |  |  |  |
| Pre${}^{\text{S+I}}$ | $\textbf{0.78}\pm 0.04$ | $0.80\pm 0.11$ | $\textbf{0.18}\pm 0.04$ | $\textbf{0.29}\pm 0.05$
| Per${}^{\text{S+T}}$ | $0.57\pm 0.06$ | $0.62\pm 0.23$ | $0.11\pm 0.02$ | $0.17\pm 0.02$
| All${}^{\text{S+I+T}}$ | $\textbf{0.78}\pm 0.04$ | $0.79\pm 0.10$ | $0.17\pm 0.02$ | $0.28\pm 0.03$

###  4.2 Explainability

Global explanation: Table 4 shows the average relative contribution of each
modality as well as the pre-operative and per-operative groups. The features
extracted from the static patient data contribute the most with a relative
contribution of 70.3%, while other modalities contribute much less. The second
highest contributor is the per-operative vital signs with 10.4% but with a
high standard deviation of 10.5%. Also, the pre-operative image modalities
have a standard deviation close to their mean value. On average the per-
operative medication data contributes the least towards the prediction.

Local explanation: Figure 2 shows a local explanation for two cases. Each case
is explained with two plots, where the first shows how much each modality
contributed towards the prediction, and the second shows feature-specific
contribution for the _static_ modality. Note that the contribution values of
the _Static_ features sum up to the _Static_ contribution on the modality
level. The local multimodal explanations start at $\mathbb{E}[f(x)]=0.362$,
which equals the average prediction for the training set. The plots show how
each modality influenced the patient prediction compared to the training set
average.

The positive case prediction is strongly influenced by static features with a
contribution of 46% towards the prediction of 80.7%, while the other
modalities contribute 3% or less. The help with bed transfer is the most
important static feature in this particular case with a contribution of 8%.
For the negative case prediction (plots at the bottom), the static features
are most important in reducing the mortality prediction by 31%. The fact that
this case concerned a femoral neck fracture reduced the prediction by 9%.

Table 4: Mean relative contribution for mortality prediction of input modalities taken over 25 runs and in brackets the standard deviation. RC: relative contribution. Modality | RC % (st. dev.)
---|---
Pre-operative | 84.6% (11.1)
| Static patient data |  | 70.3% (10.6)
| Hip image |  | 7.0% (6.0)
| Chest image |  | 7.3% (8.3)
Per-operative | 15.4% (11.1)
| Vitals |  | 10.4% (10.5)
| Medication |  | 5.0% (2.0)
Figure 2: Shapley plots for single cases in the test set. First, the
contribution of each modality is shown, followed by the contribution of
individual static features.

##  5 Discussion

_Comparison to state-of-the-art:_ The addition of per-operative data did not
yield a significant performance improvement compared to our pre-operative
multimodal model. Our unimodal image models (Pre-Hip${}^{\text{I}}$ and Pre-
Chest${}^{\text{I}}$) score worse compared to the image models developed by
Yenidogan et al. [34], who reported an AUC of 0.67 and 0.70 on the hip and
chest images, respectively. The lower performance could be due to our smaller
data set and smaller CNN architecture. On the other hand, our Pre-
Static${}^{\text{S}}$ model performs on par with state-of-the-art models with
a smaller dataset [34, 22, 5].

_Usage of explanations:_ Using Shapley values, we explained the prediction of
the All${}^{\text{S+I+T}}$ model at the modality level. Globally the pre-
operative static features are most important with a relative importance of
70.3%, still the relative importance of per-operative features in our
All${}^{\text{S+I+T}}$ model was 15.4%. Further, we provide an example of a
local explanation in Figure 2. We explained our multimodal model in two steps.
First, the contribution of the extracted features from each modality was
computed with which we calculated the contribution of individual pre-operative
static features in the second step. We only show the explanation for the pre-
operative static data as we found this to be the most important modality.
Explanations for other unimodal models can be added as well, e.g. using Grad-
CAM [29] for images. Static features describing kidney function (Urea level,
diuretic drug, and sodium level) are important for both the single-patient
local explanations for predicting 30-day mortality. Furthermore, the level of
self-reliance is important given the importance of: help with bed-chair
transfer, help with dressing, and living situation.

_Limitations and future work:_ Our results show that predicting mortality for
hip fracture patients is difficult and starting with _predicting_ if any
_complication_ occurs within 30 days could improve the assessment of the data
set predictive capabilities. Coincidentally, this resolves the class imbalance
issue, because in our data set 49% of the patients experience at least one
complication within 30 days after surgery. Furthermore, as an intermediate
step complications could be grouped by severity or cases could be scored on a
scale from no complication to mortality. It would be interesting to look into
this in future work. Clinically, this could help determine, whether a patient
is at risk after surgery and requires more attention.

Our _fusion method_ could be described as mid fusion, because we used the pre-
classification layer of each unimodal model, however we did do dimension
reduction before concatenation. This method ensured each modality contributed
the same number of features to the classification layer, however, this might
not be optimal for post-operative complication prediction. Future work could
include a deeper investigation of fusion methods, like late fusion and
bottleneck fusion. Bottleneck fusion restricts cross-modal information flow by
using a very limited amount of neurons for information to pass through. The
idea is that the model is forced to condense the most important cross-modal
features leading to better performance at negligible computational cost [21].

During our experiments, we found that the models that include static patient
data or vital signs are overfitting the data suggesting that the model might
be memorizing the training data, with poor _generalization capability_ to the
test data. This might indicate that: i) there is some variation within the
instances and does not contain many common patterns, ii) there is not enough
data for generalization. The addition of dropout layers and L2-regularization
did not solve this problem, therefore a different approach is required. It has
been shown that reducing the number of pre-operative static features based on
their importance can prevent overfitting [5]. We could use the Shapley values
to iteratively select the most important feature, up until we reach a certain
subset size. Additionally, if we prevent strong overfitting on the pre-
operative static data, this could incentivize the multimodal models to focus
more on the other modalities for information.

Our multimodal model is not robust against _missing data_ , only the per-
operative medication data and part of the pre-operative static data are
allowed to be missing. In clinical practice, this would mean patients are
excluded if pre-operative images or per-operative vitals are missing. We
impute the pre-operative static patient data, so having some missing values
there does not lead to exclusion, however, the imputation quality is dependent
on the number of non-missing features. Therefore, for clinical applicability
future models should be robust against _missing modalities_ , to avoid patient
exclusion.

##  6 Conclusion

We conclude that the addition of per-operative data to pre-operative data does
not significantly improve 30-day mortality prediction. Further, investigation
confirmed that pre-operative features are most important for mortality
prediction, while per-operative features contribute little except for a few
per-operative medications. We show that multimodal black box models can be
made explainable by applying the chain rule to the feature attributions of
each model in the sequence. Future work, could restrict to only using pre-
operative data and further explain model predictions by providing
interpretable explanations for the contribution of the image modalities.

####  6.0.1 Disclosure of Interests.

Christin Seifert is a member of the XAI 2024 World Conference committee. All
other authors have no competing interests to declare that are relevant to the
content of this article.

## References

  * [1] Agarap, A.F.: Deep Learning using Rectified Linear Units (ReLU) (2019). https://doi.org/10.48550/arXiv.1803.08375
  * [2] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLOS ONE 10(7), e0130140 (2015). https://doi.org/10.1371/journal.pone.0130140
  * [3] Briganti, G., Le Moine, O.: Artificial Intelligence in Medicine: Today and Tomorrow. Frontiers in Medicine 7 (2020)
  * [4] Cai, L., Gao, J., Zhao, D.: A review of the application of deep learning in medical image classification and segmentation. Annals of Translational Medicine 8(11), 713–713 (2020). https://doi.org/10.21037/atm.2020.02.44
  * [5] Cao, Y., Forssten, M.P., Mohammad Ismail, A., Borg, T., Ioannidis, I., Montgomery, S., Mohseni, S.: Predictive Values of Preoperative Characteristics for 30-Day Mortality in Traumatic Hip Fracture Patients. J Pers Med 11(5), 353 (2021). https://doi.org/10.3390/jpm11050353
  * [6] Chen, H., Lundberg, S.M., Lee, S.I.: Explaining a series of models by propagating shapley values. Nature Communications 13(1), 4512 (2022). https://doi.org/10.1038/s41467-022-31384-3, https://www.nature.com/articles/s41467-022-31384-3, number: 1 Publisher: Nature Publishing Group
  * [7] Das, A., Rad, P.: Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey (2020). https://doi.org/10.48550/arXiv.2006.11371
  * [8] de Munter, L., Polinder, S., Lansink, K.W.W., Cnossen, M.C., Steyerberg, E.W., de Jongh, M.A.C.: Mortality prediction models in the general trauma population: A systematic review. Injury 48(2), 221–229 (2017). https://doi.org/10.1016/j.injury.2016.12.009
  * [9] Fritz, B.A., Cui, Z., Zhang, M., He, Y., Chen, Y., Kronzer, A., Ben Abdallah, A., King, C.R., Avidan, M.S.: Deep-learning model for predicting 30-day postoperative mortality. British Journal of Anaesthesia 123(5), 688–695 (2019). https://doi.org/10.1016/j.bja.2019.07.025
  * [10] Gowd, A.K., Agarwalla, A., Amin, N.H., Romeo, A.A., Nicholson, G.P., Verma, N.N., Liu, J.N.: Construct validation of machine learning in the prediction of short-term postoperative complications following total shoulder arthroplasty. Journal of Shoulder and Elbow Surgery 28(12), e410–e421 (2019). https://doi.org/10.1016/j.jse.2019.05.017
  * [11] Gullberg, B., Johnell, O., Kanis, J.: World-wide Projections for Hip Fracture. Osteoporos Int 7(5), 407–413 (1997). https://doi.org/10.1007/PL00004148
  * [12] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
  * [13] Hu, F., Jiang, C., Shen, J., Tang, P., Wang, Y.: Preoperative predictors for mortality following hip fracture surgery: A systematic review and meta-analysis. Injury 43(6), 676–685 (2012). https://doi.org/10.1016/j.injury.2011.05.017
  * [14] Jones, H.J., de Cossart, L.: Risk scoring in surgical patients. Br J Surg 86(2), 149–157 (1999). https://doi.org/10.1046/j.1365-2168.1999.01006.x
  * [15] Junaid, M., Ali, S., Eid, F., El-Sappagh, S., Abuhmed, T.: Explainable machine learning models based on multimodal time-series data for the early detection of Parkinson’s disease. Computer Methods and Programs in Biomedicine 234 (2023). https://doi.org/10.1016/j.cmpb.2023.107495
  * [16] Karim, F., Majumdar, S., Darabi, H.: Insights Into LSTM Fully Convolutional Networks for Time Series Classification. IEEE Access 7, 67718–67725 (2019). https://doi.org/10.1109/ACCESS.2019.2916828
  * [17] Karres, J., Kieviet, N., Eerenberg, J.P., Vrouenraets, B.C.: Predicting Early Mortality After Hip Fracture Surgery: The Hip Fracture Estimator of Mortality Amsterdam. Journal of Orthopaedic Trauma 32(1), 27–33 (2018). https://doi.org/10.1097/BOT.0000000000001025
  * [18] King, G., Zeng, L.: Logistic Regression in Rare Events Data. Political Analysis 9(2), 137–163 (2001/ed). https://doi.org/10.1093/oxfordjournals.pan.a004868
  * [19] Lee, C.K., Hofer, I., Gabel, E., Baldi, P., Cannesson, M.: Development and Validation of a Deep Neural Network Model for Prediction of Postoperative In-hospital Mortality. Anesthesiology 129(4), 649–662 (2018). https://doi.org/10.1097/ALN.0000000000002186
  * [20] Lundberg, S.M., Lee, S.I.: A Unified Approach to Interpreting Model Predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017)
  * [21] Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., Sun, C.: Attention Bottlenecks for Multimodal Fusion. In: Advances in Neural Information Processing Systems. vol. 34, pp. 14200–14213. Curran Associates, Inc. (2021)
  * [22] Nijmeijer, W.S., Folbert, E.C., Vermeer, M., Slaets, J.P., Hegeman, J.H.: Prediction of early mortality following hip fracture surgery in frail elderly: The Almelo Hip Fracture Score (AHFS). Injury 47(10), 2138–2143 (2016). https://doi.org/10.1016/j.injury.2016.07.022
  * [23] Pawar, U., O’Shea, D., Rea, S., O’Reilly, R.: Explainable AI in Healthcare. In: 2020 International Conference on Cyber Situational Awareness, Data Analytics and Assessment (CyberSA). pp. 1–2 (2020). https://doi.org/10.1109/CyberSA49311.2020.9139655
  * [24] Perng, J.W., Kao, I.H., Kung, C.T., Hung, S.C., Lai, Y.H., Su, C.M.: Mortality Prediction of Septic Patients in the Emergency Department Based on Machine Learning. Journal of Clinical Medicine 8(11), 1906 (2019). https://doi.org/10.3390/jcm8111906
  * [25] Pham, T., Tran, T., Phung, D., Venkatesh, S.: Predicting healthcare trajectories from medical records: A deep learning approach. Journal of Biomedical Informatics 69, 218–229 (2017). https://doi.org/10.1016/j.jbi.2017.04.001
  * [26] Ribeiro, M.T., Singh, S., Guestrin, C.: "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 1135–1144. KDD ’16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778
  * [27] Schoenfeld, A.J., Le, H.V., Marjoua, Y., Leonard, D.A., Belmont, P.J., Bono, C.M., Harris, M.B.: Assessing the utility of a clinical prediction score regarding 30-day morbidity and mortality following metastatic spinal surgery: The New England Spinal Metastasis Score (NESMS). The Spine Journal 16(4), 482–490 (2016). https://doi.org/10.1016/j.spinee.2015.09.043
  * [28] Schuster, M., Paliwal, K.: Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on 45, 2673–2681 (1997). https://doi.org/10.1109/78.650093
  * [29] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [30] Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: International Conference on Machine Learning. pp. 3145–3153. PMLR (2017)
  * [31] Tonekaboni, S., Joshi, S., McCradden, M.D., Goldenberg, A.: What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. In: Proceedings of the 4th Machine Learning for Healthcare Conference. pp. 359–380. PMLR (2019)
  * [32] Wang, W., Krishnan, E.: Big Data and Clinicians: A Review on the State of the Science. JMIR Medical Informatics 2 (Jan 2014). https://doi.org/10.2196/medinform.2913, https://medinform.jmir.org/2014/1/e1
  * [33] Xue, B., Li, D., Lu, C., King, C.R., Wildes, T., Avidan, M.S., Kannampallil, T., Abraham, J.: Use of Machine Learning to Develop and Evaluate Models Using Preoperative and Intraoperative Data to Identify Risks of Postoperative Complications. JAMA Network Open 4(3), e212240–e212240 (2021). https://doi.org/10.1001/jamanetworkopen.2021.2240
  * [34] Yenidogan, B., Pathak, S., Geerdink, J., Hegeman, J.H., van Keulen, M.: Multimodal Machine Learning for 30-Days Post-Operative Mortality Prediction of Elderly Hip Fracture Patients. In: 2021 International Conference on Data Mining Workshops (ICDMW). pp. 508–516 (2021). https://doi.org/10.1109/ICDMW53433.2021.00068

##  Appendix 0.A Hyperparameters

Table 5: Hyperparameters for each model: Learning Rate (LR) Model | LR
---|---
Pre-Static${}^{\text{S}}$ | $10^{-3}$
Pre-Hip${}^{\text{I}}$ | $10^{-5}$
Pre-Chest${}^{\text{I}}$ | $10^{-5}$
Per-Vitals${}^{\text{T}}$ | $5\cdot 10^{-4}$
Pre${}^{\text{S+I}}$ | $5\cdot 10^{-2}$
Per${}^{\text{S+T}}$ | $5\cdot 10^{-2}$
All${}^{\text{S+I+T}}$ | $5\cdot 10^{-2}$

##  Appendix 0.B Detailed feature overview

Table 6: Available pre-operative features grouped by category

Demographics |  | Daily living
---
activities
Nutrition |  | Surgery
---
information
Age | Help with transfer from bed to chair | Malnutrition risk | Fracture type
Surgery start/end | Help with showering | Unintended weight loss | Surgery type
Falling risk | Help with dressing | Drink or tube feeding | Fracture laterality
Fall last year | Help with going to toilet | Decreased appetite | ASA score
Pre-fracture mobility | Help with eating | SNAQ score |
Living situation | Help with self-care |  |
Prone to delirium | Katz ADL score |  |
Memory problems | Incontinence material used |  |
Delirium in the past |  |  |
CCI score |  |  |

Lab results | Medication (reason/effect) | Comorbidities
---|---|---
HB | Blood thinners | Chronic pulmonary disease
HT | Vitamin D | Congestive heart failure
CRP | Polypharmacy | Peripheral vascular disease
LEUC | A02 (acid related disorders) | Cerebrovascular disease
THR | A10 (diabetes) | Dementia
BLGR | B01 (antithrombotic) | Renal disease
IRAI | B02 (antihemmorrhagics) | Rheumatological disease
ALKF | B03 (antianemic) | Cancer
GGT | C01 (cardiac therapy) | Cerebrovascular event
ASAT | C03 (diuretics) | Liver disease
ALAT | C07 (beta blockers) | Lymphoma
LDH1 | C08 (calcium channel blockers) | Leukemia
UREU | C09 (renin-angiotensin system) | Peptic ulcer disease
KREA | C10 (lipid modification) | Diabetes
GFRM | L04 (immunosuppressants) | Prior myocardial infarction
NA | M01 (anti-inflammatory) |
XKA | N05 (psycholeptics) |
GLUCGLUC | R03 (airway obstruction) |
Table 7: List of medications that were at least administered in 100 unique cases, also includes the general reason for usage. Percentages are with respect to a total of 1616 cases for which medication data is available. Medication | # unique cases | Effect
---|---|---
Bupivacaine | 649 (40.2%) | Anesthetic
Cefazolin | 692 (42.8%) | Antibiotic
Dexemethasone | 198 (12.3%) | Anti-inflammatory
Efedrine | 445 (27.5%) | Increase blood pressure
Elektrolytes | 468 (29.0%) | Minerals
Esketamine | 561 (34.7%) | Anesthetic
Lidocaine | 405 (25.1%) | Anesthetic
Metamizole | 187 (11.6%) | Painkiller
Midazolam | 475 (29.4%) | Anesthetic
Noradrenaline | 887 (54.9%) | Increase blood pressure
Ondansetron | 282 (17.5%) | Counter post-operative nausea
Piritramide | 446 (27.6%) | Painkiller
Propofol | 648 (40.1%) | Anesthetic
Rocuronium | 260 (16.1%) | Muscle relaxant
Sufentanil | 746 (46.2%) | Painkiller
Sugammadex | 112 (6.9%) | Reverse muscle relaxant
Tranexamic acid | 465 (28.8%) | Prevent blood loss