# Attri-VAE: attribute-based interpretable representations of medical images
with variational autoencoders

Irem Cetin  Maialen Stephens  Oscar Camara  Miguel A. González Ballester
<EMAIL_ADDRESS>BCN Medtech, Department of Information and Communication
Technologies, Universitat Pompeu Fabra, Barcelona, Spain  Vicomtech
Foundation, San Sebastián, Spain  ICREA, Barcelona, Spain

###### Abstract

Deep learning (DL) methods where interpretability is intrinsically considered
as part of the model are required to better understand the relationship of
clinical and imaging-based attributes with DL outcomes, thus facilitating
their use in the reasoning behind the medical decisions. Latent space
representations built with variational autoencoders (VAE) do not ensure
individual control of data attributes. Attribute-based methods enforcing
attribute disentanglement have been proposed in the literature for classical
computer vision tasks in benchmark data. In this paper, we propose a VAE
approach, the Attri-VAE, that includes an attribute regularization term to
associate clinical and medical imaging attributes with different regularized
dimensions in the generated latent space, enabling a better-disentangled
interpretation of the attributes. Furthermore, the generated attention maps
explained the attribute encoding in the regularized latent space dimensions.
Using the Attri-VAE approach we analyzed healthy and myocardial infarction
patients with clinical, cardiac morphology, and radiomics attributes. The
proposed model provided an excellent trade-off between reconstruction
fidelity, disentanglement, and interpretability, outperforming state-of-the-
art VAE approaches according to several quantitative metrics. The resulting
latent space allowed the generation of realistic synthetic data in the
trajectory between two distinct input samples or along a specific attribute
dimension to better interpret changes between different cardiac conditions.

###### keywords:

deep learning , interpretability, attribute regularization , variational
autoencoder , cardiac image analysis

##  1 Introduction

Deep learning (DL) methods have recently shown great success in many fields,
from computer vision [52, 74, 23] to natural language processing [70, 19],
among numerous others. In addition, DL methods have started to dominate the
medical imaging field [66], being used in a variety of medical imaging
problems, such as segmentation of anatomical structures in the images [7, 57,
43], disease prediction [30], medical image reconstruction [26, 34] and
clinical decision support [62]. Despite achieving exceptional results, DL
methods face challenges when applied to medical data regarding explainability,
interpretability, and reliability because of their underlying black-box nature
[69, 47]. Hence, the need for tools that investigate interpretability in DL is
also emerging in healthcare.

Recent reviews of interpretable DL can be found in [69, 5, 48, 46]. Some
methods have been proposed that employ backpropagation-based attention maps to
either generate class activation maps that visualize the regions with high
activations in specific units of the network [63] or saliency maps using
gradients of the inputs with respect to the outputs [68, 31]. Other methods
also proposed creating proxy models that focus on complexity reduction such as
LIME [55] or by approximating a value based on game theory optimal Shapley
values to explain the individual predictions of a model [45]. However, it is
key to design models that are inherently interpretable, rather than creating
posthoc models to explain the black-box ones [59].

Recently, models based on latent representations, such as variational
autoencoders (VAE), have become powerful tools in this direction [41, 8], as
their latent space is able to encode important hidden variables of the input
data [33]. Especially, when dealing with data that contains different
interpretable features (data attributes), it is interesting to see how and if
these attributes have been encoded in the latent space. Even though the
proposed approaches provide promising results, they have some limitations, one
of which is that the encoded variables cannot be easily controlled; they
mostly show an entangled behavior, meaning each latent factor maps to more
than one aspect in the generative process [6].

In order to bypass this limitation, much effort has been done to enforce
disentanglement in the latent space [27, 32, 58, 16, 15], being the majority
of them unsupervised techniques [6, 42]. While many of these methods show good
disentanglement performance, they are not only sensitive to inductive biases
(e.g., choice of the network, hyperparameters, or random seeds), but also some
amount of supervision is necessary for learning effective disentanglement
[42]. Moreover, since these methods are able to learn a factorized latent
representation without attribute specification, they require a posthoc
analysis to determine how different attributes are encoded to different
dimensions of the latent space [51].

On the other hand, attribute-based methods aim to establish a correspondence
between data attributes of interest and the latent space [25, 39, 9, 51].
However, these methods also have their drawbacks: some of them are limited to
work only on certain types of data attributes [39]; some impose additional
constraints [9]; very few of them are designed to work with continuous
variables [25, 51]; some require differentiable computation of the attributes;
and they are extremely sensitive to the hyperparameters [25]. However, [51]
have recently shown promising results for interpretability with their
approach, associating each data attribute to a different regularized dimension
of the latent space, which they have applied in the MNIST database for digit
number recognition. The same approach was also employed as a post-processing
step to generate interpretable and temporally consistent segmentations of
echocardiography images [50].

In this paper, we proposed a VAE-based approach (Attri-VAE), as it is able to
encode certain hidden attributes of the data [12] which can then be used to
control data generation, and thus, improve the interpretation of the data
attributes [51]. Attri-VAE is an attribute-interpreter VAE based on attribute-
based regularization [51] in the latent space, for an enhanced interpretation
of clinical and imaging attributes obtained from multi-modal sources.
Additionally, we also employed a classification network (MLP) that enables to
identify different clinical conditions, e.g., healthy vs. pathological cases.
Furthermore, we incorporate gradient-based attention map computation [41] to
visually explain our proposed network by generating attention maps that show
the high-response regions for each value of data attributes that are encoded
in the regularized latent space dimensions. The main contributions of this
work can be described as follows:

  * 1.

The proposed approach is able to interpret different data attributes where
specific ones are forced to be encoded along specific latent dimensions
without the need for any posthoc analysis, while encouraging attribute
disentanglement by employing $\beta$-VAE as a backbone [27].

  * 2.

The structured latent space enables controllable data generation by changing
the latent code of the regularized dimension (i.e., following the
corresponding attribute), generating new data samples as a result of
manipulating these dimensions. For instance, if the attribute represents the
volume in a region of interest (ROI) and the corresponding regularized
dimension is the first one of the latent code, then increasing values of this
dimension would result in an increase in the ROI volume. The ability to
generate a heterogeneous set of medical images is promising as collecting
large annotated images is especially an issue in the medical image domain. As
our approach allows the generation of diverse datasets, it may be applied in a
variety of clinical scenarios. It could, for example, be used to train robust
deep-learning models, or it may be trained with various clinical conditions to
generate synthetic images by modifying the latent code of regularized
dimensions that follows the corresponding data attribute. Furthermore, a
controlled virtual cohort generation could also be employed to remove the
burden of needing to gather and select real data for clinical trials.

  * 3.

Attribute-based gradient-based attention maps provide a way to explain how the
gradient information of individual attributes flow inside the proposed
architecture by showing high-response regions in the images.

  * 4.

The classification network provides a way to stratify different cohorts, based
on the attributes in the latent space. In this way, the most discriminative
features for the classification task are identified by projecting original
samples into the latent space.

Figure 1: Training framework of the proposed approach. Loss functions are
shown in red arrows. The total loss function of the model is:
$\mathcal{L}=\mathcal{L}_{recon}+\beta\mathcal{L}_{KL}+\mathcal{L}_{MLP}+\gamma\mathcal{L}_{AR}$.
(a) Losses computed for each data sample: multilayer perceptron (MLP) loss
($\mathcal{L}_{MLP}$), Kullback-Leibler (KL) loss ($\mathcal{L}_{KL}$), and
reconstruction loss ($\mathcal{L}_{recon}$). (b) Attribute-regularization loss
($\mathcal{L}_{AR}$), computed inside a training batch that has $n$ data
samples. The input, a 3D image ($X$), first goes through the 3D convolutional
encoder, $q_{\varphi}(Z|X)$, which learns to map $X$ to the low dimensional
space $Z$ by outputting the mean ($\mu$) and variance ($\sigma$) of the latent
space distributions. The decoder, $p_{\theta}(\hat{X}|Z)$, then takes $Z$ and
outputs the reconstruction of the original input, ($\hat{X}$). The predicted
classes of the inputs, $y_{c}$, are computed with a MLP module that consists
of three fully connected (FC) layers. The corresponding MLP loss function is
computed between $y_{c}$ and the ground truth label $y_{GT}$. In (b),
$\mathcal{L}_{AR}$ is shown to regularize the first dimension of the latent
space ($Z^{1}$) with the attribute $a_{1}$ ($a_{1}$ and $a_{2}$ represent the
first and the second attributes, respectively). $Dist_{Z^{1}}$ is the distance
matrix of the first latent dimension, while $Dist_{{a}_{1}}$ represents the
distance matrix of the attribute $a_{1}$.

In this work, we have applied the proposed Attri-VAE approach to study
cardiovascular pathological conditions, such as myocardial infarction, using
the EMIDEC111http://emidec.com/ cardiac imaging dataset [36], including
clinical and imaging features, also exploring the association with radiomics
descriptors. Additionally, we used ACDC MICCAI17
database222https://www.creatis.insa-lyon.fr/Challenge/acdc/ as an external
testing dataset.

The remainder of this paper is organized as follows. Firstly, we present the
methodology and the details of our architecture in Section 2. We then describe
the experimental setup and employed dataset in Section 3. Section 4 provides
the results that are discussed in Section 5. Finally, in Section 6 we conclude
our findings.

##  2 Methodology

The overall structure of our framework is shown in Figure 1 (training) and
Figure 2 (testing). The proposed Attri-VAE incorporates attribute
regularization into a $\beta$-VAE framework that was used as a backbone for
the interpretation of data attributes. The trained network enables to generate
new data samples by manipulating the data attributes, whereas the generated
attribute-based attention maps explain how the gradient information of each
attribute flows inside the proposed architecture. This section is organized
firstly explaining the overall training criterion of the proposed model, with
the following subsections describing each of the elements of our methodology
and their integration.

Figure 2: The trained network can be used for: (a) latent space manipulation;
and (b) generating attribute-based attention maps. For a given 3D data sample,
$X$, the trained 3D convolutional encoder, $q_{\varphi}(Z|X)$, outputs the
mean ($\mu$) and variance ($\sigma$) vectors, then $Z$ being sampled with the
reparameterization trick. (a) Data generation process by changing only first
($Z^{1}$) and second ($Z^{2}$) regularized latent dimensions of $Z$, which
correspond to two different data attributes (volume and maximum 2D diameter,
respectively). Then, the decoder, $p_{\theta}(X|Z)$, generates 3D outputs,
$X_{1}$ and $X_{2}$, using the manipulated latent vectors, $Z_{1}$ and
$Z_{2}$, respectively. (b) Attribute-based attention map generation for a
given attribute, which is encoded in the first latent dimension $(Z^{1})$.
First, $(Z^{1})$ is backpropagated to the encoder´s last convolutional layer
to obtain the gradient maps ($Grads_{1}$ and $Grads_{2}$) with respect to the
feature maps ($F_{1}$ and $F_{2}$). The gradient maps of ($Z^{1}$) measure the
linear effect of each pixel in the corresponding feature map on the latent
values. After that, we compute the weights ($w_{1}$ and $w_{2}$) using global
average pooling (GAP) on each gradient map. A heat map is generated by
multiplying these values ($w_{1},w_{2}$) with the corresponding feature map,
summing them up and applying an activation unit (ReLU). Finally, the heat map
is upsampled and overlaid with the input image to obtain the superimposed
image (3D attention map). Additionally, the class score of the input, $y_{c}$,
is computed with the multilayer perceptron (MLP) that is connected to $Z$.
Note that, in the figure it is assumed that the last convolutional layer of
the encoder has 2 feature maps.

###  2.1 Training criterion

Attri-VAE is trained with a loss function, $\mathcal{L}$, which is composed of
four terms, as follows:

$\mathcal{L}=\mathcal{L}_{recon}+\beta\mathcal{L}_{KL}+\mathcal{L}_{MLP}+\gamma\mathcal{L}_{AR}.$
(1)

The reconstruction loss, $\mathcal{L}_{recon}$, is based on the binary cross-
entropy (BCE) between the input $X$ and its reconstruction $\hat{X}$, while
the second term, $\mathcal{L}_{KL}$, employs the Kullback-Leibler (KL)
divergence between the learned prior and the posterior distributions, weighted
by a hyperparameter ($\beta$). An additional term, $\mathcal{L}_{MLP}$,
estimates the BCE loss for the classification between the network prediction,
$y_{c}$, and the ground truth label, $y_{GT}$. The final loss term,
$\mathcal{L}_{AR}$, includes the attribute regularization, with a tunable
hyperparameter ($\gamma$) that weights its strength. In the following
sections, detailed explanations of each loss term in our training criterion
can be found (also see Figure 1).

###  2.2  Variational autoencoder (VAE) and $\beta$-VAE

A variational autoencoder [33] is a generative model that consists of an
encoder and a decoder, and aims to maximize the marginal likelihood of the
reconstructed output, which is written as:

$log_{p_{\theta}}(X)\geq\mathbb{E}_{Z\sim
q_{\varphi}(Z|X)}[logp_{\theta}(X|Z)]-D_{KL}(q_{\varphi}(Z|X)\|p(Z))$ (2)

In this objective function, the first term is the log likelihood expectation
that the input $X$ can be generated by the sampled $Z$ from the inferred
distribution, $q_{\varphi}(Z|X)$. The second term corresponds to the KL
divergence between the distribution of $Z$ inferred from $X$, and the prior
distribution of $Z$. Note that both distributions are assumed to follow a
multivariate normal distribution.

In practice, the loss function of the VAE consists of two terms: a first term
that penalizes the reconstruction error between the input and output; and a
second term forcing the learned distribution, $q_{\varphi}(Z|X)$, to be as
similar as possible to the prior distribution, $p(Z)$. In this case, the
overall VAE loss can be written as:

$\mathcal{L}_{VAE}(\theta,\varphi)=\mathcal{L}_{recon}(\theta,\varphi)+\mathcal{L}_{KL}(\theta,\varphi),$
(3)

where the reconstruction loss, $\mathcal{L}_{recon}(\theta,\varphi)$, and the
KL loss, $\mathcal{L}_{KL}(\theta,\varphi)$, are computed as follows:

$\mathcal{L}_{recon}(\theta,\varphi)=\sum_{i=1}^{N}{\|\hat{X}-X\|}_{2}^{2},$
(4)
$\mathcal{L}_{KL}(\theta,\varphi)=\mathcal{D}_{KL}(q_{\varphi}(Z|X)\|p(Z)).$
(5)

In this work we chose to use $\beta$-VAE as the backbone of our approach to
encourage the disentanglement as it is easy to formulate and it has shown good
performance based on one or more disentanglement metrics [27, 10].

The $\beta$-VAE approach [27] is an extension of the standard VAE that aims to
learn a disentangled representation of the encoded variables in a completely
unsupervised manner [42, 27] by simply giving more weight to the KL term,
compared to the original VAE, with an extra hyperparameter $\beta$:

$\mathcal{L}_{VAE}(\theta,\varphi)=\mathcal{L}_{recon}(\theta,\varphi)+\beta\mathcal{L}_{KL}(\theta,\varphi),$
(6)

###  2.3 Attribute-based regularization

In order to better interpret the data attributes that are encoded in the
latent space, we employ an attribute-based regularization loss [51], which
aims at encoding encode an attribute $a$ along a dimension $d$ of the latent
space (regularized dimension).

The attribute regularization loss, $\mathcal{L}_{AR}$, is calculated for the
dimension $d$ of the latent space in a training batch containing $n$ training
examples for the purpose of forcing the dimension $d$ to have a monotonic
relationship with the attribute values of $a$. The attribute regularization
loss is then computed as follows:

$\mathcal{L}_{AR}(d,a)=MAE(tanh(\delta{Dist}_{Z^{d}})-sgn({Dist}_{a})),$ (7)

where $MAE$ is the mean absolute error, ${Dist}_{a}$ is the attribute distance
matrix, and ${Dist}_{Z^{d}}$ is the distance matrix of the latent dimension
$d$. These matrices are computed for all $n$ data examples in the
corresponding training batch, such that:

${Dist}_{a}=a(X_{i})-a(X_{j}),$ (8) ${Dist}_{Z^{d}}=Z_{i}^{d}-Z_{j}^{d},$ (9)

where $i,j\in[0,n)$, $X_{i}$ and $X_{j}$ are two exemplary samples (Equation
8), and each $D$-dimensional latent vector is represented as $Z=\\{Z^{d}\\}$,
where $d\in[0,D)$ (Equation 9).

In Equation 7, $tanh$ and $sgn$ refer to hyperbolic tangent function and sign
function, respectively, whereas $\delta$ is the hyperparameter that modulates
the spread of the posterior distribution. For multiple selected attributes of
interest to be encoded in the latent space, the overall loss function can be
computed by summing all the corresponding objective functions together.
Specifically, when the attribute set is $A:\\{a_{k}\\}$, where $k\in[0,K)$
contains $K$ attributes ($K\leq D$, being $D$ the latent size), then the
overall loss function is computed as:

$\mathcal{L}_{AR}=\sum_{k=0}^{K-1}\mathcal{L}_{d_{k},a_{k}},$ (10)

where $d_{k}$ represents the index of the regularized dimension for the
attribute $k$. This process is represented in Figure 1 (b).

###  2.4 Classification network

Recently, performing a classification task using VAEs has been proposed to
learn and separate different cohorts in the latent space. For example, Biffi
et. al. [8] classified heart pathologies with cardiac remodelling using
explainable task-specific shape descriptors learned directly with a VAE
architecture from the input segmentations. Additionally, other approaches
based on VAE have also been applied to analyse coronary artery diseases [18],
Alzheimer’s disease [64] or to predict the response of cardiomyopathy patients
to cardiac resynchronization therapy [53].

In this line, to enforce class separation to the Attri-VAE, a multilayer
perceptron (MLP) prediction network was connected to the latent vector,
$p(y_{c}|Z)$ ( see Figure 1). The corresponding objective function can be
computed as the binary cross entropy (BCE) between the network prediction
$y_{c}$ and the ground truth label $y_{GT}$, such that:

$\mathcal{L}_{MLP}=BCE(y_{c},y_{GT})$ (11)

###  2.5 Attribute-based attention generation

The Attri-VAE facilitates data interpretation by generating new data samples
as a result of scanning the regularized latent dimensions. Furthermore, it
also provides a way to obtain attention maps from these dimensions (attribute-
based attention map generation) for a better understanding on how gradient
information of these attributes flows inside the proposed architecture (as can
be seen in Figure 2).

Attribute-based visual attention maps were generated by means of gradient-
based computation (Grad-CAM) [63], as proposed by [41]. Basically, a score is
calculated from the latent space that is then used to estimate the gradients
and attention maps. Specifically, given the posterior distribution inferred by
the trained network for a data sample $X$, $q_{\varphi}(Z|X)$, the
corresponding $D$-dimensional latent vector $Z$ is sampled using the
reparameterization trick [33]. Subsequently, for a given attribute set
$A:\\{a_{k}\\}$, where $k\in[0,K)$ contains $K$ attributes, attribute-based
attention maps, $M^{d_{k}}$, are generated for each regularized latent
dimension $Z^{d_{k}}$ by backpropagating the gradients to the encoder´s last
convolutional feature maps ($F:\\{F_{i}\\}$ where $i\in[0,n)$):

$M^{d_{k}}=ReLU(\sum_{i=1}^{n}w_{i}F_{i}),$ (12)

where $d_{k}$ is index of the regularized latent dimension for a given
attribute $k$. The weights, $w_{i}$, are computed using global average pooling
(GAP), which allows us to obtain a scalar value, as follows:

$w_{i}=GAP(\frac{\partial Z^{d_{k}}}{\partial
F_{i}})=\frac{1}{T}\sum_{p=1}^{j}\sum_{q=1}^{l}(\frac{\partial
Z^{d_{k}}}{\partial F_{i}^{pq}}),$ (13)

where $T=j\times l$, (i.e., $width\times height$), and $F_{i}^{pq}$ is the
pixel value at location $(p,q)$ of the $j\times l$ matrix $F_{i}$. This
process is visually summarized in Figure 2.

##  3 Application for interpretable cardiology

###  3.1 Datasets

Initially, the EMIDEC dataset [36] was used in our experiments. It is a
publicly available database with delay-enhancement magnetic resonance images
(DE-MRI) of 150 cases (100 and 50 cases for training and testing,
respectively), with the corresponding clinical information. Each case includes
a DE-MRI acquisition of the left ventricle (LV), covering from base to apex.
The training set, with ground-truth segmentations, includes 67 myocardial
infarction (MINF) cases and 33 healthy subjects. The testing set includes 33
MINF and 17 healthy subjects. Some clinical parameters were also provided
along with the MRI: sex, age, tobacco (yes, no, and former), overweight,
arterial hypertension, diabetes, family history of coronary artery disease,
electrocardiogram (ECG), Killip max333A score based on physical examination
and the development of the heart failure to predict the risk of mortality.,
troponin444A parameter that shows the level of the protein that is released
into the bloodstream., LV ejection fraction (EF), and NTproBNP555A parameter
that shows a level of a peptide, which is an indicator for the diagnosis of
heart failure.. Furthermore, we also used an additional external testing
dataset for a more robust assessment of the classification performance, the
ACDC MICCAI17 challenge training dataset 666https://www.creatis.insa-
lyon.fr/Challenge/acdc/ (end-diastole, ED, and end-systole, ES, cine-MRI from
20 healthy volunteers and 20 MINF cases). The ACDC dataset includes ground-
truth segmentations of the left ventricle, myocardium, and right ventricle by
an experienced manual observer at both ED and ES time points [7]. The reader
is referred to [36, 7] for more details on the MRI acquisition protocol.

As a pre-processing step, the intensities of the left ventricle in all images
were scaled between 0 and 1. Additionally, each image was cropped and padded
($x=80$; $y=80$; $z=80$; $t=1$).

###  3.2 Cardiac attributes

Three different types of attributes were studied in our experiments.
Initially, the Attri-VAE was trained with cardiac shape descriptors (e.g.,
wall thickness, LV and myocardial volumes, ejection fraction), extracted from
ground-truth segmentations, which can easily be visually interpreted. In
addition, attributes available in the clinical information with the highest
discriminative performance were identified using recursive feature elimination
(RFE) with a support vector machine (SVM) classification model (linear kernel,
regularization parameter $C=10$) since this approach has already shown good
performance for feature selection tasks [28, 61, 72]. In total 12 clinical
attributes were provided with the EMIDEC dataset as introduced in Section 3.1
and the most discriminative attributes were then included in our analysis
(e.g., gender, age, tobacco). The feature selection pipeline was done using
the python-based machine learning library scikit-learn (version
1.0.2).777https://scikit-learn.org/stable/

Finally, the Attri-VAE was also trained with radiomics features. Radiomics
analysis was originally proposed to capture alterations at both the
morphological and tissue levels in oncology applications[2, 38], deriving
multiple quantifiable features from pixel-level data. More recently, radiomics
approaches have provided promising results on cardiac MRI data, for
discriminating different cardiac conditions [49, 40, 4, 14], and to study
cardiovascular risk factors in large databases [13]. Radiomics analysis
represents a step towards interpretability compared to other black-box
approaches since some features can be related to pathophysiological mechanisms
[13]. However, there is a need for improving the robustness and
reproducibility of radiomics outcomes across different feature selection
strategies and imaging protocols, which would lead to enhanced explainability.
For this reason, radiomics features were employed in our experiments to
benefit from the proposed network’s ability to explain the encoded attributes.
The open-source library PyRadiomics (version 3.0.1)
888https://pyradiomics.readthedocs.io/ was used to derive 214 features per
analysed cardiac structure including 28 shape-based, 36 intensity-based, and
150 texture-based features. Subsequently, radiomics features with the highest
discriminative performance were identified using the above-mentioned feature
selection approach as this strategy has also demonstrated good performance in
previous radiomics studies [73, 71, 17]. The top-performing features of this
process were then selected to train the Attri-VAE.

###  3.3 Architectural details

The 3D convolutional encoder of the proposed Attri-VAE framework compresses
the input into a 250-dimensional embedding through a series of 5 3D
convolutional layers with kernel size 3 and stride 2, except the last
convolutional layer that has stride 1. Note that, using a series of 3 fully
connected layers, the latent dimension was set to 64. The prediction network
was constructed with a shallow 3-layer MLP to be able to discriminate between
the healthy and infarct subjects, using a ReLU activation function as a non-
linearity after the first two layers. The upsampling and convolutional layers
used in the encoder and the decoder were followed by batch normalization and
ReLU non-linearity, except the decoder’s last convolutional layer (Attri-VAE
output) where a sigmoid function was applied. All the network weights were
randomly initialized with Xavier initialization [22]. The tunable parameters
of the loss function (Equation 1) were fixed as follows: KL weight $\beta=2$;
and regularization weight $\gamma=200$. Additionally, $\delta$ (Equation 7)
was set to 10. We provided detailed information on the model architecture of
the proposed Attri-VAE, including our publicly available code, in our GitHub
repository 999https://github.com/iremcetin/Attri-VAE.

The Attri-VAE was trained on an NVIDIA Tesla T4 GPU using Adam optimizer with
a learning rate equals to 0.0001 and batch size of 16 for 10000 epochs. The
dataset was split into 70/30 training (47 pathological, 23 healthy) and
testing (20 pathological and 10 healthy subjects) sets. Subsequently, random
oversampling of the normal subjects was employed in the training set as a
strategy to treat the unbalanced behavior of the dataset; however, the testing
set was kept unchanged. Note that the proposed model is implemented using
python programming language and PyTorch library (version 1.10.0)
101010https://pytorch.org/. Image pre-processing and transformations were done
using the python-based MONAI library (version 0.8.0) 111111https://monai.io/.

###  3.4 Experimental setting and evaluation criteria

The performance of the proposed Attri-VAE, both qualitatively and
quantitatively, was compared with VAE, $\beta$-VAE, and AR-VAE. As all three
methods are the special cases of the proposed model, we demonstrated this
comparison in several ablation studies by removing different components of the
proposed network (i.e., $\beta$, MLP, and attribute-regularization (AR)
components) such that VAE represents the removal of $\beta$, MLP and AR
components of the Attri-VAE, $\beta$-VAE represents Attri-VAE without MLP and
AR losses and AR-VAE is Attri-VAE without MLP loss. First of all, the degree
of disentanglement of the proposed latent space was evaluated with respect to
different data attributes, using the following metrics available in the
literature: the modularity metric, to analyse the dependence of each dimension
of the latent space on only one attribute [56] such that if the latent
dimension is ideally modular then it will have high mutual information with a
single attribute and zero mutual information with others [56]; the mutual
information gap (MIG), to evaluate the MI difference between a given attribute
and the top two dimensions of the latent space that share maximum MI with the
corresponding attribute [16]; the separated attribute predictability (SAP), to
measure the difference in the prediction error of the two most predictive
dimensions of the latent space for a given attribute [35]; and the Spearman
correlation coefficient (SCC) score, to compute its maximum value between an
attribute and each dimension of the latent space.

In parallel, the interpretability metric introduced in [1] was used to measure
the ability to predict a given attribute using only one dimension of the
latent space. For this, a linear probabilistic relationship between the
corresponding data attribute and the regularized latent space dimension was
calculated. Then, the interpretability score was computed by summing up the
logarithms of these probabilities corresponding to each test sample [1]. As
for the $\beta$-VAE model, dimensions having a high MI with the corresponding
data attribute were chosen for the interpretability estimation. The
reconstruction fidelity performance was also evaluated, employing the maximum
mean discrepancy (MMD) score [24], which measures the distance between the
distributions of real and reconstructed data examples, as well as their mutual
information (MI) as an image similarity metric. The interpretability and MI
metrics were then used to identify the optimal values of the most relevant
hyperparameters in Equation 1 and Equation 7 (i.e., $\beta$, $\gamma$, and
$\delta$), evaluating the influence of the KL divergence ($\beta$) and
attribute regularization ($\gamma$) loss terms, as well as the weight of the
distance matrix between two samples in a latent dimension. As a proof-of-
concept, the hyperparameter sensitivity analysis was performed with only the
four cardiac shape-based interpretable attributes.

Another set of experiments was carried out to explore the potential of the
latent space generated by the Attri-VAE to create synthetically realistic
samples. First, two samples in the Attri-VAE latent space, corresponding to
input data with distinct cardiac characteristics (e.g., thin vs. thick
myocardium, absence vs. presence of myocardium infarct), were chosen as
references to synthetically generate interpolated images through their
trajectory. Secondly, we qualitatively evaluated the control over individual
data attributes during the generation process of the Attri-VAE model. Given a
sample with a latent code $z$, a given attribute (e.g., LV volume) can be
scanned from low to high values changing the latent code of the corresponding
regularized dimension, due to their monotonic relationship. The attribute
scanning creates synthetically generated samples in a latent space trajectory
where only the chosen attribute is changed, facilitating its interpretation.
To further facilitate the identification of each attribute’s visual influence
in the synthetically generated images, gradient-based attention maps were also
estimated.

Finally, the performance of the Attri-VAE model for classifying healthy and
pathological hearts was assessed using the area under the curve (AUC) and
accuracy (ACC) metrics, using both the EMIDEC and the ACDC17 challenge
datasets. The Attri-VAE results were benchmarked against other VAE-type
approaches (VAE+MLP, $\beta$-VAE+MLP), as well as to classical radiomics
analysis (with SVM). The latent space projections of the Attri-VAE model,
regularized by different attributes, were also qualitatively analysed to
identify the attributes better differentiating healthy and pathological
clusters of samples.

##  4 Results

###  4.1 Disentanglement and interpretability

The proposed Attri-VAE approach obtained better disentanglement metric scores
than its ablated variants (i.e., $\beta$-VAE and AR-VAE) using shape and
clinical attributes, implying a more disentangled latent space.

Firstly, all models provided high modularity values (Attri-VAE: 0.98 vs.
$\beta$-VAE: 0.97 vs. 0.98 AR-VAE), signaling that each dimension of the
latent spaces in all models only depended on one data attribute. The Attri-VAE
also resulted in higher MIG/SAP scores than the others (Attri-VAE: 0.60/0.63
vs. $\beta$-VAE: 0.02/0.05 vs. AR-VAE: 0.51/0.55). In its turn, the SCC metric
estimated for Attri-VAE was substantially higher than the corresponding
$\beta$-VAE one and only slightly higher than the one for AR-VAE (Attri-VAE:
0.97 vs. $\beta$-VAE: 0.46 vs. AR-VAE: 0.96) due to the monotonic relationship
between a given attribute and the regularized latent dimension enforced by
both Attri-VAE and AR-VAE. When using radiomics features, the same trend was
observed, with some Attri-VAE disentanglement metrics (MIG and SAP) slightly
lower than when using shape and clinical attributes (Attri-VAE / $\beta$-VAE
/AR-VAE): modularity, 0.98/0.98 /0.98; MIG, 0.49/0.01/0.49; SAP,
0.51/0.06/0.51; SCC, 0.98/0.42/0.97).

Figure 3: Three examples of real and reconstructed images using the VAE, $\beta$-VAE, AR-VAE, and Attri-VAE approaches. Three slices are shown in every example: apical (APEX), mid-ventricle (MID), and basal (BASE) slices. Sample 1 and 3 correspond to healthy hearts while Sample 2 shows an infarcted myocardium. Table 1: Interpretability score [1] of most relevant shape, clinical and radiomics attributes, as encoded in the latent space, with the Attri-VAE, $\beta$-VAE, and AR-VAE approaches. LV: left ventricle, MYO: myocardium, EF: ejection fraction. Max: maximum, DE: difference entropy, ZP: zone percentage. Maximum interpretability is 1.0. | Attri-VAE |  $\beta$-VAE | AR-VAE
---|---|---|---
LV volume | 0.89 | 0.14 | 0.80
MYO volume | 0.93 | 0.02 | 0.87
Wall thickness | 0.95 | 0.10 | 0.90
EF | 0.94 | 0.03 | 0.90
Gender | 0.98 | 0.19 | 0.94
Age | 0.93 | 0.12 | 0.84
Tobacco | 0.70 | 0.19 | 0.74
Radiomics | 0.91 | 0.06 | 0.90

Table 1 shows the interpretability scores for Attri-VAE, $\beta$-VAE, and AR-
VAE obtained with clinical and shape descriptors as well as radiomics
features. The radiomics feature selection identified seven of them having the
most discriminative power: four shape-based, being the sphericity of the left
ventricle, the maximum 2D diameter of the myocardium, as well as left
ventricle and myocardial volumes; three texture-based, being the correlation
of the left ventricle, the difference entropy of the myocardium and the
inverse variance of the left ventricle.

It can easily be observed that the Attri-VAE provided a high degree of
interpretability (i.e., close to 1.0) for all attributes, with the exception
of tobacco (0.70). Among shape and clinical features, gender was the attribute
with a higher interpretability (0.98), followed by the wall thickness (0.95),
meaning that they could be predicted with only one dimension of the latent
space. As for radiomics features, the average interpretability metric value
was 0.91, with shape-based ones showing slightly larger values than texture
features (0.93 and 0.89, respectively); the maximum 2D diameter of the
myocardium presented the highest value (0.97). On the other hand, the
$\beta$-VAE clearly resulted in lower interpretability values (average of 0.11
for shape/clinical attributes and 0.06 for radiomics features) and AR-VAE
obtained slightly lower interpretability metric score (average of 0.86 for
shape/clinical attributes and 0.90 for radiomics features) than the proposed
Attri-VAE.

###  4.2 Reconstruction fidelity

Reconstruction fidelity is another important factor as the high degree of
disentanglement and interpretability should not be accompanied by a reduced
reconstruction. Hence in this experiment, we demonstrated the reconstruction
quality of the proposed approach by comparing its performance with its ablated
variants, specifically VAE, $\beta$-VAE, and AR-VAE.

The proposed Attri-VAE approach obtained the lowest MMD values, representing a
lower distance between input and reconstructed images (see Table 2). We
observed that removing only the MLP loss from Attri-VAE (AR-VAE) obtained the
best MI value of 0.92 (0.91 and 0.89 for VAE and Attri-VAE, respectively).

Table 2: Ablation study on the reconstruction accuracy of Attri-VAE on the EMIDEC dataset, quantified with the maximum mean discrepancy (MMD) and mutual information (MI) metrics. The MMD results are given as $\pm$ standard deviation. w/o : without, AR: attribute-regularization | MMD $\times 10^{2}$ | MI
---|---|---
VAE (w/o $\beta$, MLP and AR) | 1.86 $\pm$ 0.06 | 0.91
$\beta$-VAE (w/o MLP and AR) | 1.38 $\pm$ 0.04 | 0.87
AR-VAE (w/o MLP) | 1.74 $\pm$ 0.06 | 0.92
Attri-VAE | 1.18 $\pm$ 0.03 | 0.89

The reconstructions of three data examples from the EMIDEC dataset where the
performance of the Attri-VAE was compared with its different variants, can be
seen in Figure 3. Even though all models achieved similar qualitative
reconstruction results, the Attri-VAE model generated images better preserving
the heart shape and details than the other models: see the papillary muscles
in mid-myocardium slices (dark regions in the blood pool) or the left
ventricular cavity in apical slices of Sample 2 and Sample 3 in Figure 3. We
can also observe in the figure that apical slices were more difficult to
reconstruct than mid-ventricle and basal ones for the three tested models.

###  4.3 Hyperparameter sensitivity analysis

Figure 4: Effect of hyperparameters on the interpretability and reconstruction
fidelity of the Attri-VAE approach. The hyperparameters $\beta$ and $\gamma$
of the Attri-VAE model control the influence of the loss terms for the
Kullback-Leibler divergence between learned prior and posterior distributions,
and attribute regularization, respectively. Each marker represents a unique
combination of the hyperparameters $\beta$ and $\gamma$, which is indicated by
color and shape, respectively. For comparison, the performance of $\beta$-VAE
($\beta$ = 3) is also represented. Best performance combinations are located
in the top right corner of the graph.

This study evaluates the impact of hyperparameters on the interpretability and
reconstruction quality of the Attri-VAE. The trade-off between reconstruction
quality and interpretability can be seen in Figure 4 where the performance of
$\beta$-VAE ($\beta$ = 3) is also provided for comparison. A visual inspection
of the figure suggests that $\gamma$, i.e., the hyperparameter controlling the
attribute regularization, was the key to obtaining good interpretability
values while keeping reasonable reconstruction fidelity (mutual information
$\geq$ 0.88), with values of $\gamma\geq$ 100.

On the other hand, the $\beta$ hyperparameter was not as relevant as the
$\gamma$ hyperparameter. As expected, the $\beta$-VAE provided acceptable
reconstruction fidelity results but low values of interpretability. We need to
point out that the same results were obtained when using radiomics features
instead of shape-based attributes.

###  4.4 Latent space interpolation and attribute scanning

This experiment aims to qualitatively evaluate the proposed latent space by
demonstrating its ability to interpolate between different data examples and
control individual attributes during the generation process.

Figure 5: Linear latent space interpolation between two data samples (extremes
of each row in yellow frames) from the EMIDEC dataset. Each row depicts the
interpolation from the left to the right latent vector dimension. Top: from
thin to thick myocardium. Middle: from a myocardium with a scar to one
without. Bottom: from a healthy subject to a patient with a myocardial
infarct. Figure 6: Scanning of attributes and corresponding gradient-based
attention maps for shape and radiomics features. The image in the middle (4th
column, in yellow frame) shows the original reconstructed image. DE:
difference entropy, IV: inverse variance, Max 2D dia: maximum 2-dimensional
diameter, LV: left-ventricle, MYO: myocardium. Note that the first three rows
demonstrate the attribute scanning that was done on the latent space of Attri-
VAE, which was trained with clinical and shape features. The remaining rows
represent the attribute scanning on the latent space of Attri-VAE trained with
selected radiomics features.

For the first experiment, the interpolation was employed, between distinct and
well-separated samples in the learned latent space of the Attri-VAE. The
proposed approach generates synthetic interpolated images that have a
realistic appearance, gradually changing the main sample characteristics in
the trajectory between the chosen samples. The first row of Figure 5 clearly
demonstrates the Attri-VAE model’s ability to create smooth transitions
between hearts having largely different characteristics such as (thin to
thick) wall thickness. The other two rows of the figure demonstrate a similar
behavior from non-infarcted/scar to infarcted/scar patients.

The second experiment presents the effect of scanning an individual attribute
along its corresponding regularized dimension in the Attri-VAE model, where
all the remaining attributes remain fixed. The first three rows of Figure 6
exemplify the attribute scanning that was done on the latent space of Attri-
VAE, which was trained with clinical plus shape features. The rest of the rows
represent the attribute scanning on the latent space of Attri-VAE trained with
selected radiomics features. For shape-based attributes, the changes in the
attribute when moving along different values of the regularized dimension are
clearly seen. For instance, from the left to the right in Figure 6, how LV and
myocardial volumes are increasing in the first and second rows, respectively,
or how the LV becomes more spherical. More subtle changes are observed with
texture-based radiomics but they can still be identified with a careful
inspection of the generated images. For example, moving along the latent space
dimension corresponding to the correlation LV, we find more or less intensity
homogeneity in the LV. The LV inverse variance (LV-IV) and the difference
entropy of the myocardium (DE-MYO) only produced small changes that consisted
in slightly thicker myocardium with lower values of LV-IV (left samples in
Figure 6) and some darker patches and heterogeneous texture in the myocardium
for higher values of DE-MYO (right in Figure 6). It needs to be pointed out
that attribute scanning for clinical attributes such as age, gender, and
tobacco is not shown since the images do not visually change along the
corresponding regularized dimensions.

To further illustrate the impact of each studied attribute, we have also
generated the gradient-based attention maps linked to the changes in each
regularized latent dimension of the Attri-VAE model. Attention maps show the
high response regions in the images when changing specific dimensions of the
latent space that correspond to specific attributes. We can see in Figure 6
that more attention (i.e., higher response) is paid to more varying regions
for shape-based attributes (e.g., right side of the slide for LV volume, where
LV is increasing from the left to the right in the regularized dimension). In
general, attention maps for texture-based features have less high-response
regions than for shape-based attributes. However, in some texture-based
features such as the difference entropy of the myocardium, a higher response
can still be localized (in this example, darker regions in the top left part
of the slice). On the other hand, interpretation and validation of the
resulting attention maps for other attributes such as for LV-IV are more
challenging.

###  4.5 Classification

This experiment illustrates the classification performance of the proposed
Attri-VAE. Additionally, we have also compared its performance with several
models, such as radiomics analysis and its ablated versions. The best result
was obtained in both EMIDEC and ACDC datasets with the Attri-VAE trained with
radiomics features (accuracy of 0.97 and 0.58 for both datasets), while
radiomics+MLP was the worst for EMIDEC (accuracy of 0.76) and the
$\beta$-VAE+MLP for ACDC (accuracy of 0.45) (see Table 3). There were only
minor differences in the accuracy of the Attri-VAE method when trained with
clinical and shape attributes or radiomics features. All the evaluated models,
trained with the EMIDEC data, substantially dropped their performance when
tested on the external ACDC dataset, especially the VAE-based approaches.

Table 3: Ablation study on the classification performance of Attri-VAE with EMIDEC and ACDC datasets (healthy vs. myocardial infarction) with different models. The results are reported as accuracy / AUC scores. SVM: support vector machine, w/o : without, AR: attribute-regularization | EMIDEC | ACDC
---|---|---
Attri-VAE (Clinical+Shape) | 0.93 / 0.94 | 0.55 / 0.54
Attri-VAE (Radiomics) | 0.97 / 0.96 | 0.58 / 0.52
$\beta$-VAE+MLP (w/o AR) | 0.90 / 0.90 | 0.45 / 0.31
VAE+MLP (w/o $\beta$ and AR) | 0.87 / 0.80 | 0.53 / 0.35
Radiomics analysis (SVM) | 0.77 / 0.75 | 0.60 / 0.61
Radiomics analysis (MLP) | 0.76 / 0.75 | 0.58 / 0.60
Figure 7: Latent space projections of regularized dimensions for different
clinical, shape, and radiomics attributes. Each point in the graphs represents
a healthy or a myocardial infarction patient (red and blue, respectively), LV:
left-ventricle, MYO: myocardium, IV: inverse variance, DE: difference entropy,
Max 2D dia: maximum 2-dimensional diameter.

Finally, the latent space projections of different regularized latent
dimensions are visualized in Figure 7, with plot axes representing the encoded
data attributes. As it can be observed in the figure, our model is able to
build several reduced dimensionality spaces, based on different attributes,
where healthy and pathological cases (red and blue in the figure,
respectively) can easily be clustered. For instance, the maximum 2D diameter
of the myocardium and the LV volume attributes correctly separate most samples
into two clusters. Interestingly, despite Attri-VAE having poor control over
clinical attributes such as age or gender, they also facilitate the
construction of the latent spaces and sample discrimination, as can be seen in
the gender-age plot of Figure 7.

##  5 Discussion

The analysis of medical data demands for interpretable methods. However, the
majority of deep learning methods do not fulfill the minimum level of
interpretability to be used in reasoning medical decisions [62], being
difficult to relate clinically and physiologically meaningful attributes with
model parameters and outcomes. Fortunately, interpretable and explainable deep
learning methods are starting to emerge. Models creating latent space
representations, such as variational autoencoders, are promising but
attributes are usually entangled in the resulting reduced dimensionality
space, hampering its interpretation. In this work, we have presented the
Attri-VAE approach that generates disentangled and interpretable
representations where different types of attributes (e.g., clinical, shape,
radiomics) are individually encoded into a given dimension of the resulting
latent space. The results obtained by the proposed Attri-VAE model based on
disentanglement and interpretability metrics clearly outperformed the state-
of-the-art $\beta$-VAE approach and obtained slightly better results than AR-
VAE, indicating a high degree of disentanglement and a monotonic relationship
between a given attribute and the corresponding regularized dimension.
However, Attri-VAE values for some metrics such as the MIG and SAP, although
substantially better than those of $\beta$-VAE, were far from the maximum
(e.g., 1.0). The same trend was observed by [51] in the MNIST (i.e., for digit
number identification) dataset, suggesting that other latent dimensions,
beyond the regularized ones, share a high MI with different attributes. We
would like to point out here that the performance of Attri-VAE was compared
with several methods including, AR-VAE [51], further work is needed to compare
the performance of the proposed approach with additional models, such as
Guided-VAE [20]. More work is also required to visually evaluate the model
interpretability by human experts in addition to the quantitative results, in
order to better illustrate the interpretation performance of the proposed
approach.

Hyperparameter selection was a key step for finding the optimal Attri-VAE
configuration providing an excellent trade-off between reconstruction
fidelity, at the level of state-of-the-art alternatives, and interpretability;
even though the Attri-VAE approach had a more constrained latent space, it
generated reconstructions that are less smooth than other VAE-based models and
more similar to the original input images. This explains why the proposed
Attri-VAE had a slightly lower MI value and higher MMD and interpretability
values than the other models. The most critical parameter to enforce
interpretability was the weight of the attribute regularization loss term
($\gamma$ in Equation 1). The Attri-VAE plot of reconstruction fidelity vs.
interpretability, shown in Figure 4 had the same pattern as the one obtained
by [51]. Interestingly, their optimal $\gamma$ values were lower than ours
([5.0, 10.0] vs $\geq$ 100), likely due to the higher complexity of the
cardiac MRI data and corresponding latent space compared to the MNIST dataset.
On the other hand, the best $\delta$ values were the same in the two studies
([1.0, 10.0]). However, it is worth noting that we did not evaluate the effect
of the size of the latent space and the position of regularized dimensions.
Therefore, future work is needed to analyze the size of the latent space and
the importance of the position of regularized dimensions by, for example,
randomly replacing them during inference. Additionally, more work is required
to determine how various losses (i.e., classification and attribute losses)
affect the latent space as a whole and if this is dependent on the latent
space’s dimensionality.

One of the most interesting characteristics of the Attri-VAE approach is the
ability to create realistic synthetic data by sampling the created latent
space and interpolating between different original reconstructed inputs, which
can be very useful for controllable and attribute-based data augmentation of
training datasets in machine learning applications. Scanning a regularized
dimension of the latent space creates synthetic images where the corresponding
attribute changes its values, as can easily be observed for shape descriptors
(e.g., LV and myocardial volumes, wall thickness) in Figure 6. In addition,
the proposed approach allows a better understanding of some (texture-based)
radiomics features, which are often difficult to interpret. However, clinical
attributes such as age, gender, or tobacco consumption, despite obtaining good
interpretability scores, did not create visually different interpolated
samples over the regularized dimensions. One potential reason is the
difficulty of the attribute regularization to control binary attributes, as
suggested by [51]. Furthermore, the studied clinical attributes cannot be
disassociated from the shape and image intensity variations (e.g.,
morphological changes of the heart with age), thus it is too restrictive to
keep all attributes fixed except a clinical one. In consequence, we have
employed the clinical attribute referring to healthy vs. myocardial infarction
as a task-specific label, as the aim of this classification is to be able to
separate different groups of patients in the latent space, by enforcing the
continuous variables to be able to predict the desired class and provide
interpretable results. More work is needed to better construct latent spaces
where clinical information can be disentangled from other attributes.
Additionally, we would like to point out that the proposed Attri-VAE was
trained on delineated left ventricle images; future work needs to include
other cardiac structures to integrate global changes in cardiac tissue and
shape.

The generated gradient-based attention maps contributed to locally identifying
the cardiac regions where the attributes were influencing, which was
particularly useful for global attributes and for complex features such as the
texture ones. Additionally, we only employed the well-known Grad-CAM method,
which could be complemented with additional interpretability methods (e.g.,
LIME and its variations [55]) to better understand the attribute effects on
the latent space. However, the attention maps have some limitations that have
already been addressed in different studies in the literature, which
demonstrate that saliency maps underperform in some key criteria such as
localization, parameter sensitivity, repeatability, and reproducibility [3].
Thus the reliability of attention maps still requires further investigation to
assess its robustness and reliability with respect to data input and model
parameter perturbations [54]. In parallel, enhanced 3D visualizations of the
generated samples are needed to have an overall perspective of the cardiac
differences, beyond 2D slice views of the resulting images.

The proposed Attri-VAE model also achieved excellent classification
performance (healthy vs. myocardial infarction), outperforming the other VAE-
based approaches, with slightly better results when trained with radiomics.
When evaluated with the EMIDEC training dataset using ground-truth labels, the
Attri-VAE approach provided accuracy results (0.98) equivalent to the best
challenge participants reporting their performance on the same dataset (1.0
[44], 0.95 [67], 0.94 [29] and 0.90 [65]). For the testing EMIDEC dataset
[37], the best participant method obtained a decreased accuracy (0.82, [44,
21]), increasing to 0.92 for the challenge organizers [67]. As for the ACDC
dataset, which was tested as an external database (i.e., without considering
it in training), classification accuracy was substantially reduced (0.58),
being worst than results reported by challenge participants [7] (0.96) to
classify between the different pathologies (not only between healthy and
myocardial infarction). On the other hand, we would like to point out that
these two datasets have employed different image acquisition techniques and
contain images from different imaging modalities, such that the EMIDEC dataset
consists of DE-MRI images and the ACDC dataset contains cine-MRI images.
However, further work is required to find out the main reason behind this
performance drop including evaluating the Attri-VAE’s performance in
comparison with an additional model, such as a baseline CNN. Additionally,
more work is also needed to improve the reconstruction quality and the
generalization of the Attri-VAE model to unseen data, being more robust to
different quality and imaging acquisition protocols, through domain adaptation
techniques or image registration or integrating these differences into its
latent space, using databases such as the M&Ms challenge [11].

One limitation of the Attri-VAE approach, also acknowledged by Pati and Lerch
[51], is the dependence on the selection of the data attributes to train the
model. An incorrect attribute selection could lead to undesired strong
correlations of several attributes that will not ensure a monotonic
relationship with the corresponding regularized dimension, leading to less
attribute interpretation and reconstruction quality. However, the projection
of original samples in latent spaces with regularized dimensions for different
attributes (see Figure 7) could be used as an interpretable attribute
selection, identifying the ones better separating the analyzed classes such as
the maximum 2D diameter of the myocardium and the LV volume attributes in our
experiments. Further work will focus on fully integrating advanced feature
selection techniques with the Attri-VAE model, as well as exploring
alternative interpretability methods (see the recent review of Salahuddin et
al. [60]) to better understand the role of clinical and imaging attributes on
medical decisions in cardiovascular applications. Self-supervision will also
be explored, as an opportunity to make use of unlabeled data to further
improve our results.

##  6 Conclusions

We have presented an approach, referred to as Attri-VAE, which implements
attribute-based regularization in a $\beta$-VAE scheme with a classification
module for the purpose of attribute-specific interpretation, synthetic data
generation, and classification of cardiovascular images. The basis of the
proposed Attri-VAE model is to structure its latent space for encoding
individual data attributes to specific latent dimensions, being guided by an
attribute regularization loss term. The resulting constrained latent space can
be easily manipulated along its regularized dimensions for an enhanced
interpretation of different attributes. Additionally, the proposed approach
improves the current state-of-the-art for classifying cardiovascular images
and allows the visualization of the most discriminative attributes by
projecting the trained latent space. Future work will be focused on improving
the generalization of the trained Attri-VAE model to images with different
acquisition characteristics.

##  7 Competing interests

The authors declare that they have no competing interests.

##  8 Author contributions

All authors participated in the critical revision of the manuscript, and final
approval of the submitted manuscript. IC, MAGB, and OC contributed to the
study concepts, methods, and underlying data collection. IC, MS, MAGB, and OC
drafted the manuscript. IC, MAGB, and OC designed the machine learning
methods. IC performed the data pre-processing and data analysis.

##  9 Funding

This work was partly funded by the European Union’s Horizon 2020 research and
innovation programme under grant agreement No 825903 (euCanSHare project).

##  10 Availability of data and materials

This research was conducted using the publicly available EMIDEC and ACDC
datasets. These datasets can be accessed in http://emidec.com/dataset and
https://www.creatis.insa-lyon.fr/Challenge/acdc/. We have also made our code
publicly available and can be found in https://github.com/iremcetin/Attri-VAE

##  11 Ethical approval

The datasets employed in this study are publicly available sources. Thus, for
the detailed information see http://emidec.com/ and https://www.creatis.insa-
lyon.fr/Challenge/acdc/.

## References

  * Adel et al. [2018] Adel, T., Ghahramani, Z., Weller, A., 2018.  Discovering interpretable representations for both deep generative and discriminative models, in: Dy, J., Krause, A. (Eds.), Proceedings of the 35th International Conference on Machine Learning, PMLR. pp. 50–59.  URL: https://proceedings.mlr.press/v80/adel18a.html.
  * Aerts et al. [2014] Aerts, H.J., Velazquez, E.R., Leijenaar, R.T.H., Parmar, C., Grossmann, P., Cavalho, S., Bussink, J., Monshouwer, R., Haibe-Kains, B., Rietveld, D.H.F., Hoebers, F.J., Rietbergen, M.M., Leemans, C.R., Dekker, A., Quackenbush, J., Gillies, R.J., Lambin, P., 2014.  Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach.  Nature Communications 5, 4006\.  doi:10.1038/ncomms5006.
  * Arun et al. [2021] Arun, N., Gaw, N., Singh, P., Chang, K., Aggarwal, M., Chen, B., Hoebel, K., Gupta, S., Patel, J., Gidwani, M., Adebayo, J., Li, M.D., Kalpathy-Cramer, J., 2021\.  Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging.  Radiology: Artificial Intelligence 3, e200267.  URL: https://doi.org/10.1148/ryai.2021200267, doi:10.1148/ryai.2021200267, arXiv:https://doi.org/10.1148/ryai.2021200267.
  * Baessler et al. [2018] Baessler, B., Luecke, C., Lurz, J., Klingel, K., von Roeder, M., de Waha, S., Besler, C., Maintz, D., Gutberlet, M., Thiele, H., Lurz, P., 2018.  Cardiac MRI Texture Analysis of T1 and T2 Maps in Patients with Infarctlike Acute Myocarditis.  Radiology 289, 357--365.  doi:10.1148/radiol.2019190101.
  * Barredo Arrieta et al. [2020] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F., 2020\.  Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  Information Fusion 58, 82--115.  doi:10.1016/j.inffus.2019.12.012.
  * Bengio et al. [2013] Bengio, Y., Courville, A., Vincent, P., 2013.  Representation learning: a review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence 35, 1798—1828.  doi:10.1109/tpami.2013.50.
  * Bernard et al. [2018] Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.A., Cetin, I., Lekadir, K., Camara, O., Gonzalez Ballester, M.A., Sanroma, G., Napel, S., Petersen, S., Tziritas, G., Grinias, E., Khened, M., Kollerathu, V.A., Krishnamurthi, G., Rohé, M.M., Pennec, X., Sermesant, M., Isensee, F., Jäger, P., Maier-Hein, K.H., Full, P.M., Wolf, I., Engelhardt, S., Baumgartner, C.F., Koch, L.M., Wolterink, J.M., Išgum, I., Jang, Y., Hong, Y., Patravali, J., Jain, S., Humbert, O., Jodoin, P.M., 2018.  Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: Is the problem solved?  IEEE Transactions on Medical Imaging 37, 2514--2525.  doi:10.1109/TMI.2018.2837502.
  * Biffi et al. [2020] Biffi, C., Cerrolaza, J.J., Tarroni, G., Bai, W., de Marvao, A., Oktay, O., Ledig, C., Folgoc, L.L., Kamnitsas, K., Doumou, G., Duan, J., Prasad, S.K., Cook, S.A., O’Regan, D.P., Rueckert, D., 2020.  Explainable anatomical shape analysis through deep hierarchical generative models.  IEEE Transactions on Medical Imaging 39, 2088--2099.  doi:10.1109/TMI.2020.2964499.
  * Bouchacourt et al. [2018] Bouchacourt, D., Tomioka, R., Nowozin, S., 2018.  Multi-level variational autoencoder: Learning disentangled representations from grouped observations.  Proceedings of the AAAI Conference on Artificial Intelligence 32.  URL: https://ojs.aaai.org/index.php/AAAI/article/view/11867.
  * Burgess et al. [2018] Burgess, C.P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., Lerchner, A., 2018.  Understanding disentangling in $\beta$-VAE.  CoRR abs/1804.03599.  URL: http://arxiv.org/abs/1804.03599.
  * Campello et al. [2021] Campello, V.M., Gkontra, P., Izquierdo, C., Martín-Isla, C., Sojoudi, A., Full, P.M., Maier-Hein, K., Zhang, Y., He, Z., Ma, J., Parreño, M., Albiol, A., Kong, F., Shadden, S.C., Acero, J.C., Sundaresan, V., Saber, M., Elattar, M., Li, H., Menze, B., Khader, F., Haarburger, C., Scannell, C.M., Veta, M., Carscadden, A., Punithakumar, K., Liu, X., Tsaftaris, S.A., Huang, X., Yang, X., Li, L., Zhuang, X., Viladés, D., Descalzo, M.L., Guala, A., Mura, L.L., Friedrich, M.G., Garg, R., Lebel, J., Henriques, F., Karakas, M., Çavuş, E., Petersen, S.E., Escalera, S., Seguí, S., Rodríguez-Palomares, J.F., Lekadir, K., 2021\.  Multi-centre, multi-vendor and multi-disease cardiac segmentation: The M&Ms challenge.  IEEE Transactions on Medical Imaging 40, 3543--3554.  doi:10.1109/TMI.2021.3090082.
  * Carter and Nielsen [2017] Carter, S., Nielsen, M., 2017\.  Using artificial intelligence to augment human intelligence.  Distill doi:10.23915/distill.00009. https://distill.pub/2017/aia.
  * Cetin et al. [2020] Cetin, I., Raisi-Estabragh, Z., Petersen, S.E., Napel, S., Piechnik, S.K.P., Neubauer, S., Gonzalez Ballester, M.A., Camara, O., Lekadir, K., 2020.  Radiomics signatures of cardiovascular risk factors in cardiac mri: Results from the uk biobank.  Frontiers in Cardiovascular Medicine 7\.  doi:10.3389/fcvm.2020.591368.
  * Cetin et al. [2017] Cetin, I., Sanroma, G., Petersen, S.E., Napel, S., Camara, O., González Ballester, M.A., Lekadir, K., 2017.  A radiomics approach to computer-aided diagnosis with cardiac cine-mri, in: Pop, M., Sermesant, M., Jodoin, P., Lalande, A., Zhuang, X., Yang, G., Young, A.A., Bernard, O. (Eds.), Statistical Atlases and Computational Models of the Heart. ACDC and MMWHS Challenges - 8th International Workshop, STACOM 2017, Held in Conjunction with MICCAI 2017, Quebec City, Canada, September 10-14, 2017, Revised Selected Papers, Springer. pp. 82--90.  URL: https://doi.org/10.1007/978-3-319-75541-0_9, doi:10.1007/978-3-319-75541-0\\_9.
  * Chartsias et al. [2019] Chartsias, A., Joyce, T., Papanastasiou, G., Semple, S., Williams, M., Newby, D.E., Dharmakumar, R., Tsaftaris, S.A., 2019\.  Disentangled representation learning in cardiac image analysis.  Medical Image Analysis 58, 101535\.
  * Chen et al. [2018a] Chen, R.T.Q., Li, X., Grosse, R.B., Duvenaud, D.K., 2018a.  Isolating sources of disentanglement in variational autoencoders, in: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.  URL: https://proceedings.neurips.cc/paper/2018/file/1ee3dfcd8a0645a25a35977997223d22-Paper.pdf.
  * Chen et al. [2018b] Chen, W., Liu, B., Peng, S., Sun, J., Qiao, X., 2018b.  Computer-aided grading of gliomas combining automatic segmentation and radiomics.  International Journal of Biomedical Imaging 2018\.  doi:https://doi.org/10.1155/2018/2512037.
  * Clough et al. [2019] Clough, J.R., Oksuz, I., Puyol-Antón, E., Ruijsink, B., King, A.P., Schnabel, J.A., 2019\.  Global and local interpretability for cardiac mri classification, in: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2019, Springer International Publishing, Cham. pp. 656--664.  doi:10.1007/978-3-030-32251-9_72.
  * Deng and Liu [2018] Deng, L., Liu, Y., 2018.  Deep Learning in Natural Language Processing.  Springer Publishing Company, Incorporated.
  * Ding et al. [2020] Ding, Z., Xu, Y., Xu, W., Parmar, G., Yang, Y., Welling, M., Tu, Z., 2020\.  Guided variational autoencoder for disentanglement learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7920--7929.
  * Girum et al. [2021] Girum, K.B., Skandarani, Y., Hussain, R., Grayeli, A.B., Créhange, G., Lalande, A., 2021\.  Automatic myocardial infarction evaluation from delayed-enhancement cardiac mri using deep convolutional networks, in: Puyol Anton, E., Pop, M., Sermesant, M., Campello, V., Lalande, A., Lekadir, K., Suinesiaputra, A., Camara, O., Young, A. (Eds.), Statistical Atlases and Computational Models of the Heart. M&Ms and EMIDEC Challenges, Springer International Publishing, Cham. pp. 378--384.
  * Glorot and Bengio [2010] Glorot, X., Bengio, Y., 2010\.  Understanding the difficulty of training deep feedforward neural networks, in: Teh, Y.W., Titterington, M. (Eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR, Chia Laguna Resort, Sardinia, Italy. pp. 249--256.  URL: https://proceedings.mlr.press/v9/glorot10a.html.
  * Goodfellow et al. [2014] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., 2014\.  Generative adversarial nets, in: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 2672--2680.  URL: https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf.
  * Gretton et al. [2007] Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B., Smola, A., 2007.  A kernel method for the two-sample-problem, in: Schölkopf, B., Platt, J., Hoffman, T. (Eds.), Advances in Neural Information Processing Systems, MIT Press.  URL: https://proceedings.neurips.cc/paper/2006/file/e9fb2eda3d9c55a0d89c98d6c54b5b3e-Paper.pdf.
  * Hadjeres et al. [2017] Hadjeres, G., Nielsen, F., Pachet, F., 2017.  GLSR-VAE: Geodesic latent space regularization for variational autoencoder architectures.  2017 IEEE Symposium Series on Computational Intelligence (SSCI) , 1--7doi:10.1109/SSCI.2017.8280895.
  * Higaki et al. [2020] Higaki, T., Nakamura, Y., Zhou, J., Yu, Z., Nemoto, T., Tatsugami, F., Awai, K., 2020.  Deep learning reconstruction at ct : Phantom study of the image characteristics.  Academic Radiology 27, 82--87.  doi:10.1016/j.acra.2019.09.008.
  * Higgins et al. [2017] Higgins, I., Matthey, L., Pal, A., Burgess, C.P., Glorot, X., Botvinick, M.M., Mohamed, S., Lerchner, A., 2017\.  Beta-vae: Learning basic visual concepts with a constrained variational framework, in: ICLR.
  * Huang et al. [2014] Huang, M.L., Hung, Y.H., Lee, W.M., Li, R.K., Jiang, B.R., 2014.  SVM-RFE based feature selection and Taguchi parameters optimization for multiclass SVM classifier.  ScientificWorldJournal 2014, 795624\.  doi:10.1155/2014/795624.
  * Ivantsits et al. [2021] Ivantsits, M., Huellebrand, M., Kelle, S., Schönberg, S.O., Kuehne, T., Hennemuth, A., 2021\.  Deep-learning-based myocardial pathology detection, in: Puyol Anton, E., Pop, M., Sermesant, M., Campello, V., Lalande, A., Lekadir, K., Suinesiaputra, A., Camara, O., Young, A. (Eds.), Statistical Atlases and Computational Models of the Heart. M&Ms and EMIDEC Challenges, Springer International Publishing, Cham. pp. 369--377.
  * Jo et al. [2019] Jo, T., Nho, K., Saykin, A.J., 2019.  Deep Learning in Alzheimer’s Disease: Diagnostic Classification and Prognostic Prediction Using Neuroimaging Data.  Frontiers in Aging Neuroscience 11, 220.  doi:10.3389/fnagi.2019.00220.
  * Kapishnikov et al. [2019] Kapishnikov, A., Bolukbasi, T., Viegas, F., Terry, M., 2019\.  XRAI: Better attributions through regions.  2019 IEEE/CVF International Conference on Computer Vision (ICCV) , 4947--4956doi:10.1109/ICCV.2019.00505.
  * Kim and Mnih [2018] Kim, H., Mnih, A., 2018.  Disentangling by Factorising, in: Dy, J., Krause, A. (Eds.), Proceedings of the 35th International Conference on Machine Learning, PMLR. pp. 2649--2658.
  * Kingma and Welling [2014] Kingma, D.P., Welling, M., 2014\.  Auto-Encoding Variational Bayes, in: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.  arXiv:http://arxiv.org/abs/1312.6114v10.
  * Kofler et al. [2020] Kofler, A., Haltmeier, M., Schaeffter, T., Kachelriess, M., Dewey, M., Wald, C., Kolbitsch, C., 2020.  Neural networks-based regularization for large-scale medical image reconstruction.  Physics in medicine and biology 65, 135003.  doi:10.1088/1361-6560/ab990e.
  * Kumar et al. [2017] Kumar, A., Sattigeri, P., Balakrishnan, A., 2017.  Variational inference of disentangled latent concepts from unlabeled observations.  CoRR abs/1711.00848.  URL: http://arxiv.org/abs/1711.00848, arXiv:1711.00848.
  * Lalande et al. [2020] Lalande, A., Chen, Z., Decourselle, T., Qayyum, A., Pommier, T., Lorgis, L., de la Rosa, E., Cochet, A., Cottin, Y., Ginhac, D., Salomon, M., Couturier, R., Meriaudeau, F., 2020.  Emidec: A database usable for the automatic evaluation of myocardial infarction from delayed-enhancement cardiac mri.  Data 5.  doi:10.3390/data5040089.
  * Lalande et al. [2021] Lalande, A., Chen, Z., Pommier, T., Decourselle, T., Qayyum, A., Salomon, M., Ginhac, D., Skandarani, Y., Boucher, A., Brahim, K., de Bruijne, M., Camarasa, R., Correia, T., Feng, X., Girum, K.B., Hennemuth, A., Huellebrand, M., Hussain, R., Ivantsits, M., Ma, J., Meyer, C.H., Sharma, R., Shi, J., Tsekos, N.V., Varela, M., Wang, X., Yang, S., Zhang, H., Zhang, Y., Zhou, Y., Zhuang, X., Couturier, R., Mériaudeau, F., 2021.  Deep learning methods for automatic evaluation of delayed enhancement-mri. the results of the emidec challenge.  CoRR abs/2108.04016.  URL: https://arxiv.org/abs/2108.04016, arXiv:2108.04016.
  * Lambin et al. [2017] Lambin, P., Leijenaar, R.T.H., Deist, T.M., Peerlings, J., de Jong, E.E.C., van Timmeren, J., Sanduleanu, S., Larue, R.T.H.M., Even, A.J.G., Jochems, A., van Wijk, Y., Woodruff, H., van Soest, J., Lustberg, T., Roelofs, E., van Elmpt, W., Dekker, A., Mottaghy, F.M., Wildberger, J.E., Walsh, S., 2017\.  Radiomics: the bridge between medical imaging and personalized medicine.  Nat Rev Clin Oncol 14, 749--762.  doi:10.1038/nrclinonc.2017.141.
  * Lample et al. [2017] Lample, G., Zeghidour, N., Usunier, N., Bordes, A., Denoyer, L., Ranzato, M., 2017\.  Fader networks: Manipulating images by sliding attributes, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY, USA. p. 5969–5978.
  * Larroza et al. [2018] Larroza, A., López-Lereu, M.P., Monmeneu, J.V., Gavara, J., Chorro, F.J., Bodí, V., Moratal, D., 2018.  Texture analysis of cardiac cine magnetic resonance imaging to detect nonviable segments in patients with chronic myocardial infarction.  Medical Physics 45, 1471--1480.  doi:10.1002/mp.12783.
  * Liu et al. [2020] Liu, W., Li, R., Zheng, M., Karanam, S., Wu, Z., Bhanu, B., Radke, R.J., Camps, O.I., 2020.  Towards visually explaining variational autoencoders, pp. 8639--8648.  doi:10.1109/CVPR42600.2020.00867.
  * Locatello et al. [2019] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., Bachem, O., 2019.  Challenging common assumptions in the unsupervised learning of disentangled representations, in: Chaudhuri, K., Salakhutdinov, R. (Eds.), Proceedings of the 36th International Conference on Machine Learning, PMLR. pp. 4114--4124.  URL: https://proceedings.mlr.press/v97/locatello19a.html.
  * López-Linares et al. [2018] López-Linares, K., Aranjuelo, N., Kabongo, L., Maclair, G., Lete, N., Ceresa, M., García-Familiar, A., Macía, I., González Ballester, M.A., 2018.  Fully automatic detection and segmentation of abdominal aortic thrombus in post‐operative cta images using deep convolutional neural networks.  Medical Image Analysis 46, 202–214.  doi:10.1016/j.media.2018.03.010.
  * Lourenço et al. [2021] Lourenço, A., Kerfoot, E., Grigorescu, I., Scannell, C.M., Varela, M., Correia, T.M., 2021\.  Automatic myocardial disease prediction from delayed-enhancement cardiac mri and clinical information, in: Puyol Anton, E., Pop, M., Sermesant, M., Campello, V., Lalande, A., Lekadir, K., Suinesiaputra, A., Camara, O., Young, A. (Eds.), Statistical Atlases and Computational Models of the Heart. M&Ms and EMIDEC Challenges, Springer International Publishing, Cham. pp. 334--341.
  * Lundberg and Lee [2017] Lundberg, S.M., Lee, S.I., 2017\.  A unified approach to interpreting model predictions, in: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 4768--4777.
  * Masis [2021] Masis, S., 2021.  Interpretable Machine Learning with Python.  Packt Publishing.  URL: https://www.packtpub.com/product/interpretable-machine-learning-with-python/9781800203907.
  * McCrindle et al. [2021] McCrindle, B., Zukotynski, K., Doyle, T.E., Noseworthy, M.D., 2021\.  A radiology-focused review of predictive uncertainty for ai interpretability in computer-assisted segmentation.  Radiology: Artificial Intelligence 3, e210031.  doi:10.1148/ryai.2021210031.
  * Molnar [2022] Molnar, C., 2022.  Interpretable Machine Learning.  2 ed.  URL: christophm.github.io/interpretable-ml-book/.
  * Neisius et al. [2019] Neisius, U., El-Rewaidy, H., Nakamori, S., Rodriguez, J., Manning, W.J., Nezafat, R., 2019\.  Radiomic analysis of myocardial native t1 imaging discriminates between hypertensive heart disease and hypertrophic cardiomyopathy.  JACC: Cardiovascular Imaging 12, 1946--1954.  URL: https://www.sciencedirect.com/science/article/pii/S1936878X18311008, doi:https://doi.org/10.1016/j.jcmg.2018.11.024.
  * Painchaud et al. [2022] Painchaud, N., Duchateau, N., Bernard, O., Jodoin, P.M., 2022\.  Echocardiography segmentation with enforced temporal consistency.  IEEE Transactions on Medical Imaging , 1--1doi:10.1109/TMI.2022.3173669.
  * Pati and Lerch [2021] Pati, A., Lerch, A., 2021.  Attribute-based Regularization of Latent Spaces for Variational Auto-Encoders.  Neural Computing and Applications 33, 4429–--4444.  doi:10.1007/s00521-020-05270-2.
  * Pitale et al. [2021] Pitale, R., Kale, H., Kshirsagar, S., Rajput, H., 2021\.  A schematic review on applications of deep learning and computer vision, in: 2021 Asian Conference on Innovation in Technology (ASIANCON), pp. 1--6.  doi:10.1109/ASIANCON51346.2021.9544941.
  * Puyol-Antón et al. [2020] Puyol-Antón, E., Chen, C., Clough, J.R., Ruijsink, B., Sidhu, B.S., Gould, J., Porter, B., Elliott, M., Mehta, V., Rueckert, D., Rinaldi, C.A., King, A.P., 2020\.  Interpretable deep models for cardiac resynchronisation therapy response prediction , 284--293doi:10.1007/978-3-030-59710-8_28.
  * Reyes et al. [2020] Reyes, M., Meier, R., Pereira, S., Silva, C.A., Dahlweid, F.M., von Tengg-Kobligk, H., Summers, R.M., Wiest, R., 2020\.  On the Interpretability of Artificial Intelligence in Radiology: Challenges and Opportunities.  Radiol Artif Intell 2, e190043.  doi:10.1148/ryai.2020190043.
  * Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C., 2016.  "Why Should I Trust You?": Explaining the predictions of any classifier, in: Proceedings of the Demonstrations Session, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, The Association for Computational Linguistics. pp. 97--101.  doi:10.18653/v1/n16-3020.
  * Ridgeway and Mozer [2018] Ridgeway, K., Mozer, M.C., 2018\.  Learning deep disentangled embeddings with the f-statistic loss, in: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 185--194.
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., Brox, T., 2015.  U-Net: Convolutional networks for biomedical image segmentation, in: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (Eds.), Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015, Springer International Publishing, Cham. pp. 234--241.  doi:10.1007/978-3-319-24574-4_28.
  * Rubenstein et al. [2018] Rubenstein, P.K., Schölkopf, B., Tolstikhin, I.O., 2018.  Learning disentangled representations with wasserstein auto-encoders, in: ICLR.
  * Rudin [2019] Rudin, C., 2019.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature Machine Intelligence 1, 206--215.  URL: https://doi.org/10.1038/s42256-019-0048-x, doi:10.1038/s42256-019-0048-x.
  * Salahuddin et al. [2022] Salahuddin, Z., Woodruff, H.C., Chatterjee, A., Lambin, P., 2022\.  Transparency of deep neural networks for medical image analysis: A review of interpretability methods.  Computers in Biology and Medicine 140, 105111.  URL: https://www.sciencedirect.com/science/article/pii/S0010482521009057, doi:10.1016/j.compbiomed.2021.105111.
  * Samb et al. [2012] Samb, M.L., Camara, F., Ndiaye, S., Slimani, Y., Esseghir, M.A., Anta, C., 2012\.  A novel rfe-svm-based feature selection approach for classification.  International Journal of Advanced Science and Technology 43.  doi:10.1155/2014/795624.
  * Sanchez-Martinez et al. [2022] Sanchez-Martinez, S., Camara, O., Piella, G., Cikes, M., González Ballester, M.A., Miron, M., V.A., Gomez, E., Fraser, A., Bijnens, B., 2022.  Machine learning for clinical decision-making: Challenges and opportunities in Cardiovascular imaging.  Frontiers in Cardiovascular Medicine 8\.  doi:10.3389/fcvm.2021.765693.
  * Selvaraju et al. [2017] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D., 2017\.  Grad-cam: Visual explanations from deep networks via gradient-based localization, in: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618--626.  doi:10.1109/ICCV.2017.74.
  * Shakeri et al. [2016] Shakeri, M., Lombaert, H., Tripathi, S., Kadoury, S., 2016\.  Deep spectral-based shape features for alzheimer’s disease classification, in: Reuter, M., Wachinger, C., Lombaert, H. (Eds.), Spectral and Shape Analysis in Medical Imaging, Springer International Publishing, Cham. pp. 15--24.  doi:http://dx.doi.org/10.1007/978-3-319-51237-2_2.
  * Sharma et al. [2021] Sharma, R., Eick, C.F., Tsekos, N.V., 2021.  Sm2n2: A stacked architecture for multimodal data and its application to myocardial infarction detection, in: Puyol Anton, E., Pop, M., Sermesant, M., Campello, V., Lalande, A., Lekadir, K., Suinesiaputra, A., Camara, O., Young, A. (Eds.), Statistical Atlases and Computational Models of the Heart. M&Ms and EMIDEC Challenges, Springer International Publishing, Cham. pp. 342--350.
  * Shen et al. [2017] Shen, D., Wu, G., Suk, H.I., 2017.  Deep Learning in Medical Image Analysis.  Annu Rev Biomed Eng 19, 221--248.  doi:10.1146/annurev-bioeng-071516-0444422.
  * Shi et al. [2021] Shi, J., Chen, Z., Couturier, R., 2021.  Classification of pathological cases of myocardial infarction using convolutional neural network and random forest, in: Puyol Anton, E., Pop, M., Sermesant, M., Campello, V., Lalande, A., Lekadir, K., Suinesiaputra, A., Camara, O., Young, A. (Eds.), Statistical Atlases and Computational Models of the Heart. M&Ms and EMIDEC Challenges, Springer International Publishing, Cham. pp. 406--413.
  * Simonyan et al. [2014] Simonyan, K., Vedaldi, A., Zisserman, A., 2014.  Deep inside convolutional networks: Visualising image classification models and saliency maps, in: Bengio, Y., LeCun, Y. (Eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings.  URL: http://arxiv.org/abs/1312.6034.
  * Singh et al. [2020] Singh, A., Sengupta, S., Lakshminarayanan, V., 2020.  Explainable deep learning models in medical image analysis.  Journal of Imaging 6.  URL: https://www.mdpi.com/2313-433X/6/6/52, doi:10.3390/jimaging6060052.
  * Wu et al. [2019] Wu, S., Roberts, K., Datta, S., Du, J., Ji, Z., Si, Y., Soni, S., Wang, Q., Wei, Q., Xiang, Y., Zhao, B., Xu, H., 2019.  Deep learning in clinical natural language processing: a methodical review.  Journal of the American Medical Informatics Association 27, 457--470.  doi:10.1093/jamia/ocz200.
  * Xiao et al. [2020] Xiao, G., Rong, W.C., Hu, Y.C., Shi, Z.Q., Yang, Y., Ren, J.L., Cui, G.B., 2020\.  MRI Radiomics Analysis for Predicting the Pathologic Classification and TNM Staging of Thymic Epithelial Tumors: A Pilot Study.  AJR Am J Roentgenol 214, 328--340.  doi:10.2214/AJR.19.21696.
  * Yang [2018] Yang, X., 2018.  Identification of risk genes associated with myocardial infarction based on the recursive feature elimination algorithm and support vector machine classifier.  Mol Med Rep 17, 1555--1560.  doi:10.3892/mmr.2017.8044.
  * Zhang et al. [2017] Zhang, X., Xu, X., Tian, Q., Li, B., Wu, Y., Yang, Z., Liang, Z., Liu, Y., Cui, G., Lu, H., 2017.  Radiomics assessment of bladder cancer grade using texture features from diffusion-weighted imaging.  J Magn Reson Imaging 46, 1281--1288.  doi:10.1002/jmri.25669.
  * Zhu et al. [2017] Zhu, J.Y., Park, T., Isola, P., Efros, A.A., 2017\.  Unpaired image-to-image translation using cycle-consistent adversarial networks, in: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242--2251.  doi:10.1109/ICCV.2017.244.