# From explainable to interpretable deep learning for natural language
processing in healthcare: how far from reality?

Guangming Huang<EMAIL_ADDRESS>Yunfei Long<EMAIL_ADDRESS>Yingya
Li<EMAIL_ADDRESS>Giorgos Papanastasiou
<EMAIL_ADDRESS>
###### Abstract

Deep learning (DL) has substantially enhanced healthcare research by
addressing various natural language processing (NLP) tasks. Yet, the
increasing complexity of DL-based NLP methods necessitates transparent model
interpretability, or at least explainability, for reliable decision-making.
This work presents a thorough scoping review on explainable and interpretable
DL in healthcare NLP. The term “XIAI” (eXplainable and Interpretable
Artificial Intelligence) was introduced to distinguish XAI from IAI. Methods
were further categorized based on their functionality (model-, input-, output-
based) and scope (local, global). Our analysis shows that attention mechanisms
were the most dominant emerging IAI. Moreover, IAI is increasingly used
against XAI. The major challenges identified are that most XIAI do not explore
“global” modeling processes, the lack of best practices, and the unmet need
for systematic evaluation and benchmarks. Important opportunities were raised
such as using “attention” to enhance multi-modal XIAI for personalized
medicine and combine DL with causal reasoning. Our discussion encourages the
integration of XIAI in LLMs and domain-specific smaller models. Our review can
stimulate further research and benchmarks toward improving inherent IAI and
engaging complex NLP in healthcare.

###### keywords:

explainable , interpretable , deep learning , NLP , healthcare

††journal: review

[label1]organization=School of Computer Science and Electronic Engineering,
University of Essex,city=Colchester, postcode=CO4 3SQ, country=United Kingdom
[label2]organization=Harvard Medical School and Boston Children’s
Hospital,city=Boston, postcode=02115, country=United States
[label3]organization=Archimedes Unit, Athena Research Centre,city=Athens,
postcode=15125, country=Greece

##  1 Introduction

Deep learning (DL)-based natural language processing (NLP) enhanced healthcare
research [1, 2, 3], improving the analysis of textual data such as electronic
health records (EHRs) [4], clinical notes [5, 6, 7] and medical reports [8, 9,
10]. DL advancements have considerably improved medical diagnosis [11, 12, 9],
patient monitoring [13] and drug discovery [14, 15] through addressing NLP
tasks as text classification [5, 16, 17], summarization [18, 8, 19] and named
entity recognition [20, 21, 22].

Since the first implementation of recurrent neural networks (RNN) [23], long
short-term memory (LSTM) [24] and convolution neural networks (CNN) [25] up to
the most recent self-attention mechanisms and Transformers [26], the evolution
of DL models in NLP is associated with substantial performance improvements
against traditional machine learning methods. However, these performance gains
are associated with gradually increasing complexities in model architectures.
In healthcare settings, trustworthy and transparent decision-making is a top
priority. Hence there is an increasing requirement for interpretable or at
least explainable models [27]. Addressing explainability/ interpretability is
complex due to difficulties in mapping nuanced linguistic embeddings and
semantics inherent within textual data [28, 29] to model decision-making
processes. However, recent work has been promising in the field of explainable
[30, 31] or interpretable [32, 33] DL for NLP in healthcare. Furthermore, the
recent developments of large language models (LLMs) further emphasize the need
to assess which explainable/ interpretable methods can potentially be useful
in healthcare as data and models become increasingly complex over time.

In our work, the evolution of eXplainable and Interpretable Artificial
Intelligence (summarized with the new term “XIAI”) was reviewed [34, 35, 36,
37]. Since there is no systematic definition of IAI and XAI [34, 35], there is
an ongoing inconsistency regarding how these terms are used in DL [36]. To
define XIAI in this review, the statistical definition by Rudin [37] was
followed: IAI focuses on designing models that are inherently interpretable;
XAI aims to provide post hoc model explanations. Moreover, XIAI were grouped
into either local or global: local XIAI provides insights based on specific
inputs, whereas global XIAI offers a broad understanding based on the model’s
overall predictive process [27]. Further, XIAI method designs were evaluated,
by organizing them into model-based, input-based, and output-based approaches.
Our analysis aims to improve clarity and understanding regarding how DL and
XIAI methods can be most efficiently combined in healthcare NLP, through
unraveling key architectural designs. By quantitatively evaluating IAI and XAI
across NLP and medical tasks, current and future opportunities, challenges,
advantages, and drawbacks were identified. In line with the most recent LLM
developments, our review evaluates whether attention mechanisms can reinforce
the implementation of IAI against XAI. By carefully investigating XIAI DL in
healthcare NLP, our review aims to provide insights of scientific and clinical
impact that can potentially be important for XIAI democratization in
healthcare NLP.

##  2 Methods

Figure 1:  (a) PRISMA flowchart. The flowchart presents the inclusion and
exclusion of papers at each review stage. (b) Publication records of (b) XIAI,
(c) main DL model category, (d) main XAI and (e) main IAI methods, per year.

###  2.1 Literature review strategy

A comprehensive scoping review of explainable and interpretable DL-based NLP
techniques applied to healthcare was performed, by considering publications
between January 1, 2018 and December 31, 2022. The following databases were
included: Scopus, Web of Science, PubMed, and the Association for Computing
Machinery (ACM), following the Preferred Reporting Items for Systematic
Reviews and Meta-Analyses (PRISMA) guidelines [38]. To address real-world data
needs and review XIAI methods that can potentially be democratized, our review
concentrated on electronic health record (EHR) and electronic medical record
(EMR) NLP data. The emergence of Transformers as an efficient approach to
learning long-range interactions via attention modules in 2017 [26] marked the
beginning of a new era in NLP, with Transformer variants becoming the most
prevalent and promising techniques [39]. To focus our review on these
significant developments, our observation window was selected to start in
2018. Figure1(a) illustrates the PRISMA flow.

Initial Filtering: To broaden our search, publications were initially
extracted by using the following keywords in the title, abstract, and
manuscript keywords: (“natural language processing” OR “NLP” ) AND (
“healthcare” OR “medic” OR “clinic” OR “EHR” OR “EMR”). This led to a
preliminary list of 4,147 papers. Subsequently, to narrow our search, papers
irrelevant to XIAI were removed, resulting in 234 papers. In the subsequent
step, removing duplicate records across databases resulted in 146 unique
papers for title and abstract screening.

Title and Abstract Screening: All authors were engaged in reviewing the titles
and abstracts across all 146 papers. Excluding articles that were non-related
to the field of study led to 110 papers eligible for full-paper review.

Full-Text Screening: In the course of full-paper review, 68 papers were
excluded (25 papers were not based on DL, 5 were irrelevant to XIAI, 4 were
not medical, 21 were review papers, and 13 were irrelevant to either DL, NLP,
or healthcare). In total, 42 journal and conference papers were included in
our review analysis.

###  2.2 Review aspects

During evaluation of full paper texts, our review focused on the following
aspects: 1) publication date; 2) DL model (see Results for DL model
categorization); 3) IAI versus XAI; 4) scope of XIAI method (i.e.
local/global); 5) XIAI method (model-, input-, and ouput-based XIAI); 6)
medical task; 7) NLP task; 8) data source (public or private data); 9) whether
the source code was available and 10) if XIAI evaluation was performed.

Furthermore, challenges, opportunities, advantages, and drawbacks were
identified by analyzing XIAI methods in terms of the following three aspects:
XAI and IAI methods and how these were combined with a) the DL model used, b)
NLP task analyzed, and c) medical task addressed.

##  3 Results

###  3.1 Research trends

Published work on XAI and IAI NLP in healthcare was reviewed and a rapid
increase in the number of studies was observed from 2021 to 2022, compared to
the first years of our observation window (Figure 1(b)). This trend
corresponds to a high rate of Transformer-based methods during the same time
(Figure 1(c)).

###  3.2 IAI against XAI

Subsequently, all methods across papers were examined to assess whether they
focused on IAI or XAI. More studies involved IAI methods ($n=29$) than XAI
methods ($n=15$), with this trend being evident in the period 2020-2022.
Figure 1(d) illustrates the publication record of XAI methods used per year.
Among XAI methods, t-SNE ($n=4$) was the most widely used, followed by LIME
($n=3$), SHAP ($n=3$) and other methods ($n=5$), i.e., maximum aggregator
(MaXi) [40], SKET eXplained (SKET X) [41], syntax tree-guided semantic
explanation (STEP) [42], evidence-based [43] and sentiment intensity score
[44]. As illustrated in Figure 1(e), most IAI methods involved attention
mechanisms ($n=11$), followed by feature importance ($n=8$), knowledge
base/graph ($n=5$) and other methods ($n=5$), including causal graphs [45],
logistic regression-based parametric predictions [46], opinion aggregator
[47], case-based reasoning [48] and interactive classification [49].

XIAI method scopes were further grouped into local and global [27]. The
majority of papers employed local methods ($n=37$), with only $n=5$ studies
involving global XIAI (see Table 1).

###  3.3 Three XIAI paradigms

Further, three distinct XIAI categories were introduced: model-, input- and
output-based methods. These definitions were conceptualized based on whether a
XIAI method relies on internal/external modules (model) to perform XIAI,
measures how input features affect model decisions, or explains/interprets
model behavior through analyzing prediction outcomes, respectively. Table 1
presents a comprehensive overview of sixteen different XIAI approaches
identified in our review and their distinct categories (model-, input-, and
output-based methods). There were eighteen, thirteen and thirteen model-,
input- and output-based methods identified, respectively.

According to Table 1 , the most popular model-based XIAI was t-SNE ($n=4$),
followed by LIME ($n=3$), SHAP ($n=3$) and other methods that have been
sparsely used. Among input-based XIAI, studies either used feature importance-
based ($n=8$) or knowledge graph-based ($n=5$) methods. Regarding output-based
methods, the attention mechanism was the most frequently used approach (Table
1). While benefiting from their performance advancements, attention mechanisms
may therefore be an important IAI approach.

XIAI category | XIAI methods | Explainable/Interpretable | Local/Global | Articles
---|---|---|---|---
Model-based | Causal graph | Interpretable | Global |  [45]
Logistic regression-based parametric predictor | Interpretable | Global |  [46]
Opinion aggregator | Interpretable | Global |  [47]
Case-based reasoning | Interpretable | Global |  [48]
Interactive classification | Interpretable | Global |  [49]
LIME | Explainable | Local |  [50, 30, 51]
MAXi | Explainable | Local |  [40]
SHAP | Explainable | Local |  [31, 52, 53]
SKET X | Explainable | Local |  [41]
STEP | Explainable | Local |  [42]
t-SNE | Explainable | Local |  [54, 55, 56, 57]
Input-based | Feature importance | Interpretable | Local |  [58, 59, 60, 61, 62, 55, 63, 64]
Knowledge base/graph | Interpretable | Local |  [65, 66, 67, 68, 32]
Output-based | Attention | Interpretable | Local |  [33, 32, 69, 70, 71, 72, 73, 74, 75, 76, 77]
Evidence-based | Explainable | Local |  [43]
Sentiment intensity score | Explainable | Local |  [44]

Table 1: Explainable and interpretable artificial intelligence (XIAI) methods,
their category, and scope

####  3.3.1 Model-based XIAI

Model-based XIAI focuses on describing how DL functions through the use of
internal or external modules. The following section presents the most
promising XAI and IAI methods identified.

LIME. Ribeiro et al. [78] introduced the local interpretable model-agnostic
explanation (LIME) method. To produce explanations, LIME perturbs an instance,
generates neighboring data, and learns linear models within that neighborhood.
For instance Zhang et al. [50] analyzed physician ordering for magnetic
resonance imaging (MRI) scans and corresponding radiology reports (outcomes),
by developing an RNN model integrated with a LIME module. The authors showed
that it is possible to justify whether a patient should undergo MRI or not and
if any brain abnormalities are likely to be detected, based on certain
keywords from the physician order text entries.

SHAP. Shapley Additive exPlanations (SHAP) is an algorithm used to obtain
local, post-hoc explanations [79]. The SHAP method is a model agnostic
approach, that allows to elucidate how individual input features contribute to
model decisions. The method is based on deriving Shapley values from
cooperative game theory [80],which aims to quantify the contribution of each
input, both individually as well as collectively in combination with all other
inputs. Naseem et al. [53]introduced a Transformer-based encoder with a SHAP
module for suicide risk identification. The SHAP module collected the
contribution of important words that were most relevant to mental health and
suicide.

t-SNE. t-SNE can effectively capture local interactions within data, although
it is less efficient in identifying long-range interactions [81]. Using t-SNE,
it is possible to visualize local clusters consisted of specific word
embeddings, towards enhancing model explainability [54, 55, 56, 57].

Although less explored, model-based techniques may be promising for IAI (Table
1). Lu et al. [47] employed an opinion aggregator in combination with BERT,
which involved k-means clustering to disentangle responses for COVID-19
questions. Another study developed a Bayesian network-derived causal graph to
identify causal relationships between NLP variables extracted from mammography
reports [45]. The authors fed the causal graph into a TabNet model and showed
that they could improve model interpretability. Causal modeling can be a
promising technique to perform IAI.

####  3.3.2 Input-based XIAI

Input-based XIAI methods focus on understanding how specific inputs influence
network decisions. All input-based methods identified were either under the
feature importance or knowledge base/graphs categories.

Feature Importance. Lindsay et al. [58] extracted semantic, syntactic and
paralinguistic features using NLP from spontaneous picture descriptions to
identify cognitive impairments for Alzheimer’s disease (AD) detection. The
authors quantified statistically significant (“generalizable” ) NLP features,
across two different languages. Subsequently, they performed binary (AD vs
healthy) patient classification using DL and showed more accurate results when
the generalizable features were used, against when all features were
considered. Holderness et al. [61] integrated clinically relevant multiword
expressions during preprocessing which improved the accuracy and
interpretability of their DL models (multilayer perceptrons and radial basis
function DL) in identifying psychosis patients at risk of hospital
readmission. Mellado et al. [62] engineered clinically interpretable NLP-based
features from topic extraction and clinical sentiment analysis to predict
early readmission risk in psychiatry patients. In a different setting, Xue and
Chuah [63] designed a knowledge extraction framework from heterogeneous
medical data sources, capable to generate an aggregated dataset to
characterize diseases. Based on this dataset, they introduced an end-to-end
DL-based medical multi-disease diagnosis system.

Knowledge graphs. Knowledge graphs (KGs) are recruited to structure knowledge
representations from either internal or external sources [82, 83, 84]. Teng et
al. [32] introduced an end-to-end KG-based framework for diagnosis code
predictions. Gu et al. [68] proposed a novel method to solve complex question-
answering from a medical dataset by combining KG information. The authors used
Wikipedia as an external text source to extract documents related to questions
and then mined triples (subject, predicate, object) from the documents to
construct a KG. They subsequently extracted evidence graphs from the KG and
embedded them into an Attention-based Graph Neural Network to perform complex
question answering.

####  3.3.3 Output-based XIAI

Output-based XIAI methods focus on explaining/interpreting DL models by
unraveling how internal computations within DL converge to output decisions.

Attention Mechanism. The attention mechanism was initially designed to learn
long-range interactions in NLP and improve machine translation [85]. The
attention mechanism allows to search for a set of positions in a source
sentence and encourages the model to predict a target word based on the
context vectors associated with these source positions and all the previously
generated target words [85]. Following attention, the self-attention mechanism
was designed to further enhance modeling of long-range interactions [86].
Within a self-attention layer, each position (known as query, key, and value)
can attend to all positions in the output of the previous layer. Attention and
self-attention mechanisms are the building blocks of Transformers. Transformer
models have been recently applied in biomedical NLP [72, 73] and image
analysis [87]. Next to performance gains, attention mechanisms and
Transformers have demonstrated their applicability in enhancing
interpretability [88]. Attention mechanisms were the most widely used XIAI
techniques in our systematic review.

Dong et al. [33] designed an Hierarchical Label-wise Attention Network to
perform medical coding. The proposed model was able to provide interpretations
in the form of attention weights quantified from both words and sentences.
Duarte et al. [69] engineered attention mechanisms inside a neural network to
derive death certifications through the analysis of diagnosis codes. Their
method produced accurate and interpretable results and can potentially be
transferrable to clinical settings. Zhang et al. [70] incorporated a series of
self-attention modules into an RNN architecture. They showed accurate and
interpretable predictions of patients at risk for all-cause hospitalizations
through the analysis of EHR data. Chen et al. [72] trained BERT-based models
via federated learning for ICD-10 coding. To interpret the model outputs, the
authors added a label attention mechanism to the BERT model, which was able to
display federated learning outcomes.

Figure 2: Publication record showing combinations between (a) deep learning
model and XAI method, (b) deep learning model and IAI method, (c) NLP task and
XAI method and (d) NLP task and IAI method

###  3.4 Deep learning

Table LABEL:tab:2 presents all DL models identified across papers. DL models
were classified as: 1) CNN/RNN/GRU/LSTM-based models; 2) Transformer/BERT-
based models, featuring pre-trained language models; 3) Multilayer Perceptrons
(MLPs); 4) Graph neural networks (GNNs) and; 5) other which includes models
used once like unsupervised learning [46] and stacked denoising autoencoder
[55].

The publication trends across DL models were examined (Figure 1(c)). Figures
2(a) and (b) illustrate DL model combinations with XAI and IAI, respectively.
CNN/RNN/GRU/LSTM-based models were the most frequently used ($n=17$), followed
by Transformer/ BERT-based ($n=14$), MLP ($n=4$), GNN-based ($n=2$) and other
models ($n=7$). Considering the period 2021-2022, Transformer/BERT-based
models were the most dominant backbone architecture used. Although
Transformer-based techniques were introduced in 2017 [26], our analysis shows
that their adoption in the healthcare domain was not immediately adopted.

###  3.5 XIAI and NLP

As shown in Table LABEL:tab:2, seven NLP tasks were identified: 1) text
classification (which includes sentiment analysis [62]), 2) information
extraction, including knowledge extraction [41, 63] and relation extraction
[67], 3) named entity recognition, 4) question answering, 5) document
clustering, 6) information retrieval and 7) topic extraction. Text
classification was the main NLP task ($n=35$), followed by information
extraction ($n=5$), named entity recognition ($n=4$) and other 4 tasks
($n=5$).

Figure 2(c-d) presents NLP tasks against XAI and IAI, respectively. Of note,
IAI methods have been employed across all NLP tasks defined. In contrast, XAI
methods have been exclusively used for text classification and information
extraction.

###  3.6 XIAI and medical tasks

Medical tasks across all studies were extracted and categorized into sixteen
categories (see Table LABEL:tab:2). Figure 3 illustrates the primary medical
tasks, including medical diagnosis ($n=10$), medical coding ($n=6$), medical
decision making ($n=6$), mental health diagnosis ($n=5$), disease
classification ($n=3$) and other ($n=12$).

Figure 3 (a-b) illustrates all medical tasks addressed using XAI and IAI
methods, respectively. IAI methods were broadly applied to a more diversified
set of medical tasks, against XAI methods which were mainly utilized in mental
health diagnosis, medical diagnosis and decision making.

###  3.7 Evaluation and reproducibility

In our review, only 3 out of all 42 articles involved dedicated evaluation
processes and metrics of XIAI methods. These studies either involved
qualitative evaluations by developers [33] and end-users (i.e., clinicians)
[32], or developed a quantitative explainability index [42].

By assessing reproducibility, only 5 papers out of 42 provided code access
[46, 33, 59, 65, 57].

###  3.8 Datasets

Finally, methods were categorized based on whether they were developed using
public datasets (open-source and accessible), or private datasets (otherwise).
According to Table LABEL:tab:2, the majority of papers(n = 32 out of 42
papers) used private data and only 10 papers analyzed public datasets. EHR
from hospitals was the most prevalent type of private data analyzed (Table
LABEL:tab:2). The most widely used public dataset was MIMIC ($n=5$), which is
a large clinical database consisting of multi-modal (text and imaging) data
(Table LABEL:tab:2).

Figure 3: Publication record showing combinations between medical task and (a) XAI and (b) IAI method. Table 2: All the articles grouped based on the XIAI method along with the associated medical task, NLP task and the dataset used for. DL Model | Medical task category | Medical task | NLP task | Dataset | Article
---|---|---|---|---|---
Transformer/ BERT-based |  Medical entity disambiguation |  Biomedical named entity disambiguation | NER | ClinWikiNED | [59]
Medical decision making |  Chinese medicine instruction parsing |  NER; Information extraction | Private dataset | [67]
Medical bias mitigation |  Reducing genderization | Text classification | MIMIC-III | [56]
Medical decision making |  Drug indication classification | Text classification |  curated drug indication corpus | [57]
Medical research |  Accelerate COVID-19 research | QA | Private dataset | [47]
Management of health services |  Automatically scoring request for proposals (RFP) |  Information extraction; Text classification | Private dataset | [52]
Medical note generation |  Radiology report generation assistance | NER | MIMIC-CXR | [48]
Medical coding |  Medical coding | Text classification | Private dataset | [72]
Medical diagnosis |  Long-term COVID eﬀects | Text classification | Private dataset | [73]
Medical fake news detection |  Fake news detector to covid-19 | Text classification | Private dataset | [43]
Mental health diagnosis |  Suicide risk identification | Text classification | Private dataset | [53]
Medical diagnosis |  Chronic cough | Text classification | Private dataset | [76]
Medical diagnosis |  Alzheimer’s disease | Text classification | Private dataset | [77]
Mental health diagnosis |  Depression symptoms detection (DSD) | Text classification | Private dataset | [42]
CNN/ RNN/ GRU/ LSTM-based |  Medical coding |  Medical coding | Text classification | MIMIC-III | [32]
Medical diagnosis |  Ischemic stroke | Text classification | Private dataset | [60]
Disease classification |  Predict adherence to ACR guidelines | Text classification | Private dataset | [50]
Medical coding |  Causes of death | Text classification | Private dataset | [69]
Medical diagnosis |  Diagnosis | Text classification | Private dataset | [70]
Medical decision making |  Entrustment assessment | Text classification | Private dataset | [64]
Medical diagnosis |  Cancer pathology | Text classification | Private dataset | [40]
Survival detection |  Discrete-time survival estimates | Text classification | Private dataset | [31]
Medical coding |  ICD multi-label classifcation | Text classification |  MIMIC-III; Spanish Osa Corpus (Private dataset) | [71]
Management of health services |  Automatically scoring request for proposals (RFP) |  Information extraction; Text classification | Private dataset | [52]
Medical diagnosis |  Disease diagnosis |  Information extraction; Text classification | Private dataset | [63]
Monitor public sentiments during pandemics |  COVID-19 pandemic |  Information extraction; Text classification | Private dataset | [44]
Telehealth services |  Telehealth services | Text classification | Private dataset | [30]
Mental health diagnosis |  Internet-delivered psychological treatment (IDPT) | Text classification | Private dataset | [74]
Mental health diagnosis |  Depression | Text classification | Amazon Mechanical Turk | [75]
|  Mental health diagnosis |  Depression | Text classification | Private dataset | [51]
Disease classification |  Diabetes-related biomedical literature exploration |  Document clustering; Text classification | Private dataset | [49]
GNN-based |  Medical diagnosis |  Breast cancer | Text classification | Private dataset | [45]
Medical question-answering |  Medical QA | QA | Head-QA | [68]
MLP |  Medical diagnosis |  Language impairment in Alzheimer’s disease | Text classification | Private dataset | [58]
Readmission prediction |  Psychiatric readmission prediction | Text classification | Private dataset | [61]
Readmission prediction |  Psychiatric readmission risk prediction |  Text classification; Topic extraction | Private dataset | [62]
Medical diagnosis |  Alzheimer’s disease | Text classification | Private dataset | [77]
Other |  Medical coding |  Patient medical conditions |  NER; Text classification | Private dataset | [46]
Medical decision making |  Clustering of patients’ visits | Text classification | Private dataset | [54]
Medical coding |  Medical coding | Text classification |  MIMIC-III-50; MIMIC-III-shielding; MIMIC-III | [33]
Medical decision making |  Esophageal achalasia | Information retrieval | Private dataset | [65]
Disease classification |  Cardiovascular diseases | Text classification | OHSUMED | [66]
Medical decision making |  Patient mortality prediction; Primary diagnostic category prediction; Primary procedural category prediction; Gender prediction | Text classification | Private dataset | [55]
Medical diagnosis |  Pathology | Information extraction | Private dataset | [41]

##  4 Discussion

###  4.1 Opportunities and challenges

All papers were studied from the perspective of three professionals with
extensive experience in DL and NLP in healthcare and a PhD researcher (GH, in
the field of DL for NLP). Challenges and opportunities were identified from
the interdisciplinary perspective of computer scientists and end-users of XIAI
(DL). Based on our literature review, there is no previous survey focusing on
these topics. Considering their early development phase, more extensive
studies are needed in the future to develop XIAI methods that will allow
”global” (end-to-end) interpretations and go beyond visualization maps, and
design evaluation metrics that can be useful for both modelers and end-users.
The following points can therefore guide future work and systematic reviews
towards strengthening and democratizing XIAI in healthcare NLP.

Challenges (with mitigation where applicable): 1) The majority of publications
focused on local XIAI, with only 5 papers involving a global XIAI approach
[45, 46, 47, 48, 49]. Moving beyond local XIAI (which provides relatively
limited insights based on specific inputs or features) towards ”globally”
enhancing the transparency of the entire modeling process is one of the main
technical challenges. A possible mitigation here is the combination of causal
modeling and DL which can potentially enhance end-to-end XIAI by unraveling
how causal variables (text tokens) interact with low-level representations
(see opportunity 8). 2) Comprehensive evaluation methods. In our review, the
evaluation of XIAI methods was substantially limited (only 3 papers involved
XIAI evaluation). Developing robust evaluation metrics for XIAI is challenging
given the limited availability of ground truths in healthcare data [87] and
the complexity of rapidly evolving models (like LLMs) [89]. 3) XIAI
reproducibility. In our review, the majority of papers analyzed private data
(32 papers) and did not provide source code (37 papers). Hence, it is
particularly challenging to assess the reproducibility of these XIAI NLP
methods. Legal implications around private data pose another critical
challenge in terms of data democratization and XIAI reproducibility assessment
[87]. 4) Based on the previous challenges, there is an unmet yet pivotal need
to develop benchmark data/ XIAI methods/ metrics with associated open-access
codes. 5) A significant variability in the literature was observed in terms of
model-, input- and output-based methodologies used (Table 1) and NLP and
medical tasks addressed (Figures 2 and 3). In addition, each of these method
types presents different advantages and drawbacks (see section 4.2). In
combination with the absence of benchmark studies, it is therefore
significantly challenging to develop generalization frameworks regarding which
XIAI method is most optimal per NLP and medical application. As a possible
mitigation, attention mechanisms can potentially serve as generalization
vehicles, given that they can enhance both performance and interpretability
across applications. 6) However, attention weights do not linearly correlate
with model outcomes[28, 77]. Moreover, attention-based IAI is mainly based on
visualization (of important attention weights) heat maps. These two
limitations currently compromise attention-based IAI. 7) The adoption of
current XIAI in global healthcare and industry systems is challenging, given
the absence of robust XIAI metrics which is an important barrier for their
systematization. 8) XIAI systematization and deployment necessitate further
thorough work. Next to algorithmic opportunities that arise, a possible
practical mitigation to accelerate the systematization of robust and
transparent XIAI developments for NLP in healthcare is to bring “humans into
the DL loop” [90, 91]: domain experts, end-users, policymakers and patients
will be able to contribute to the design, development, evaluation and XIAI
evaluation.

Opportunities: 1) Attention mechanisms were the most diversified XIAI
techniques in terms of DL used (Figure 2a, b) as well as NLP (Figure 2c, d)
and medical tasks (Figure 3) addressed. It is noteworthy that although
Transformers were the dominant DL backbones in the period 2021-2022,
individual attention mechanisms were the most frequently used IAI technique.
Attention and self-attention mechanisms have also recently been used in
combination with CNN models for medical image analysis, as lighter
alternatives of Transformers [87]. This demonstrates their diverse
applicability across data domains and tasks. A major opportunity that
therefore arises is the versatility of attention mechanisms to be combined
with multiple DL structures (e.g., CNN [71], RNN [70], BERT [72], full
Transformers) [77] and to simultaneously model information from variable
multi-modal data (e.g., text, images, genetics, clinical history) [87]. 2)
Based on the previous, developing multi-modal XIAI can support personalized
medicine which benefits from combining patient-level information from multiple
sources [92]. 3) A clear trend towards IAI against XAI was observed across
studies, particularly in the period 2021-2022. To date, end-users are commonly
blinded to training processes and are mainly involved in DL validation. IAI
can enhance DL transparency both during training and validation and can
inspire close interactions between computer scientists and end-users at early
DL stages. 4) Although our review highlights the benefits of using IAI against
XAI, their combination can be useful as an auxiliary assessment to cross-
evaluate IAI and XAI outcomes. For instance, SHAP method that can provide both
global and local explanations[79], has been efficiently combined with a
Transformer encoder for suicide risk prediction [53]. Although this work
focused on the SHAP method to perform XAI, future work could aim to also co-
extract IAI information from specific attention mechanisms within
Transformers. 5) Of note, involving more global (i.e, at least two or all
three: model-, input- and/ or output-based) methods for XIAI across
applications, can reinforce intuitive perspectives regarding how XIAI
interacts with DL models and data. Capturing “all possible or more” XIAI
aspects can improve method interpretability and explainability and potentially
enhance their scientific and clinical impact. 6) Our endeavor is to inspire
and guide benchmark studies to thoroughly examine XIAI in terms of
strengthening NLP applications in healthcare. For example, leveraging
attention mechanisms to better understand how foundation models such as
Transformers (which were dominant as XIAI and DL methods in our review,
respectively) interact with data. 7) Our review also aims to guide further
scientific work in terms of elucidating how attention mechanisms can move
beyond visualization maps, towards interpreting which long- and local-range
interactions directly contribute to DL outcomes. 8) A recent emerging trend is
the field of causal representation learning (CRL). CRL aims to incorporate
causal logic into DL models based on which to identify how high-level causal
variables (specific text tokens) interact with low-level observations
(embeddings) through model interventions and counterfactuals [93]. In that
context, causal abstraction methods have been developed which aim to unravel
how neural representations are aligned with causal variables and their role in
input/ output model behavior [94, 95]. Although initially explored for small
scale models (e.g., BERT) [94, 95], there is work in progress towards
exploring CRL in LLMs [96]. In this work, a BN-derived causal graph has been
fed into a TabNet, showing promising IAI results [45]. Further work in this
field can truly enhance IAI in NLP for healthcare.

###  4.2 The three XIAI paradigms: advantages and drawbacks

Model-based XIAI methods, such as SHAP [79], LIME [78], and t-SNE [81], offer
important advantages by providing both global and local DL interpretability
options. These methods are designed to be transparent and accessible as ready-
to-use entities thus, potentially being able to democratize XIAI tools for a
wide audience [31]. Intuitive visualizations can to some extend elucidate
complex model predictions, enhancing trust and facilitating the communication
of findings. However, they also have drawbacks, as for instance the
instability of methods like LIME, which is prone to minor data variations that
can affect XAI reliability [37, 77]. Moreover, relying on separate modules for
XIAI may be subject to misinterpretations, in case these modules have inherent
biases or limitations. Hence, despite their ease of use, they require
technical expertise to evaluate any biases or implementation
incompatibilities.

Input-based XIAI methods allow for DL model interpretation by leveraging
specific input feature importance [58, 59, 60] and medical KG [65, 66, 67], to
discern how different inputs influence model decisions. These methods are
intuitive and accessible to computer scientists and NLP researchers, due to
their relative ease of being combined with DL architectures. In that context,
the incorporation of ”important features” or medical KGs into AI can enrich
models with domain-specific insights, which in turn allows them to interpret
even complex medical concepts that are common in clinical practice [65].

However, feature importance input-based methods require the incorporation of
methods that are not readily transparent to non-technical medical
professionals. In other words, it may not always be straightforward for end-
users to understand ”how an important feature was derived and/ or evaluated”
[58, 61, 62]. In addition, the effectiveness of KG methods relies on medical
professionals with expertise and capacity to develop comprehensive medical KG,
posing a challenge if such expertise is scarce. Further, integrating medical
KG into DL models presents challenges in terms of ontology construction as
well knowledge fusion and extraction [32], which are labor-intensive.

Output-based XIAI methods, are important to interpret DL outputs and can offer
computational insights through mechanisms like attention [88]. Output-based
methods focus on explaining/interpreting DL models by uncovering how internal
computations within DL converge to output decisions, which can be mainly
useful to computer scientists and developers. Nevertheless, attention
mechanism-based IAI faces debates [97, 98, 99, 100, 101], as high attention
weights don’t necessarily linearly correlate with model predictions [28]. This
can lead to ambiguity whilst emphasizing the need to design further research
on IAI methods and their evaluation.

###  4.3 Inspired by the future

Go Large: Recently, LLMs have attracted significant attention in the field of
AI [102, 103, 104]. The intersection of LLMs and healthcare creates unique
opportunities towards designing future studies, from drug discovery to
personalized diagnosis and treatment [105, 106, 107]. Healthcare data analysis
is one of the high gain-high risk domains for LLMs [89]. One of the main
limitations of LLMs, is that they sometimes ”hallucinate” results [102, 103,
104, 102, 103, 104].This can add considerable barriers to their utilization in
healthcare settings [105, 106, 107]. As discussed in subsection 4.1, CRL can
be an efficient solution towards enhancing IAI, as it aims to connect DL
representations (low-level embeddings) with high-level causal variables that
in turn determine model predictions [93]. Although designing causal logic
inside LLMs is challenging due to their architectural complexity and the fact
that models are already “trained” on a causal-agnostic mode, there have been
recent attempts that aim to develop causal reasoning between prompts and
responses [108, 109].

Go Small: An ongoing discussion in the community is whether LLMs or domain-
specific smaller models can be more robust solutions for healthcare data
[110]. Recently, smaller-parameter domain-specific LMs have outperformed
larger LMs when examined on clinical notes from large public databases (MIMIC)
[111]. This approach has several possible benefits: a) small-parameter LMs can
be trained using in-house computational capabilities, which b) minimizes the
risks associated with transmitting sensitive patient data to cloud-based or
external servers (thus, adhering to privacy regulations). Moreover, c) CRL
solutions can be more optimally designed and validated since models can be
trained on domain-specific data from scratch.

##  5 Conclusion

In conclusion, IAI is increasingly used against XAI, with attention mechanisms
being the most dominant emerging technique to perform interpretations.
Moreover, to efficiently adopt DL in healthcare NLP, it is important to
develop “global” XIAI techniques which can potentially inform about the entire
modeling process. Based on our analysis, additional important opportunities
have been raised, such as to combine and interpret multi-modal information
(text, images, genetic, clinical history) for personalized medicine
approaches, through leveraging attention mechanisms. Another important XIAI
opportunity is the combination of DL with causal modeling, which can further
enhance inherent interpretability. The benefits of model-, input- and output-
based XIAI are demonstrated, but conclude that all these methods necessitate
in-house technical knowledge to infer XIAI. The main challenges are the lack
of best XIAI practices based on the data and problem, as well as the unmet
need for systematic evaluation and high-quality benchmarks. Further, our
discussion encourages the integration of XIAI in the continued development of
LLMs and domain-specific smaller language models.

Despite the challenges raised, the XIAI techniques detailed in this review can
inspire further work and benchmark studies towards enhancing further inherent
interpretability and engaging complex NLP algorithms in healthcare settings.

## References

  * Moro et al. [2022] G. Moro, L. Ragazzi, L. Valgimigli, D. Freddi,  Discriminative marginalized probabilistic neural method for multi-document summarization of medical literature,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 180–189.
  * Otmakhova et al. [2022] J. Otmakhova, K. Verspoor, T. Baldwin, J. H. Lau,  The patient is more dead than alive: exploring the current state of the multi-document summarisation of the biomedical literature,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 5098–5111.
  * Nguyen et al. [2022] T. Nguyen, A. Yates, A. Zirikly, B. Desmet, A. Cohan,  Improving the generalizability of depression detection by leveraging clinical questionnaires,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 8446–8459.
  * Koleck et al. [2019] T. A. Koleck, C. Dreisbach, P. E. Bourne, S. Bakken,  Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review,  Journal of the American Medical Informatics Association 26 (2019) 364–379.
  * Kim et al. [2022] B.-H. Kim, Z. Deng, S. Y. Philip, V. Ganapathi,  Can current explainability help provide references in clinical notes to support humans annotate medical codes?,  in: Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), 2022, pp. 26–34.
  * Zhang et al. [2022] F. Zhang, I. Laish, A. Benjamini, A. Feder,  Section classification in clinical notes with multi-task transformers,  in: Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), 2022, pp. 54–59.
  * Cahyawijaya et al. [2022] S. Cahyawijaya, B. Wilie, H. Lovenia, H. Zhong, M. Zhong, Y. Y. N. Ip, P. Fung,  How long is enough? exploring the optimal intervals of long-range clinical note language modeling,  in: Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), 2022, pp. 160–172.
  * Michalopoulos et al. [2022] G. Michalopoulos, K. Williams, G. Singh, T. Lin,  Medicalsum: A guided clinical abstractive summarization model for generating medical reports from patient-doctor conversations,  in: Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 4741–4749.
  * Wu et al. [2022] X. Wu, S. Yang, Z. Qiu, S. Ge, Y. Yan, X. Wu, Y. Zheng, S. K. Zhou, L. Xiao,  Deltanet: Conditional medical report generation for covid-19 diagnosis,  in: Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 2952–2961.
  * You et al. [2022] J. You, D. Li, M. Okumura, K. Suzuki,  Jpg-jointly learn to align: Automated disease prediction and radiology report generation,  in: Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 5989–6001.
  * Grundmann et al. [2022] P. Grundmann, T. Oberhauser, F. Gers, A. Löser,  Attention networks for augmenting clinical text with support sets for diagnosis prediction,  in: Proceedings of the 29th international conference on computational linguistics, 2022, pp. 4765–4775.
  * Yan and Pei [2022] B. Yan, M. Pei,  Clinical-bert: Vision-language pre-training for radiograph diagnosis and reports generation,  in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2022, pp. 2982–2990.
  * Liu et al. [2019] Z. Liu, H. Lim, N. F. A. Suhaimi, S. C. Tong, S. Ong, A. Ng, S. Lee, M. R. Macdonald, S. Ramasamy, P. Krishnaswamy, et al.,  Fast prototyping a dialogue comprehension system for nurse-patient conversations on symptom monitoring,  in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), 2019, pp. 24–31.
  * Yang et al. [2022] J. Yang, Y. Ding, S. Long, J. Poon, S. C. Han,  Ddi-mug: Multi-aspect graphs for drug-drug interaction extraction,  in: Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), 2022, pp. 127–137.
  * Iinuma et al. [2022] N. Iinuma, M. Miwa, Y. Sasaki,  Improving supervised drug-protein relation extraction with distantly supervised models,  in: Proceedings of the 21st Workshop on Biomedical Language Processing, 2022, pp. 161–170.
  * Kim et al. [2022] J. Kim, A. Sharma, S. Shanbhogue, J. Weiss, P. Ravikumar,  Anemic: A framework for benchmarking icd coding models,  in: Proceedings of the The 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2022, pp. 109–120.
  * Yang et al. [2022] Z. Yang, S. Wang, B. P. S. Rawat, A. Mitra, H. Yu,  Knowledge injected prompt based fine-tuning for multi-label few-shot icd coding,  in: Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 1767–1781.
  * Luo et al. [2022] Z. Luo, Q. Xie, S. Ananiadou,  Readability controllable biomedical document summarization,  in: Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 4667–4680.
  * Sotudeh et al. [2022] S. Sotudeh, N. Goharian, H. Deilamsalehy, F. Dernoncourt,  Curriculum-guided abstractive summarization for mental health online posts,  in: Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), 2022, pp. 148–153.
  * Nguyen et al. [2022] N. D. Nguyen, L. Du, W. Buntine, C. Chen, R. Beare,  Hardness-guided domain adaptation to recognise biomedical named entities under low-resource scenarios,  in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 4063–4071.
  * Kwon et al. [2022] S. Kwon, Z. Yao, H. Jordan, D. Levy, B. Corner, H. Yu,  Medjex: A medical jargon extraction model with wiki’s hyperlink span and contextualized masked language model score,  in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 11733–11751.
  * Rojas et al. [2022] M. Rojas, C. P. Carrino, A. González-Agirre, J. Dunstan, M. Villegas,  Assessing the limits of straightforward models for nested named entity recognition in spanish clinical narratives,  in: Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), 2022, pp. 14–25.
  * Rumelhart et al. [1986] D. E. Rumelhart, G. E. Hinton, R. J. Williams,  Learning representations by back-propagating errors,  nature 323 (1986) 533–536.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter, J. Schmidhuber,  Long short-term memory,  Neural computation 9 (1997) 1735–1780.
  * Kim [2014] Y. Kim,  Convolutional neural networks for sentence classification,  in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1746–1751.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin,  Attention is all you need,  Advances in neural information processing systems 30 (2017).
  * Danilevsky et al. [2020] M. Danilevsky, K. Qian, R. Aharonov, Y. Katsis, B. Kawas, P. Sen,  A survey of the state of explainable ai for natural language processing,  in: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, 2020, pp. 447–459.
  * Sun et al. [2021] X. Sun, D. Yang, X. Li, T. Zhang, Y. Meng, H. Qiu, G. Wang, E. Hovy, J. Li,  Interpreting deep learning models in natural language processing: A review,  arXiv preprint arXiv:2110.10470 (2021).
  * Payrovnaziri et al. [2020] S. N. Payrovnaziri, Z. Chen, P. Rengifo-Moreno, T. Miller, J. Bian, J. H. Chen, X. Liu, Z. He,  Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review,  Journal of the American Medical Informatics Association 27 (2020) 1173–1185.
  * Ozyegen et al. [2022] O. Ozyegen, D. Kabe, M. Cevik,  Word-level text highlighting of medical texts for telehealth services,  Artificial Intelligence in Medicine 127 (2022) 102284.
  * Thorsen-Meyer et al. [2022] H.-C. Thorsen-Meyer, D. Placido, B. S. Kaas-Hansen, A. P. Nielsen, T. Lange, A. B. Nielsen, P. Toft, J. Schierbeck, T. Strøm, P. J. Chmura, et al.,  Discrete-time survival analysis in the critically ill: a deep learning approach using heterogeneous data,  NPJ digital medicine 5 (2022) 142.
  * Teng et al. [2020] F. Teng, W. Yang, L. Chen, L. Huang, Q. Xu,  Explainable prediction of medical codes with knowledge graphs,  Frontiers in bioengineering and biotechnology 8 (2020) 867.
  * Dong et al. [2021] H. Dong, V. Suárez-Paniagua, W. Whiteley, H. Wu,  Explainable automated coding of clinical notes using hierarchical label-wise attention networks and label embedding initialisation,  Journal of biomedical informatics 116 (2021) 103728.
  * Lipton [2018] Z. C. Lipton,  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.,  Queue 16 (2018) 31–57.
  * Doshi-Velez and Kim [2017] F. Doshi-Velez, B. Kim,  Towards a rigorous science of interpretable machine learning,  arXiv preprint arXiv:1702.08608 (2017).
  * Carvalho et al. [2019] D. V. Carvalho, E. M. Pereira, J. S. Cardoso,  Machine learning interpretability: A survey on methods and metrics,  Electronics 8 (2019) 832.
  * Rudin [2019] C. Rudin,  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,  Nature machine intelligence 1 (2019) 206–215.
  * Moher et al. [2009] D. Moher, A. Liberati, J. Tetzlaff, D. G. Altman, t. PRISMA Group*,  Preferred reporting items for systematic reviews and meta-analyses: the prisma statement,  Annals of internal medicine 151 (2009) 264–269.
  * Lin et al. [2022] T. Lin, Y. Wang, X. Liu, X. Qiu,  A survey of transformers,  AI Open 3 (2022) 111–132. URL: https://www.sciencedirect.com/science/article/pii/S2666651022000146. doi:https://doi.org/10.1016/j.aiopen.2022.10.001.
  * Martina et al. [2020] S. Martina, L. Ventura, P. Frasconi,  Classification of cancer pathology reports: a large-scale comparative study,  IEEE Journal of Biomedical and Health Informatics 24 (2020) 3085–3094.
  * Marchesin et al. [2022] S. Marchesin, F. Giachelle, N. Marini, M. Atzori, S. Boytcheva, G. Buttafuoco, F. Ciompi, G. M. Di Nunzio, F. Fraggetta, O. Irrera, et al.,  Empowering digital pathology applications through explainable knowledge extraction tools,  Journal of pathology informatics 13 (2022) 100139.
  * Farruque et al. [2021] N. Farruque, R. Goebel, O. R. Zaïane, S. Sivapalan,  Explainable zero-shot modelling of clinical depression symptoms from text,  in: 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE, 2021, pp. 1472–1477.
  * De Magistris et al. [2022] G. De Magistris, S. Russo, P. Roma, J. T. Starczewski, C. Napoli,  An explainable fake news detector based on named entity recognition and stance classification applied to covid-19,  Information 13 (2022) 137.
  * Boukobza et al. [2022] A. Boukobza, A. Burgun, B. Roudier, R. Tsopra,  Deep neural networks for simultaneously capturing public topics and sentiments during a pandemic: Application on a covid-19 tweet data set,  JMIR Medical Informatics 10 (2022) e34306.
  * Chen et al. [2021] D. Chen, H. Zhao, J. He, Q. Pan, W. Zhao,  An causal xai diagnostic model for breast cancer based on mammography reports,  in: 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE, 2021, pp. 3341–3349.
  * Ferté et al. [2021] T. Ferté, S. Cossin, T. Schaeverbeke, T. Barnetche, V. Jouhet, B. P. Hejblum,  Automatic phenotyping of electronical health record: Phevis algorithm,  Journal of Biomedical Informatics 117 (2021) 103746.
  * Lu et al. [2021] Z.-H. Lu, J. X. Wang, X. Li,  Revealing opinions for covid-19 questions using a context retriever, opinion aggregator, and question-answering model: Model development study,  Journal of medical Internet research 23 (2021) e22860.
  * Amador-Domínguez et al. [2021] E. Amador-Domínguez, E. Serrano, D. Manrique, J. Bajo,  A case-based reasoning model powered by deep learning for radiology report recommendation (2021).
  * Ahne et al. [2022] A. Ahne, G. Fagherazzi, X. Tannier, T. Czernichow, F. Orchard,  Improving diabetes-related biomedical literature exploration in the clinical decision-making process via interactive classification and topic discovery: Methodology development study,  Journal of Medical Internet Research 24 (2022) e27434.
  * Zhang et al. [2019] A. Y. Zhang, S. S. W. Lam, M. E. H. Ong, P. H. Tang, L. L. Chan,  Explainable ai: classification of mri brain scans orders for quality improvement,  in: Proceedings of the 6th IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, 2019, pp. 95–102.
  * Uddin [2022] M. Z. Uddin,  Depression detection in text using long short-term memory-based neural structured learning,  in: 2022 International Conference on Innovations in Science, Engineering and Technology (ICISET), IEEE, 2022, pp. 408–414.
  * Maji et al. [2021] S. Maji, A. Appe, R. Bali, A. G. Chowdhury, V. C. Raghavendra, V. M. Bhandaru,  An interpretable deep learning system for automatically scoring request for proposals,  in: 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), IEEE, 2021, pp. 851–855.
  * Naseem et al. [2022] U. Naseem, M. Khushi, J. Kim, A. G. Dunn,  Hybrid text representation for explainable suicide risk identification on social media,  IEEE transactions on computational social systems (2022).
  * Dobrakowski et al. [2021] A. G. Dobrakowski, A. Mykowiecka, M. Marciniak, W. Jaworski, P. Biecek,  Interpretable segmentation of medical free-text records based on word embeddings,  Journal of Intelligent Information Systems 57 (2021) 447–465.
  * Sushil et al. [2018] M. Sushil, S. Šuster, K. Luyckx, W. Daelemans,  Patient representation learning and interpretable evaluation using clinical notes,  Journal of biomedical informatics 84 (2018) 103–113.
  * Minot et al. [2022] J. R. Minot, N. Cheney, M. Maier, D. C. Elbers, C. M. Danforth, P. S. Dodds,  Interpretable bias mitigation for textual data: Reducing genderization in patient notes while maintaining classification performance,  ACM Transactions on Computing for Healthcare 3 (2022) 1–41.
  * Bhatt et al. [2021] A. Bhatt, R. Roberts, X. Chen, T. Li, S. Connor, Q. Hatim, M. Mikailov, W. Tong, Z. Liu,  Dice: a drug indication classification and encyclopedia for ai-based indication extraction,  Frontiers in Artificial Intelligence 4 (2021) 711467.
  * Lindsay et al. [2021] H. Lindsay, J. Tröger, A. König,  Language impairment in alzheimer’s disease—robust and explainable evidence for ad-related deterioration of spontaneous speech through multilingual machine learning,  Frontiers in aging neuroscience (2021) 228.
  * Garcia-Olano et al. [2021] D. Garcia-Olano, Y. Onoe, I. Baldini, J. Ghosh, B. C. Wallace, K. Varshney,  Biomedical interpretable entity representations,  in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 3547–3561.
  * Ong et al. [2020] C. J. Ong, A. Orfanoudaki, R. Zhang, F. P. M. Caprasse, M. Hutch, L. Ma, D. Fard, O. Balogun, M. I. Miller, M. Minnig, et al.,  Machine learning and natural language processing methods to identify ischemic stroke, acuity and location from radiology reports,  PloS one 15 (2020) e0234908.
  * Holderness et al. [2019] E. Holderness, N. Miller, P. Cawkwell, K. Bolton, M. Meteer, J. Pustejovsky, M.-H. Hall,  Analysis of risk factor domains in psychosis patient health records,  Journal of Biomedical Semantics 10 (2019) 1–10.
  * Mellado et al. [2019] E. Á. Mellado, E. Holderness, N. Miller, F. Dhang, P. Cawkwell, K. Bolton, J. Pustejovsky, M. Hua-Hall,  Assessing the efficacy of clinical sentiment analysis and topic extraction in psychiatric readmission risk prediction,  in: Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), 2019, pp. 81–86.
  * Xue and Chuah [2019] Q. Xue, M. C. Chuah,  Explainable deep learning based medical diagnostic system,  Smart Health 13 (2019) 100068.
  * Gin et al. [2022] B. C. Gin, O. Ten Cate, P. S. O’Sullivan, K. E. Hauer, C. Boscardin,  Exploring how feedback reflects entrustment decisions using artificial intelligence,  Medical Education 56 (2022) 303–311.
  * Frisoni and Moro [2021] G. Frisoni, G. Moro,  Phenomena explanation from text: Unsupervised learning of interpretable and statistically significant knowledge,  in: Data Management Technologies and Applications: 9th International Conference, DATA 2020, Virtual Event, July 7–9, 2020, Revised Selected Papers 9, Springer, 2021, pp. 293–318.
  * Mandalios et al. [2021] A. Mandalios, A. Chortaras, G. Stamou, M. Vazirgiannis,  Enriching graph representations of text: Application to medical text classification,  in: Complex Networks & Their Applications IX: Volume 2, Proceedings of the Ninth International Conference on Complex Networks and Their Applications COMPLEX NETWORKS 2020, Springer, 2021, pp. 92–103.
  * Zhang et al. [2022] Y. Zhang, S. Yang, H. Wen, P. Xie, Y. Wu,  Unified framework for ner and re tasks with strong interpretability on chinese medicine instruction parsing,  in: 2022 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI), IEEE, 2022, pp. 460–466.
  * Gu et al. [2020] G. Gu, B. Li, H. Gao, M. Wang,  Learning to answer complex questions with evidence graph,  in: Web and Big Data: 4th International Joint Conference, APWeb-WAIM 2020, Tianjin, China, September 18-20, 2020, Proceedings, Part I 4, Springer, 2020, pp. 257–269.
  * Duarte et al. [2018] F. Duarte, B. Martins, C. S. Pinto, M. J. Silva,  Deep neural models for icd-10 coding of death certificates and autopsy reports in free-text,  Journal of biomedical informatics 80 (2018) 64–77.
  * Zhang et al. [2018] J. Zhang, K. Kowsari, J. H. Harrison, J. M. Lobo, L. E. Barnes,  Patient2vec: A personalized interpretable deep representation of the longitudinal electronic health record,  IEEE Access 6 (2018) 65333–65346.
  * Trigueros et al. [2022] O. Trigueros, A. Blanco, N. Lebeña, A. Casillas, A. Pérez,  Explainable icd multi-label classification of ehrs in spanish with convolutional attention,  International Journal of Medical Informatics 157 (2022) 104615.
  * Chen et al. [2022] P.-F. Chen, T.-L. He, S.-C. Lin, Y.-C. Chu, C.-T. Kuo, F. Lai, S.-M. Wang, W.-X. Zhu, K.-C. Chen, L.-C. Kuo, et al.,  Training a deep contextualized language model for international classification of diseases, 10th revision classification via federated learning: Model development and validation study,  JMIR Medical Informatics 10 (2022) e41342.
  * Zhu et al. [2022] Y. Zhu, A. Mahale, K. Peters, L. Mathew, F. Giuste, B. Anderson, M. D. Wang,  Using natural language processing on free-text clinical notes to identify patients with long-term covid effects,  in: Proceedings of the 13th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, 2022, pp. 1–9.
  * Ahmed et al. [2022a] U. Ahmed, G. Srivastava, U. Yun, J. C.-W. Lin,  Eandc: An explainable attention network based deep adaptive clustering model for mental health treatment,  Future Generation Computer Systems 130 (2022a) 106–113.
  * Ahmed et al. [2022b] U. Ahmed, J. C.-W. Lin, G. Srivastava,  Multi-aspect deep active attention network for healthcare explainable adoption,  IEEE Journal of Biomedical and Health Informatics (2022b).
  * Luo et al. [2021] X. Luo, P. Gandhi, Z. Zhang, W. Shao, Z. Han, V. Chandrasekaran, V. Turzhitsky, V. Bali, A. R. Roberts, M. Metzger, et al.,  Applying interpretable deep learning models to identify chronic cough patients using ehr data,  Computer Methods and Programs in Biomedicine 210 (2021) 106395.
  * Balagopalan et al. [2021] A. Balagopalan, B. Eyre, J. Robin, F. Rudzicz, J. Novikova,  Comparing pre-trained and feature-based models for prediction of alzheimer’s disease based on speech,  Frontiers in aging neuroscience 13 (2021) 635945.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, C. Guestrin,  Model-agnostic interpretability of machine learning,  arXiv preprint arXiv:1606.05386 (2016).
  * Lundberg and Lee [2017] S. M. Lundberg, S.-I. Lee,  A unified approach to interpreting model predictions,  Advances in neural information processing systems 30 (2017).
  * Shapley [1997] L. S. Shapley,  A value for n-person games,  Classics in game theory 69 (1997).
  * Van der Maaten and Hinton [2008] L. Van der Maaten, G. Hinton,  Visualizing data using t-sne.,  Journal of machine learning research 9 (2008).
  * Hamilton [2020] W. L. Hamilton,  Graph representation learning,  Synthesis Lectures on Artifical Intelligence and Machine Learning 14 (2020) 1–159.
  * Wang et al. [2018] Q. Wang, X. Pan, L. Huang, B. Zhang, Z. Jiang, H. Ji, K. Knight,  Describing a knowledge base,  in: Proceedings of the 11th International Conference on Natural Language Generation, 2018, pp. 10–21.
  * Fu et al. [2020] B. Fu, Y. Qiu, C. Tang, Y. Li, H. Yu, J. Sun,  A survey on complex question answering over knowledge base: Recent advances and challenges,  arXiv preprint arXiv:2007.13069 (2020).
  * Bahdanau et al. [2014] D. Bahdanau, K. Cho, Y. Bengio,  Neural machine translation by jointly learning to align and translate,  arXiv preprint arXiv:1409.0473 (2014).
  * Cheng et al. [2016] J. Cheng, L. Dong, M. Lapata,  Long short-term memory-networks for machine reading,  in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016, pp. 551–561.
  * Papanastasiou et al. [2023] G. Papanastasiou, N. Dikaios, J. Huang, C. Wang, G. Yang,  Is attention all you need in medical image analysis? a review.,  IEEE Journal of Biomedical and Health Informatics (2023) 1–14. doi:10.1109/JBHI.2023.3348436.
  * Mascharka et al. [2018] D. Mascharka, P. Tran, R. Soklaski, A. Majumdar,  Transparency by design: Closing the gap between performance and interpretability in visual reasoning,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4942–4950.
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al.,  Training language models to follow instructions with human feedback,  Advances in Neural Information Processing Systems 35 (2022) 27730–27744.
  * Mosqueira-Rey et al. [2023] E. Mosqueira-Rey, E. Hernández-Pereira, D. Alonso-Ríos, J. Bobes-Bascarán, Á. Fernández-Leal,  Human-in-the-loop machine learning: A state of the art,  Artificial Intelligence Review 56 (2023) 3005–3054.
  * Wu et al. [2022] X. Wu, L. Xiao, Y. Sun, J. Zhang, T. Ma, L. He,  A survey of human-in-the-loop for machine learning,  Future Generation Computer Systems 135 (2022) 364–381.
  * Salvi et al. [2023] M. Salvi, H. W. Loh, S. Seoni, P. D. Barua, S. García, F. Molinari, U. R. Acharya,  Multi-modality approaches for medical support systems: A systematic review of the last decade,  Information Fusion (2023) 102134.
  * Schölkopf et al. [2021] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, Y. Bengio,  Toward causal representation learning,  Proceedings of the IEEE 109 (2021) 612–634.
  * Geiger et al. [2020] A. Geiger, K. Richardson, C. Potts,  Neural natural language inference models partially embed theories of lexical entailment and negation,  in: Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020, pp. 163–173.
  * Geiger et al. [2021] A. Geiger, H. Lu, T. Icard, C. Potts,  Causal abstractions of neural networks,  Advances in Neural Information Processing Systems 34 (2021) 9574–9586.
  * Wu et al. [2024] Z. Wu, A. Geiger, T. Icard, C. Potts, N. Goodman,  Interpretability at scale: Identifying causal mechanisms in alpaca,  Advances in Neural Information Processing Systems 36 (2024).
  * Serrano and Smith [2019] S. Serrano, N. A. Smith,  Is attention interpretable?,  in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2019.
  * Pruthi et al. [2020] D. Pruthi, M. Gupta, B. Dhingra, G. Neubig, Z. C. Lipton,  Learning to deceive with attention-based explanations,  in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 4782–4793.
  * Wiegreffe and Pinter [2019] S. Wiegreffe, Y. Pinter,  Attention is not not explanation,  in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 11–20.
  * Vashishth et al. [2019] S. Vashishth, S. Upadhyay, G. S. Tomar, M. Faruqui,  Attention interpretability across nlp tasks,  arXiv preprint arXiv:1909.11218 (2019).
  * Brunner et al. [2019] G. Brunner, Y. Liu, D. Pascual, O. Richter, M. Ciaramita, R. Wattenhofer,  On identifiability in transformers,  in: International Conference on Learning Representations, 2019.
  * OpenAI [2022] OpenAI,  Chatgpt: Optimizing language models for dialogue (2022). URL: https://openai.com/blog/chatgpt/.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.,  Language models are few-shot learners,  Advances in neural information processing systems 33 (2020) 1877–1901.
  * Achiam et al. [2023] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.,  Gpt-4 technical report,  arXiv preprint arXiv:2303.08774 (2023).
  * Meskó and Topol [2023] B. Meskó, E. J. Topol,  The imperative for regulatory oversight of large language models (or generative ai) in healthcare,  NPJ digital medicine 6 (2023) 120.
  * Tu et al. [2024] T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P.-C. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena, et al.,  Towards generalist biomedical ai,  NEJM AI 1 (2024) AIoa2300138.
  * Singhal et al. [2023] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al.,  Publisher correction: Large language models encode clinical knowledge,  Nature 620 (2023) E19–E19.
  * Jin et al. [2023] Z. Jin, Y. Chen, F. Leeb, L. Gresele, O. Kamal, L. Zhiheng, K. Blin, F. G. Adauto, M. Kleiman-Weiner, M. Sachan, et al.,  Cladder: Assessing causal reasoning in language models,  in: Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * Hobbhahn et al. [2022] M. Hobbhahn, T. Lieberum, D. Seiler,  Investigating causal understanding in llms,  in: NeurIPS 2022 Workshop on Causality for Real-world Impact, 2022.
  * Wang et al. [2023] Y. Wang, Y. Zhao, L. Petzold,  Are large language models ready for healthcare? a comparative study on clinical language understanding,  in: Machine Learning for Healthcare Conference, PMLR, 2023, pp. 804–823.
  * Hernandez et al. [2023] E. Hernandez, D. Mahajan, J. Wulff, M. J. Smith, Z. Ziegler, D. Nadler, P. Szolovits, A. Johnson, E. Alsentzer, et al.,  Do we still need clinical language models?,  in: Conference on Health, Inference, and Learning, PMLR, 2023, pp. 578–597.