# Can spoofing countermeasure and speaker verification systems
be jointly optimised?

###### Abstract

Spoofing countermeasure (CM) and automatic speaker verification (ASV) sub-
systems can be used in tandem with a backend classifier as a solution to the
spoofing aware speaker verification (SASV) task. The two sub-systems are
typically trained independently to solve different tasks. While our previous
work demonstrated the potential of joint optimisation, it also showed a
tendency to over-fit to speakers and a lack of sub-system complementarity.
Using only a modest quantity of auxiliary data collected from new speakers, we
show that joint optimisation degrades the performance of separate CM and ASV
sub-systems, but that it nonetheless improves complementarity, thereby
delivering superior SASV performance. Using standard SASV evaluation data and
protocols, joint optimisation reduces the equal error rate by 27% relative to
performance obtained using fixed, independently-optimised sub-systems under
like-for-like training conditions.

Index Terms—  spoofing detection, spoofing countermeasures, speaker
verification, joint optimisation

##  1 Introduction

Solutions to spoofing aware speaker verification (SASV) [1] typically comprise
a pair of essentially-independent sub-systems operating in tandem: a spoofing
countermeasure (CM) to verify that the test utterance is bona fide rather than
spoofed; an automatic speaker verification (ASV) system to verify whether
enrolment and test utterances correspond to the same speaker. Even so, ASV
also has potential to detect spoofing attacks. In support of this claim, one
can imagine the presentation of poor-quality spoofing attacks, namely
artificially generated or manipulated utterances which do not represent well
the characteristics of the target speaker. Such poor-quality spoofs may not
fool the ASV sub-system and hence be rejected even without an auxiliary CM.

ASV and CM sub-systems are therefore not independent and there is hence
potential for joint optimisation to exploit their complementarity. The CM can
be optimised to detect spoofing attacks which will fool the ASV system,
thereby avoiding the unnecessary waste of CM classifier capacity. Despite the
appeal of joint optimisation, to the best of our knowledge, no successful
solutions have been reported thus far, with all leading solutions combining
independently-optimised sub-systems with a backend classifier, e.g. using
score [2, 3] or embedding [4, 5] fusion techniques.

Our previous work [6] showed that jointly-optimised SASV solutions can succeed
in improving robustness to spoofing but that they have also a tendency to
over-fit to the speakers of data used for training. This is hardly surprising;
state-of-the-art ASV systems are nowadays trained using data collected from
many hundreds of speakers [7] whereas the training partition of the SASV
database [8] contains data collected from only 20 speakers. We showed the
number of speakers whose data is used for training is a bottleneck and that
progress is unlikely to be made without using additional, auxiliary data
collected from a larger pool of speakers.

The work reported in this paper aims to establish the potential and determine
whether spoofing countermeasures and speaker verification systems can be
jointly optimised. Our hypothesis is that, with data collected from a
sufficient speaker population, joint optimisation can deliver superior SASV
performance to that obtained using fixed, independently-optimised systems
trained under like-for-like training conditions. While joint optimisation
might even degrade CM and/or ASV performance, our expectation is that
improvements will emerge from more complementary CM and ASV sub-systems that
work in synergy to provide more reliable SASV solutions.

##  2 SASV framework

All work reported in this paper was performed using the same SASV framework
used in our previous work [6]. While the framework is the same, there are
differences in the training policy in terms of data and trial types. A summary
of the framework is presented in this section. The new training policy is
described in Section 3.

The architecture is illustrated in Fig. 1 and consists of an ASV sub-system, a
CM sub-system and a backend classifier. The ASV sub-system is a ResNet34 model
with squeeze-and-excitation (SE) blocks [9], namely the ResNetSE34 model
reported in [10]. The input waveform is first decomposed into log Mel-
filterbank features. Four convolutional layers with SE blocks and attentive
statistics pooling [11] are used for deep feature extraction and to project
the variable length input to a fixed-length embedding vector. A combination of
softmax and angular prototypical loss [12] is used for optimisation. The ASV
sub-system extracts both an enrolment embedding $e^{ASV}_{enr}$ and test
embedding $e^{ASV}_{tst}$ from corresponding utterances.

The CM sub-system is the SASV 2022 AASIST [13] baseline. Raw waveforms are
decomposed using a RawNet2-based encoder [14, 15]. Extracted features are
integrated using a graph attention network [16] before a readout operation and
a hidden linear layer are used to generate output scores. Optimisation is
performed using a weighted cross-entropy loss function. A linear layer is used
to transform representations extracted from the penultimate layer to CM
embeddings $e^{CM}_{tst}$ of the same dimension as ASV embeddings.

The backend classifier is a convolutional neural network with an adaptive
average pooling layer [17]. It operates upon the pair of ASV embeddings and
the single CM embedding. These are first stacked along a new dimension so that
three 1D convolutional layers can be used to capture the variance between
speaker representations of both enrolment and test utterances, and also the
variance between the ASV and CM embeddings. Deep representations are
aggregated using a 1D adaptive average pooling layer and are then mapped to a
bona fide, target class score using a pair of linear layers and a one class
(OC) softmax layer [18]. A OC-softmax loss function is used to learn higher
output scores for the target class.

Fig. 1: Framework for fixed, independently-optimised and jointly-optimised
SASV solutions comprising ASV and CM sub-systems and a backend classifier.

##  3 Experimental Setup

We describe the set of databases, protocols, assessment and metrics used in
this work, together with specific implementation details.

###  3.1 Databases

Experimental work was performed using three databases. The VoxCeleb2 [7]
database is used for ASV pre-training. The CM is pre-trained using the
training partition of the ASVspoof 2019 Logical Access (LA) database [8],
referred to in the remainder of this paper simply as ASVspoof. The key
characteristics of the ASVspoof training partition are detailed in the top
half of Table 1. It contains data collected from 20 speakers, a number far
less than databases typically used for ASV research [7] and, as we found in
previous work [6], a number that is insufficient to support joint
optimisation. All development and evaluation experiments are performed using
the corresponding partitions of the ASVspoof database.

The Fake Audio Detection (FAD) database [19] contains bona fide and spoofed
Mandarin-language utterances generated using TTS algorithms. It is used
optionally for training the backend classifier or, in the case of joint
optimisation, the backend classifier and both CM and ASV sub-systems. The FAD
database is sourced from 6 different public databases [20, 21, 22, 23]. Its
non-homogeneous nature makes it ideal as a source of speaker augmentation. To
be consistent with the ASVspoof database, we used the data collected from
speakers for whom there is both bona fide and spoofed data. The key
characteristics of the resulting training partition used in this work are
detailed in the lower half of Table 1. It contains data collected from 40
speakers, a number twice that of the ASVspoof training partition.

Table 1: Details of the ASVspoof 2019 LA and FAD training partitions used for
all experiments reported in this paper.

Database # of speakers # of bona fide # of spoofed # of attacks ASVspoof train
20 2580 22800 4 TTS, 2 VC FAD train 40 3200 25600 8 TTS

###  3.2 Protocols

We used SASV [1] protocols for the ASVspoof database for all work reported in
this paper, modifying only the training protocol to incorporate optional FAD
data. For fixed optimisation, pre-trained sub-systems are used without
additional optimisation. The backend is optimised according to one of three
different training conditions using either: ASVspoof data alone; ASVspoof +
FAD data; ASVspoof + FAD bona fide data only. The last training condition is
included since we aim to reduce the over-fitting of the ASV sub-system which
is usually trained with bona fide data only. In doing so we avoid, to the
extent possible, domain mismatch in terms of spoofing attacks (which is not
the goal of this work); domain mismatch in the bona fide data nonetheless
remains. Only for the jointly-optimised system are all three components
trained simultaneously through back propagation.

Training is performed using different combinations of trial pairs including a
bona fide target enrolment utterance in addition to a test utterance. The
latter can be: \raisebox{-.9pt}{1}⃝ a bona fide target speaker utterance;
\raisebox{-.9pt}{2}⃝ a bona fide non-target speaker utterance;
\raisebox{-.9pt}{3}⃝ a spoofed target speaker utterance. The proportion of
each trial type for fixed and jointly-optimised systems is illustrated in
Table 2. For joint optimisation we found the use of a fourth trial type
involving \raisebox{-.9pt}{4}⃝ a spoofed non-target speaker utterance to be
beneficial, but found no improvement for our fixed system. The full SASV
system should accept only type \raisebox{-.9pt}{1}⃝ trials. All others should
be rejected, even though type \raisebox{-.9pt}{2}⃝ trials are within the
positive class for the CM.111By convention, the CM produces higher scores for
bona fide inputs and lower scores for spoofed inputs. The conflict between
SASV and CM classes then results in degraded CM performance through back
propagation. The introduction of type \raisebox{-.9pt}{4}⃝ trials acts to
compensate for this behaviour since they are among the negative class for SASV
and both sub-systems. Finally, the development and evaluation protocols are
the standard SASV protocols reported in [1, 24].

###  3.3 Assessment and metrics

We use the output scores from the backend OC-softmax layer to assess the
performance of both fixed and jointly-optimised systems. In order to analyse
impacts of joint optimisation and complementarity, we also assess performance
at the sub-system level. ASV sub-system scores are the cosine distance between
enrolment and test utterance embeddings. CM sub-system scores are the AASIST
outputs for the bona fide class and test utterance only. Three different SASV
metrics, all equal error rate (EER) estimates, are used to assess speaker
verification performance (SV-EER) involving a set of \raisebox{-.9pt}{1}⃝ and
\raisebox{-.9pt}{2}⃝ trials, spoofing detection performance (SPF-EER)
involving a set of \raisebox{-.9pt}{1}⃝ and \raisebox{-.9pt}{3}⃝ trials and
spoofing-aware speaker verification (SASV-EER) performance involving the full
set of \raisebox{-.9pt}{1}⃝, \raisebox{-.9pt}{2}⃝ and \raisebox{-.9pt}{3}⃝
trials. The remaining trial type \raisebox{-.9pt}{4}⃝ is not used for
assessment, neither for development, nor for evaluation.

Table 2: SASV training pairs and the corresponding proportions for fixed and
joint optimisation.

Pair Test utterance Prop. in Fixed Prop. in Joint \raisebox{-.9pt}{1}⃝ Bona
fide, Target spk 50% 25% \raisebox{-.9pt}{2}⃝ Bona fide, Non-target spk 25%
25% \raisebox{-.9pt}{3}⃝ Spoofed, Target spk 25% 25% \raisebox{-.9pt}{4}⃝
Spoofed, Non-target spk None 25%

###  3.4 Implementation

All trainable network parameters are updated for 20 epochs with an initial
learning rate of 5e-5. The batch size is set to 20 for both systems. Model
selection is made according to the lowest SASV-EER estimate for the
development partition. Systems performance is estimated from the average of 5
independent runs each with a different random seed. All experiments were
performed on a single NVIDIA GeForce RTX 3090 GPU. Results are reproducible
with the same set of random seeds and GPU environment using the implementation
available online.222https://github.com/eurecom-asp/sasv-joint-optimisation

##  4 Results

Table 3: Averaged results for pre-trained, jointly-optimised systems for SASV
2022 evaluation partitions. Results for ASV and CM sub-systems under fixed
configuration are same since the networks are identical. The corresponding
results for full system are different because their backend classifiers are
trained using different databases. Results in boldface indicate better
performance for jointly-optimised systems than for corresponding fixed,
independently-optimised systems.

Training data | Configuration | Full system | ASV sub-system | CM sub-system
---|---|---|---|---
SASV-EER | SV-EER | SPF-EER | SASV-EER | SV-EER | SPF-EER | SASV-EER | SV-EER | SPF-EER
ASVspoof | Fixed | 1.15 | 1.27 | 1.08 | 19.70 | 1.27 | 25.75 | 24.50 | 49.01 | 0.65
Joint | 1.49 | 2.34 | 0.80 | 13.47 | 1.84 | 17.43 | 23.65 | 46.80 | 1.02
ASVspoof + | Fixed | 1.52 | 1.85 | 1.27 | 19.70 | 1.27 | 25.75 | 24.50 | 49.01 | 0.65
FAD | Joint | 1.74 | 2.66 | 1.09 | 8.57 | 1.73 | 10.82 | 22.83 | 46.37 | 1.81
ASVspoof + | Fixed | 1.72 | 1.47 | 1.84 | 19.70 | 1.27 | 25.75 | 24.50 | 49.01 | 0.65
FAD bona fide only | Joint | 1.26 | 1.77 | 0.83 | 8.58 | 1.49 | 10.60 | 24.66 | 47.99 | 1.35

Results are shown in Table 3. It shows SASV-EER, SV-EER, and SPF-EER estimates
for the full system (left), the ASV sub-system (middle) and CM sub-system
(right) when training is performed using ASVspoof data alone (top), ASVspoof
and FAD data (middle) and then ASVspoof and only FAD bona fide data (bottom).
In the following we outline a number of principle observations and supporting
results.

CM performance – SPF-EER estimates for the CM sub-system in the last column of
Table 3 show that the lowest SPF-EER (0.65%) is obtained using a fixed system
trained using only ASVspoof data. The degradations in CM performance (1.02%,
1.81% and 1.35%) stem from the CM being optimised to detect only attacks that
are successful in fooling the ASV system. While there is no need for the CM to
detect less potent attacks, but since performance estimates are still made
using the full set, the SPF-EER increases. The worst result is for the
ASVspoof + FAD training condition, and is the result of differences between
spoofing attacks in the ASVspoof and FAD databases. SPF-EER estimates for the
ASV sub-system (25.75%) show that even the fixed system can detect spoofing
attacks, but that joint optimisation acts to improve performance (17.43%,
10.82% and 10.60%). SPF-EER results for the full system show the
complementarity of the ASV and CM sub-systems in improving overall robustness
to spoofing for all three training conditions.

ASV performance – SV-EER estimates for the CM sub-system shown in the
penultimate column of Table 3 show evidence of speaker-awareness. Without
over-fitting to speakers, the CM sub-system should achieve an SV-EER of 50%.
Results for jointly-optimised systems (46.8%, 46.37% and 47.99%) deviate
further than the result for the fixed system (49.01%). Corresponding SV-EER
results for the ASV sub-system show that joint optimisation acts to degrade
performance, with the best result being achieved using the fixed system
(1.27%). This translates to worse ASV performance for the full system; SV-EER
results for all jointly-optimised systems (2.34%, 2.66% and 1.77%) are all
worse than those for fixed systems (1.27%, 1.85% and 1.47%). Again, the worst
result is for the ASVspoof + FAD training condition.

SASV performance – Joint optimisation makes little difference to the SASV-EER
for the CM sub-system (24.5% vs. 23.65%, 22.83% and 24.66%). This is not
surprising, since the CM sub-system operates only upon the test utterance.
Nonetheless, joint optimisation improves ASV sub-system performance, with
results for all jointly-optimised systems (13.47%, 8.57% and 8.58%) being
better than the fixed system (19.7%). Even if there is little difference in CM
results, but substantial improvements to ASV results, this does not translate
to better performance for the full system in the case of the ASVspoof (1.15%
vs. 1.49%) and ASVspoof + FAD (1.52% vs. 1.74%) training conditions. Only for
the ASVspoof + FAD bona fide condition is joint optimisation beneficial (1.72%
vs. 1.26%).

##  5 Discussion

The lowest SASV-EER comes from the fixed system trained using ASVspoof data
alone. This result may cast doubt upon the claimed merit since, even with more
training data, the jointly-optimised system does not outperform the fixed
counterpart. However, while the jointly-optimised system trained using
ASVspoof + FAD bona fide only data does give performance that is behind that
of the fixed system trained with ASVspoof data only, the use of different
training data makes for an unfair comparison. The FAD training data is out-of-
domain in the form of speech data in a different language and recording
conditions, among other differences. In this sense, comparisons should be made
only between results for the same training conditions, i.e. results for
ASVspoof or ASVspoof + FAD bona fide only training conditions. In this sense,
lower EERs observed across different training conditions are not an indication
of joint optimisation not working, but are the result of domain mismatch which
is not tackled in this work. Domain mismatch is a penalty which likely
degrades results for training conditions that use FAD data. The use of domain-
matched training data, or domain adaptation, may than show the full potential
and even better results.

Results for the ASVspoof and FAD bona fide only training condition show a 27%
relative reduction in the SASV-EER under like-for-like training conditions. It
comes as a result of using data collected from a greater number of speakers
and jointly-optimised CM and ASV sub-systems with worse performance than their
independently-optimised counterparts. Even so, the resulting sub-systems are
more complementary leading to better overall reliability (SASV-EER).

##  6 Conclusions and further work

Results presented in this paper add to the evidence that joint optimisation
has potential to better exploit the synergy between spoofing countermeasures
and speaker verification sub-systems so that they function cooperatively as a
more reliable solution to spoofing aware speaker verification (SASV).
Interestingly, results show that joint optimisation degrades the performance
of each sub-system but that it improves their complementarity. Joint
optimisation improves SASV performance by 27% relative to its fixed,
independently-optimised counterpart under like-for-like training conditions.
This result is obtained despite using a training database of the same size to
jointly-optimise a considerably more complex system.

While other approaches to joint optimisation might make better use of data
collected from fewer speakers, our solution is only successful when using a
modest quantity of auxiliary data collected from new speakers. Given that
state-of-the-art ASV solutions are trained with data collected from many
hundreds of speakers, it seems wise for future ASVspoof and SASV challenges to
collect and make available training data collected from far more speakers than
in past editions. The domain robustness problem persists; even if we can learn
speaker variation in a joint optimisation framework using bona fide data
sourced from a different database, it does not translate to reliable detection
for spoofing attacks in the same database. Research in domain robustness is
hence a priority. Other directions include the investigation of joint
optimisation strategies to reduce CM over-fitting; the CM shows signs of
speaker awareness. Speaker-dependent spoofing detection might also be an
interesting avenue for future research.

## References

  * [1] J.-w. Jung, H. Tak, H.-j. Shim, H.-S. Heo, B.-J. Lee, S.-W. Chung, H.-J. Yu, N. Evans, and T. Kinnunen,  “SASV 2022: The first spoofing-aware speaker verification challenge,”  in Proc. Interspeech, 2022, pp. 2893–2897.

  * [2] A. Alenin, N. Torgashov, A. Okhotnikov, R. Makarov, and I. Yakovlev,  “A subnetwork approach for spoofing aware speaker verification,”  in Proc. Interspeech, 2022, pp. 2888–2892.
  * [3] X. Wang, X. Qin, Y. Wang, Y. Xu, and M. Li,  “The DKU-OPPO system for the 2022 spoofing-aware speaker verification challenge,”  in Proc. Interspeech, 2022, pp. 4396–4400.
  * [4] J. Heo, J.-h. Kim, and H.-s. Shin,  “Two methods for spoofing-aware speaker verification: Multi-layer perceptron score fusion model and integrated embedding projector,”  in Proc. Interspeech, 2022, pp. 2878–2882.
  * [5] Y. Zhang, G. Zhu, and Z. Duan,  “A probabilistic fusion framework for spoofing aware speaker verification,”  in Proc. The Speaker and Language Recognition Workshop (Odyssey 2022), 2022, pp. 77–84.
  * [6] W. Ge, H. Tak, M. Todisco, and N. Evans,  “On the potential of jointly-optimised solutions to spoofing attack detection and automatic speaker verification,”  in Proc. IberSPEECH, 2022, pp. 51–55.
  * [7] J. S. Chung, A. Nagrani, and A. Zisserman,  “VoxCeleb2: Deep speaker recognition,”  in Proc. Interspeech, 2018, pp. 1086–1090.
  * [8] X. Wang, J. Yamagishi, M. Todisco, H. Delgado, A. Nautsch, N. Evans, et al.,  “ASVspoof 2019: A large-scale public database of synthetized, converted and replayed speech,”  Computer Speech & Language, vol. 64, 2020,  101114\.
  * [9] J. Hu, L. Shen, and G. Sun,  “Squeeze-and-excitation networks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
  * [10] Y. Kwon, H.-S. Heo, B.-J. Lee, and J. S. Chung,  “The ins and outs of speaker recognition: lessons from VoxSRC 2020,”  in Proc. ICASSP, 2021, pp. 5809–5813.
  * [11] K. Okabe, T. Koshinaka, and K. Shinoda,  “Attentive statistics pooling for deep speaker embedding,”  in Proc. Interspeech, 2018, pp. 2252–2256.
  * [12] J. Wang, F. Zhou, S. Wen, X. Liu, and Y. Lin,  “Deep metric learning with angular loss,”  in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2593–2601.
  * [13] J.-w. Jung, H.-S. Heo, H. Tak, H.-j. Shim, J. S. Chung, B.-J. Lee, H.-J. Yu, and N. Evans,  “AASIST: Audio anti-spoofing using integrated spectro-temporal graph attention networks,”  in Proc. ICASSP, 2022, pp. 6367–6371.
  * [14] J.-w. Jung, S.-b. Kim, H.-j. Shim, J.-h. Kim, and H.-J. Yu,  “Improved RawNet with feature map scaling for text-independent speaker verification using raw waveforms,”  in Proc. Interspeech, 2020, pp. 1496–1500.
  * [15] H. Tak, J. Patino, M. Todisco, A. Nautsch, N. Evans, and A. Larcher,  “End-to-end anti-spoofing with rawnet2,”  in Proc. ICASSP, 2021, pp. 6369–6373.
  * [16] H. Tak, J.-w. Jung, J. Patino, M. Kamble, M. Todisco, and N. Evans,  “End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection,”  in Proc. ASVspoof 2021 Workshop, 2021, pp. 1–8.
  * [17] L. Zhang, Y. Li, H. Zhao, and L. Xie,  “Backend ensemble for speaker verification and spoofing countermeasure,”  in Proc. Interspeech, 2022, pp. 4381–4385.
  * [18] Y. Zhang, F. Jiang, and Z. Duan,  “One-class learning towards synthetic voice spoofing detection,”  IEEE Signal Processing Letters, vol. 28, pp. 937–941, 2021.
  * [19] H. Ma, J. Yi, C. Wang, X. Yan, J. Tao, T. Wang, S. Wang, L. Xu, and R. Fu,  “FAD: A Chinese dataset for fake audio detection,”  arXiv preprint:2207.12308, 2022.
  * [20] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng,  “AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline,”  in Proc. 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment, 2017, pp. 1–5.
  * [21] Y. Shi, H. Bu, X. Xu, S. Zhang, and M. Li,  “AISHELL-3: A multi-speaker mandarin TTS corpus,”  in Proc. Interspeech, 2021, pp. 2756–2760.
  * [22] W. Dong and X. Zhang,  “Thchs-30: A free chinese speech corpus,”  arXiv preprint:1512.01882, 2015.
  * [23] Z. Yang, Y. Chen, L. Luo, R. Yang, L. Ye, G. Cheng, J. Xu, Y. Jin, Q Zhang, P. Zhang, L. Xie, and Y. Yan,  “Open Source MagicData-RAMC: A rich annotated Mandarin conversational (RAMC) speech dataset,”  in Proc. Interspeech 2022, 2022, pp. 1736–1740.
  * [24] H.-j. Shim, H. Tak, X. Liu, H.-S. Heo, J.-w. Jung, J. S. Chung, S.-W. Chung, et al.,  “Baseline systems for the first spoofing-aware speaker verification challenge: score and embedding fusion,”  in Proc. Odyssey 2022 Workshop, 2022, pp. 330–337.


8pt

0.7(0.16,0.88) Derks I.P., de Waal A. (2020) A Taxonomy of Explainable
Bayesian Networks. In: Gerber A. (eds) Artificial Intelligence Research.
SACAIR 2021. Communications in Computer and Information Science, vol 1342.
Springer, Cham.
https://doi.org/10.1007/978-3-030-66151-9˙14

11institutetext: Department of Statistics, University of Pretoria
22institutetext: Center for Artificial Intelligence Research (CAIR)

# A Taxonomy of Explainable Bayesian Networks

Iena Petronella Derks 11 0000-0002-7070-5036 Alta de Waal 1122
0000-0001-8121-6249

###### Abstract

Artificial Intelligence (AI), and in particular, the explainability thereof,
has gained phenomenal attention over the last few years. Whilst we usually do
not question the decision-making process of these systems in situations where
only the outcome is of interest, we do however pay close attention when these
systems are applied in areas where the decisions directly influence the lives
of humans. It is especially noisy and uncertain observations close to the
decision boundary which results in predictions which cannot necessarily be
explained that may foster mistrust among end-users. This drew attention to AI
methods for which the outcomes can be explained. Bayesian networks are
probabilistic graphical models that can be used as a tool to manage
uncertainty. The probabilistic framework of a Bayesian network allows for
explainability in the model, reasoning and evidence. The use of these methods
is mostly ad hoc and not as well organised as explainability methods in the
wider AI research field. As such, we introduce a taxonomy of explainability in
Bayesian networks. We extend the existing categorisation of explainability in
the model, reasoning or evidence to include explanation of decisions. The
explanations obtained from the explainability methods are illustrated by means
of a simple medical diagnostic scenario. The taxonomy introduced in this paper
has the potential not only to encourage end-users to efficiently communicate
outcomes obtained, but also support their understanding of how and, more
importantly, why certain predictions were made.

###### Keywords:

Bayesian network Reasoning Explainability.

##  1 Introduction

Advances in technology have contributed to the generation of big data in
nearly all fields of science, giving rise to new challenges with respect to
explainability of models and techniques used to analyse such data. These
models and techniques are often too complex; concealing the knowledge within
the machine, hence decreasing the extent of interpretability of results.
Subsequently, the lack of explainable models and techniques contribute to
mistrust among users in fields of science where interpretability and
explainability are indispensable.

To elucidate the need for explainable models, consider the following three
scenarios. Firstly, suppose a medical diagnosis system is used to determine
whether a tumour sample is malignant or benign. Here, the medical practitioner
must be able to understand how and why the system reached the decision, and,
if necessary, inspect whether the decision is supported by medical knowledge
[2]. Next, consider self-driving cars. In this context, the self-driving car
must be able to process information faster than a human, such that accidents
and fatalities can be avoided [21]. Suppose a self-driving car is involved in
an accident, then the system must be able to explain that in order to avoid
hitting a pedestrian, the only option was to swerve out of the way and, by
coincidence, into another vehicle. Lastly, consider an online restaurant
review system, where reviews are classified as positive or negative based on
the words contained in the review. Here, the classifier simply returns whether
a review is positive or negative, without explaining which words contributed
to the classification. As such, negative reviews that are expressed in, for
example, a sarcastic manner, might be classified as positive, resulting in a
restaurant receiving a higher rating and more diners – who might experience
bad service (or even food poisoning) as a result of mislabelled reviews.

Given its relevance in many application areas, the explainability problem has
attracted a great deal of attention in recent years, and as such, is an open
research area [24]. The manifestation of explainable systems in high-risk
areas has influenced the development of explainable artificial intelligence
(XAI) in the sense of prescriptions or taxonomies of explanation. These
include fairness, accountability, transparency and ethicality [3, 11, 23]. The
foundation of such a system should include these prescriptions such that a
level of usable intelligence is reached to not only understand model behaviour
[1] but also understand the context of an application task [14]. Bayesian
networks (BNs) – which lie at the intersection of AI, machine learning, and
statistics – are probabilistic graphical models that can be used as a tool to
manage uncertainty. These graphical models allow the user to reason about
uncertainty in the problem domain by updating ones beliefs, whether this
reasoning occurs from cause to effect, or from effect to cause. Reasoning in
Bayesian networks is often referred to as what-if questions. The flexibility
of a Bayesian network allows for these questions to be predictive, diagnostic
and inter-causal. Some what-if questions might be intuitive to formulate, but
this is not always the case especially on a diagnostic and inter-causal level.
This might result in sub-optimal use of explainability in BNs - especially on
an end-user level. Apart from well-established reasoning methods, the
probabilistic framework of a Bayesian network also allows for explainability
in evidence. These include most probable explanation and most relevant
explanation. To extend on the existing explainability methods, we propose an
additional approach which considers explanations concerned with the decision-
base.

In this paper, we research the current state of explainable models in AI and
machine learning tasks, where the domain of interest is BNs. In the current
research, explanation is often done by principled approaches to finding
explanations for models, reasoning, and evidence. Using this, we are able to
formulate a taxonomy of explainable BNs. We extend this taxonomy to include
explanation of decisions. This taxonomy will provide end-users with a set of
tools to better understand predictions made by BNs and will therefore
encourage efficient communication between end-users. The paper is structured
as follows. We first investigate the community and scope of explainability
methods in Section 2. Thereafter, we introduce explanation in BNs, which
includes the formulation of principled approaches, the theoretical properties
associated therewith and a hands-on medical diagnosis example. Section 4
presents our newly formulated taxonomy of explainable BNs. The final section
concludes the paper and includes a short discussion of future work.

##  2 Related Work

In application areas where erroneous decisions have a direct impact on
livelihood, relying on systems where the predictions cannot be explained may
not be an option. Explainability in such systems aids in establishing trust in
not only circumstances where the system is used as a primary decision tool,
but also cases where the system takes on a supportive role [28].

Over the past few years, explainability in AI systems has gained immense
attention from the research community. This is reflected in the launch of
various events and organisations. The Defense Advanced Research Projects
Agency (DARPA) launched the Explainable Artificial Intelligence (XAI)
initiative in 2016. The XAI programs intention is to encourage the production
of AI techniques where emphasis is placed on developing more accurate and
precise models, while still maintaining a high level of explainability.
Ultimately, XAI systems must be able to explain their rationale and enable
understanding [12]. Conferences, such as the International Joint Conferences
on Artificial Intelligence (IJCAI) conducts workshops specifically focusing on
XAI [26]. This topic has also made a noticeable appearance at the Neural
Information Processing Systems (NeurIPS) conference, with panel discussions
solely concentrating on XAI.

The scope of explainability is inherently linked to the complexity of the
model, as well as the goal thereof. Usually, but not necessarily, there is a
trade-off between model accuracy and explainability – the higher the accuracy,
the lower the explainability [31]. For example, decision trees provide a clear
explanation but are often less accurate than deep learning models, which are
less transparent. It should be mentioned that this trade-off is also connected
to the quality of data. AI and machine learning models that are transparent by
design, such as linear regression, decision trees and k-nearest neighbours,
convey a degree of explainability [1]. However, when AI and machine learning
models do not provide clear explanations, separate explainability methods are
applied to the model to gain meaningful explanations. Methods of
explainability are not limited to the behaviour of the model or decision-
making process as a whole, and may be applied to single instances, predictions
or decisions [6]. These explanations can be in the form of visualisations or
natural language [10]. Some of the existing explainability methods are layer-
wise relevance propagation (LRP), which are often used in deep neural networks
where the prediction made by the network is propagated back into the neural
network using a set of predefined rules [27]. Another explainability method is
local interpretable model-agnostic explanations (LIME). LIME methods can be
used to explain prediction instances by attempting to understand the behaviour
of the prediction function in the context of the prediction. Here, the user is
able to obtain a local explanation for that particular instance. LIME can also
be used to obtain explanations for the entire model by generating multiple
instances [17]. Methods of explainability are also extended to document
classifiers, where documents are classified based on predicted likelihood.
Here, explanations can be produced based on a search through the text-space of
possible word combinations – starting with a single word and expanding the
number of words until an explanation is found [25].

Uncertainty is present in the majority of AI fields, such as knowledge
representation, learning and reasoning [22]. Real-world data often contain
noisy and uncertain observations close to the decision boundary, which may
result in predictions that cannot be explained [7]. Probabilistic graphical
models can be seen as uncertainty management tools as they are able to
represent and reason with uncertainty. These probabilistic models are often
employed to support decision making in various application fields, including
legal and medical applications [29]. One such probabilistic model is BNs,
which is capable of combining expert knowledge and statistical data, therefore
allowing for complex scenarios to be modelled. However, not only are the inner
workings of Bayesian networks complicated to most end-users [15], the
explanation of probabilistic reasoning is challenging and as such results
appear to be counter-intuitive or wrong [16]. Therefore, there exists a demand
for explanation in Bayesian networks.

Explanation methods for Bayesian networks can be divided into three broad
approaches. The first approach consists of presenting information contained in
the knowledge base and is known as explanation of the model. There are two
objectives associated with this type of explanation. Firstly, explanation of
the model is used to assist application experts in the model-construction
phase. Secondly, it is used for instructional purposes to offer knowledge
about the domain [19]. The objective of the second approach is to justify the
conclusion and how it was obtained. This approach is referred to as
explanation of reasoning [9]. The final approach, explanation of evidence, is
concerned with the treatment of the variables in the Bayesian network [13]. In
explanation of evidence, also referred to as abduction, an explanation is seen
as the configuration of a portion of the variables present in the Bayesian
network, given evidence. Not included in the aforementioned explanation
methods are techniques that describe whether the end-user is ready to make a
decision, and if not, what additional information is required to better
prepare for decision making. Techniques such as sensitivity analysis [4] and
same-decision probability [5] provide the end-user with insight on decisions.
We group these methods into a fourth approach, explanation of decisions. For
the purpose of this paper, and accordingly, the formulation of the explainable
taxonomy, we only consider explanation of reasoning, evidence, and decisions.
Explanation of the model is excluded from this taxonomy – at the time being –
as the intent of the taxonomy is to support understanding of how and why
predictions were made and not on the model-construction itself.

##  3 Explainable Bayesian networks

We adapt the XAI terminology to the scope of BNs by defining the term XBN and
thereby referring to explainable BNs. To illustrate XBN in Bayesian networks,
consider the Asia network from Lauritzen and Spiegelhalter (1988) [20] as an
example.
Example statement: Suppose a patient visits a doctor, complaining about
shortness of breath (dyspnoea) (D). The patient is worried he might have lung
cancer. The doctor knows that lung cancer is only one of the possible causes
for dyspnoea, and other causes include bronchitis (B) and tuberculosis (T).
From her training, the doctor knows that smoking increases the probability of
lung cancer (C) and bronchitis. Both tuberculosis and lung cancer would result
in an abnormal X-ray (X) result. Lastly, a recent visit to Asia might increase
the probability of tuberculosis, as the disease is more prevalent there than
the patient’s country of origin.

From this example statement, the nodes and values are defined and then the
graphical structure of the BN is constructed. This is followed by the
quantification of the conditional probability tables (CPTs) for each node [18]
111All BN models are constructed in BayesiaLab (www.bayesialab.com). The final
BN is illustrated in Figure 1. Now that the domain and uncertainty are
represented in the BN, we will look into how to use the BN. Reasoning in BNs
takes place once we observe the value of one or more variables and we want to
condition on this new information [18]. It is important to note that this
information need not necessarily flow in the direction of the arcs, and
therefore, reasoning can occur in the opposite direction of the arcs.

Figure 1: Asia Bayesian network

###  3.1 Explanation of reasoning

Suppose during the doctor’s appointment, the patient tells the doctor he is a
smoker before any symptoms are assessed. As mentioned earlier, the doctor
knows smoking increases the probability of the patient having lung cancer and
bronchitis. This will, in turn, also influence the expectation of other
symptoms, such as the result of the chest X-Ray and shortness of breath. Here,
our reasoning is performed from new information about the causes to new
beliefs of the effects. This type of reasoning is referred to as predictive
reasoning and follows the direction of the arcs in the network. Through
predictive reasoning, we are interested in questions concerning what will
happen. In some cases, predictive reasoning is not of great insight and it is
often required to reason from symptoms (effect) to cause, which entails
information flow in the opposite direction to the network arcs. For example,
bronchitis can be seen as an effect of smoking. Accordingly, we are interested
in computing $P(S|B)$. This is referred to as diagnostic reasoning and is
typically used in situations where we want to determine what went wrong. The
final type of probabilistic reasoning in BNs is inter-causal reasoning, which
relates to mutual causes of a common effect – typically indicated by a
v-structure in the network. In other words, inference is performed on the
parent nodes of a shared child node. Note that the parent nodes are
independent of one another unless the shared child node is observed, a concept
known as d-separation. From the Asia network, we observe a v-structure between
Tuberculosis, Lung Cancer and Tuberculosis or Cancer (see Figure 2(a)). Here,
Tuberculosis is independent from Lung cancer. Suppose we observe the patient
has either Tuberculosis or Cancer – indicated by the green (or light grey if
viewed in grey-scale) bar in Figure 2(b) – then this observation increases the
probabilities of the parent nodes, Tuberculosis and Lung Cancer. However, if
it is then revealed that the patient does, in fact, have Tuberculosis it, in
turn, lowers the probability of a patient having Lung Cancer (see Figure
2(c)). We can then say Lung Cancer has been explained away. It should be noted
that the probabilistic reasoning methods discussed above can be used as is, or
can be combined to accommodate the problem at hand.

(a) Joint Probability Tables for T, C and P

(b) Adding evidence to P

(c) Adding evidence to T

Figure 2: Belief Updating for T, C and P

###  3.2 Explanation of evidence

Sometimes, users of the system find the results of reasoning unclear or
questionable. One way to address this is to provide scenarios for which the
reasoning outcomes are upheld. A fully specified scenario is easier to
understand than a set of reasoning outcomes. Explanation of evidence methods
are useful in specifying these scenarios. They are based on the posterior
probability and the generalised Bayes factor. Firstly, we focus on methods
that aim to find a configuration of variables such that the posterior
probability is maximised given the evidence. Here, we consider the Most
Probable Explanation (MPE), which is a special case of the Maximum A
Posteriori (MAP). The MAP in a BN is a variable configuration which includes a
subset of unobserved variables in the explanation set such that the posterior
probability – given evidence – is maximised. Similarly, if the variable
configuration consists of all variables present in the explanation set, we
have an MPE solution [13]. However, in some real-world applications, the
variable set often consists of a large number of variables, which may result
in over-specified or under-specified explanations obtained from the MPE. In
fact, only a few variables may be relevant in explaining the evidence. The
next approach finds a single instantiation that maximises the generalised
Bayes factor in a trans-dimensional space containing all possible partial
instantiations. In other words, this approach aims to obtain an explanation
only consisting of the most relevant variables in the BN, given the evidence.
This approach is known as the Most Relevant Explanation (MRE) [34, 32, 33].

####  3.2.1 Most Probable Explanation

Let’s first consider the MPE method. Recall that the MPE finds the complete
instantiation of the target variables – which are defined to be unobserved –
such that the joint posterior probability is maximised given evidence. Figure
3 shows the scenario (or case) that has the highest joint probability in the
Asia network. Note here the probabilities are replaced by the likelihood of
the variable state belonging to the most probable scenario, for example, if we
look at the two possible states for Bronchitis, we see that ‘False’, i.e., the
patient does not have bronchitis, is more probable. Suppose we discover the
patient suffers from shortness of breath, we can then set the evidence for
Dyspnoea as ‘True’ (illustrated in Figure 4). By introducing this new
evidence, we now observe a slightly different scenario, where it is more
probable for the patient to be a smoker and have bronchitis. Notice here that
variables that seem irrelevant to the evidence explanation, such as Visit to
Asia and XRay, are included in the explanation. This could lead to
overspecified hypotheses, especially in larger networks.

Figure 3: Initial MPE for Asia Bayesian network Figure 4: Updated MPE for Asia
Bayesian network

####  3.2.2 Most Relevant Explanation

To avoid an overspecified hypotheses, one approach is to trim or prune less
relevant variables from the explanation. That is, instead of finding the
complete instantiation of the target variables, a partial instantiation of the
target variables is found such that the generalised Bayes factor is maximised.
Let’s first consider the explanations obtained from the generalised Bayes
factor. Again, suppose the patient suffers from shortness of breath
(evidence). We are then interested in finding only those variables that are
relevant in explaining why the patient has shortness of breath. Table 1
contains the set of explanations obtained from the generalised Bayes factor.
For example, the last entry shows that a possible explanation for shortness of
breath is a trip to Asia and an abnormal X-ray. Thus including only 2
variables from the remaining 7 variables (excluding Dyspnoea). As mentioned,
the MRE is the explanation that maximises the generalised Bayes factor. From
Table 1 we see that having Bronchitis best explains the shortness of breath.
Notice that this explanation does not include Smoking, as opposed to the MPE
which included Smoking. Thus, although smoking is a probable cause for
shortness of breath, it is not the most relevant cause. An interesting
characteristic of the MRE is its ability to capture the explaining away
phenomenon [33].

Table 1: Explanations of GBF scores for Asia network Explanation | Generalised Bayes Factor
---|---
(Bronchitis) | 6.1391
(Smoker, Tuberculosis or Cancer) | 1.9818
(Tuberculosis or Cancer) | 1.9771
(Lung Cancer, Smoker) | 1.9723
(Lung Cancer) | 1.9678
(Smoker, Tuberculosis) | 1.8896
(Tuberculosis) | 1.8276
(Smoker, XRay) | 1.7779
(Smoker) | 1.7322
(Visit to Asia, XRay) | 1.5635

###  3.3 Explanation of decisions

Hidden or unobserved variables appear in most application fields, especially
in areas where decisions made by the end-user directly influence human lives.
For example, when first examining a patient, the health-state of the patient
is unknown. In these situations, one would typically ask two questions. The
first being given the available information, are we ready to make a decision?
and secondly, if we are not yet ready to make a decision, what additional
information do we require to make an informed decision?. To answer these
questions, the authors of [5] propose a threshold-based notion, named same-
decision probability, which provides the user with a confidence measure that
represents the probability that a certain decision will be made, had
information pertaining unknown variables been made available. Another possible
threshold-based solution to this is sensitivity analysis [30]. In sensitivity
analysis, the assessments for the conditional probabilities in the BN are
systematically changed to study the effect on the output produced by the
network. The idea is that some conditional probabilities will hardly influence
the decisions, while others will have significant impact.

####  3.3.1 Same-decision Probability

Suppose we are interested in making a decision on whether the patient is a
smoker (Smoking), which is conditioned on evidence Tuberculosis or Cancer. We
can then use the BN such that our decision pertaining to the hypothesis is
supported on the basis that the belief in the hypothesis given some evidence
exceeds a given threshold. Now, the patient may have access to information
that is unknown to us, for example, the patient recently visited Asia and
chose not to disclose this information. Therefore, we do not have access to
the true state of this variable. The true state knowledge may confirm or
contradict our decision based on the probability of smoking given some
evidence and the patient visiting Asia. If we now compare this probability
with some threshold, we have a degree of confidence in our original decision
regarding smoking and the available evidence. Had the patient disclosed his
trip to Asia, it is then unlikely that we would have made a different
decision. Hence, we can make use of the same-decision probability (SDP).
Consider now the BN given in Figure 5. Notice here the addition of three
nodes, P(Smoker=True), Decision Threshold and Decision. Where P(Smoker=True)
represents the hypothesis probability and the decision threshold is set to
55%. Suppose now we update our network such that Tuberculosis or Cancer (P) is
True – to reflect the scenario discussed above. The hypothesis probability
then increases from 50.00% to 84.35% (see Figure 6). Our decision is confirmed
given the threshold since the hypothesis probability now exceeds the given
threshold value. From Table 2, the SDP before adding evidence for the ‘True’
state is 0.00%. After adding evidence, the SDP for our decision is 83.88%,
indicating that our decision confidence is 83.88%222The SDP scenario was
constructed using the decision node functionality in Bayesialab. The decision
nodes are indicated as green (or dark grey if viewed in grey-scale)..

Figure 5: Addition of decision node in Asia network Figure 6: Updated decision for Asia network Table 2: Decision Confidence for Asia network | States | Minimum | Maximum | Mean | Standard Deviation
---|---|---|---|---|---
No evidence | False | 100.00% | 100.00% | 100.00% | 0.00%
True | 0.00% | 0.00% | 0.00% | 0.00%
Evidence | False | 0.00% | 100.00% | 16.12% | 36.77%
True | 0.00% | 100.00% | 83.88% | 36.77%

##  4 XBN in Action

The point of XBN is to explain the AI task at hand. In other words, the
question the decision-maker seeks to answer, and not the technique in
principle. Therefore, we need to be able to freely ask ‘why’ or ‘what’ and
from this select a method that would best address the AI task. In Figure 7 we
present a taxonomy of XBN. The purpose of this taxonomy is to categorise XBN
methods into four phases of BNs: The first phase involves the construction of
the BN model. Explanation in the ‘model’ phase is critical when the model is
based on expert knowledge. The second phase is reasoning, the third phase
evidence, and the fourth decision. Explanation of the model and sensitivity
analysis are illustrated in grey as it is out of scope for this paper.
Although we define the taxonomy along these phases, we do acknowledge that not
all phases are necessarily utilised by the decision-maker. For example, when
using BNs to facilitate participatory modelling [8], the main emphasis is on
explaining the model. Or, when using BNs as a classifier, the emphasis is on
explaining the decisions. In this section, we present typical questions of
interest to the decision-maker in each category of the XBN taxonomy.

Figure 7: A schematic view of XBN

###  4.1 Reasoning

Reasoning in the XBN taxonomy is concerned with the justification of a
conclusion. Returning to our Asia example, the end-user might ask the
following question,

  1. “Given the patient recently visited Asia, how likely is an abnormal chest X-Ray?”

Here, we are concerned with a single outcome: the X-Ray result. On the other
hand, the doctor may have knowledge about symptoms presented by the patient
and ask,

  1. “What is the probability of a patient being a smoker, given that he presented shortness of breath?”

We can extend this to a forensic context. Suppose a crime scene is
investigated where a severely burned body is found. The forensic analyst can
then ask,

  1. “The burn victim is found with a protruded tongue, was the victim exposed to fire before death or after?”

Consider now a financial service context where a young prospective home owner
is declined a loan. The service provider can then ask,

  1. “Did the prospective owner not qualify for the home loan because of his age?”

From these examples, we see that explanation of reasoning is used where
questions are asked in the context of single variable outcomes for diagnosis.

###  4.2 Evidence

When we are interested in the subset of variables that describes specific
scenarios, we use explanation of evidence methods. For example, in our Asia
example the doctor may ask,

  1. “Which diseases are most probable to the symptoms presented by the patient?”

or

  1. “Which diseases are most relevant to the symptoms presented by the patient?”

In a forensic context, the forensic analyst investigating a crime scene may
ask the following question,

  1. “What are the most relevant causes of death, given the victim is found with a severely burned body and protruded tongue?”

Similarly this can be applied to fraud detection. Suppose the analyst
investigates the credit card transactions of a consumer. The analyst can then
ask,

  1. “What are the most probable transaction features that contributed to the flagging of this consumer?”

Explanation of evidence can also be used to provide explanations for financial
service circumstances. For example, if a prospective home owner is turned down
for a loan, he may ask the service provider which features in his risk profile
are more relevant (contributed most) to being turned down.

###  4.3 Decisions

Explanation of decisions typically asks the following questions “Do we have
enough evidence to make a decision?”, and if not, “what additional evidence is
required to make a decision?”. For example, in our Asia example we can ask,

  1. “Do we have enough evidence on the symptoms presented to make a decision on the disease?”

or

  1. “Since we are not yet able to determine the disease, what additional information – test, underlying symptoms, comorbidities – is required to make a decision?”

Applied to forensic investigations, this can be used to answer questions
relating to crime scene investigations. The analyst may ask questions
regarding the actual evidence collected from the crime scene, i.e., if enough
evidence is collected to rule a crime as a homicide or what additional
evidence is required to rule the crime as a homicide. Should they investigate
further or is the evidence that is already collected enough to make an
informed decision?

##  5 Conclusion

The development of AI systems has faced incredible advances in recent years.
We are now exposed to these systems on a daily basis, such as product
recommendation systems used by online retailers. However, these systems are
also being implemented by medical practitioners, forensic analysts and
financial services – application areas where decisions directly influence the
lives of humans. It is because of these high-risk application areas that
progressively more interest is given to the explainability of these systems.

This paper addresses the problem of explainability in BNs. We first explored
the state of explainable AI and in particular BNs, which serves as a
foundation for our XBN framework. We then presented a taxonomy to categorise
XBN methods in order to emphasise the benefits of each method given a specific
usage of the BN model. This XBN taxonomy will serve as a guideline, which will
enable end-users to understand how and why predictions were made and will,
therefore, be able to better communicate how outcomes were obtained based on
these predictions.

The XBN taxonomy consists of explanation of reasoning, evidence and decisions.
Explanation of the model is reserved for future work, since the taxonomy
described in this paper is focused on how and why predictions were made and
not on the model-construction phase. Other future research endeavours include
the addition of more dimensions and methods to the XBN taxonomy – this
involves more statistical-based methods and the incorporation of causability
(which also addresses the quality of explanations) – as well as applying this
taxonomy to real-world applications.

## References

  * [1] Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58, 82–115 (2020). https://doi.org/10.1016/j.inffus.2019.12.012
  * [2] Brito-Sarracino, T., dos Santos, M.R., Antunes, E.F., de Andrade Santos, I.B., Kasmanas, J.C., de Leon Ferreira, A.C.P., et al.: Explainable Machine Learning for Breast Cancer Diagnosis. In: 2019 8th Brazilian Conference on Intelligent Systems (BRACIS). pp. 681–686. IEEE (2019)
  * [3] Cath, C.: Governing artificial intelligence: Ethical, legal and technical opportunities and challenges. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 376(2133) (2018). https://doi.org/10.1098/rsta.2018.0080
  * [4] Chan, H., Darwiche, A.: On the Robustness of Most Probable Explanations. Proceedings of the 22nd Conference on Uncertainty in Artificial Intelligence, UAI 2006 (06 2012)
  * [5] Choi, A., Xue, Y., Darwiche, A.: Same-decision probability: A confidence measure for threshold-based decisions. International Journal of Approximate Reasoning 53(9), 1415–1428 (2012)
  * [6] Das, A., Rad, P.: Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey. arXiv preprint arXiv:2006.11371 (2020)
  * [7] De Waal, A., Steyn, C.: Uncertainty measurements in neural network predictions for classification tasks. In: 2020 IEEE 23rd International Conference on Information Fusion (FUSION). pp. 1–7. IEEE (2020)
  * [8] Düspohl, M., Frank, S., Döll, P.: A review of Bayesian networks as a participatory modeling approach in support of sustainable environmental management. Journal of Sustainable Development 5(12) (2012). https://doi.org/10.5539/jsd.v5n12p1
  * [9] Gallego, M.J.F.: Bayesian networks inference: Advanced algorithms for triangulation and partial abduction (2005)
  * [10] Goebel, R., Chander, A., Holzinger, K., Lecue, F., Akata, Z., Stumpf, S., Kieseberg, P., Holzinger, A.: Explainable AI: the new 42? In: International cross-domain conference for machine learning and knowledge extraction. pp. 295–303. Springer (2018)
  * [11] Greene, D., Hoffmann, A.L., Stark, L.: Better, Nicer, Clearer, Fairer: A Critical Assessment of the Movement for Ethical Artificial Intelligence and Machine Learning. Proceedings of the 52nd Hawaii International Conference on System Sciences pp. 2122–2131 (2019). https://doi.org/10.24251/hicss.2019.258
  * [12] Gunning, D., Aha, D.W.: DARPA’s explainable artificial intelligence program. AI Magazine 40(2), 44–58 (2019)
  * [13] Helldin, T., Riveiro, M.: Explanation Methods for Bayesian Networks: review and application to a maritime scenario. In: Proc. of The 3rd Annual Skövde Workshop on Information Fusion Topics, SWIFT. pp. 11–16 (2009)
  * [14] Holzinger, A., Malle, B., Kieseberg, P., Roth, P.M., Müller, H., Reihs, R., Zatloukal, K.: Towards the Augmented Pathologist: Challenges of Explainable-AI in Digital Pathology. arXiv preprint arXiv:1712.06657 pp. 1–34 (2017), http://arxiv.org/abs/1712.06657
  * [15] Keppens, J.: Explaining Bayesian Belief Revision for Legal Applications. In: JURIX. pp. 63–72 (2016)
  * [16] Keppens, J.: Explainable Bayesian network query results via natural language generation systems. In: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law. pp. 42–51 (2019)
  * [17] Khedkar, S., Subramanian, V., Shinde, G., Gandhi, P.: Explainable AI in Healthcare. In: Healthcare (April 8, 2019). 2nd International Conference on Advances in Science & Technology (ICAST) (2019)
  * [18] Korb, K.B., Nicholson, A.E.: Bayesian Artificial Intelligence. CRC press (2010)
  * [19] Lacave, C., Díez, F.J.: A review of explanation methods for Bayesian networks. Knowledge Engineering Review 17(2), 107–127 (2002). https://doi.org/10.1017/S026988890200019X
  * [20] Lauritzen, S.L., Spiegelhalter, D.J.: Local computations with probabilities on graphical structures and their application to expert systems. Journal of the Royal Statistical Society: Series B (Methodological) 50(2), 157–194 (1988)
  * [21] Lawless, W.F., Mittu, R., Sofge, D., Hiatt, L.: Artificial Intelligence, Autonomy, and Human-Machine Teams: Interdependence, Context, and Explainable AI. AI Magazine 40(3), 5–13 (2019)
  * [22] Lecue, F.: On the role of knowledge graphs in explainable AI. Semantic Web 11(1), 41–51 (2020)
  * [23] Leslie, D.: Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of AI systems in the public sector (Jun 2019). https://doi.org/10.5281/zenodo.3240529
  * [24] Lipton, Z.C.: The mythos of model interpretability. Queue 16(3), 31–57 (2018)
  * [25] Martens, D., Provost, F.: Explaining data-driven document classifications. MIS Quarterly 38(1), 73–100 (2014)
  * [26] Miller, T., Weber, R., Magazzeni, D.: Proceedings of the IJCAI 2019 Workshop on Explainable AI (2019)
  * [27] Montavon, G., Binder, A., Lapuschkin, S., Samek, W., Müller, K.R.: Layer-wise relevance propagation: an overview. In: Explainable AI: interpreting, explaining and visualizing deep learning, pp. 193–209. Springer (2019)
  * [28] Samek, W., Müller, K.R.: Towards explainable artificial intelligence. In: Explainable AI: interpreting, explaining and visualizing deep learning, pp. 5–22. Springer (2019)
  * [29] Timmer, S.T., Meyer, J.J.C., Prakken, H., Renooij, S., Verheij, B.: A two-phase method for extracting explanatory arguments from Bayesian networks. International Journal of Approximate Reasoning 80, 475–494 (2017)
  * [30] Van Der Gaag, L.C., Coupé, V.M.: Sensitivity analysis for threshold decision making with bayesian belief networks. In: Congress of the Italian Association for Artificial Intelligence. pp. 37–48. Springer (1999)
  * [31] Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., Zhu, J.: Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11839 LNAI, 563–574 (2019). https://doi.org/10.1007/978-3-030-32236-6_51
  * [32] Yuan, C.: Some properties of most relevant explanation. In: ExaCt. pp. 118–126 (2009)
  * [33] Yuan, C., Lim, H., Lu, T.C.: Most relevant explanation in bayesian networks. Journal of Artificial Intelligence Research 42, 309–352 (2011). https://doi.org/10.1613/jair.3301
  * [34] Yuan, C., Liu, X., Lu, T.C., Lim, H.: Most relevant explanation: Properties, algorithms, and evaluations. Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, UAI 2009 pp. 631–638 (2009)