# Tourbillon: a Physically Plausible Neural Architecture

Mohammadamin Tavakoli
Department of Computer Science
University of California, Irvine
<EMAIL_ADDRESS>
&Peter Sadowski
Department of Information and Computer Sciences
University of Hawai‘i at Mānoa
<EMAIL_ADDRESS>
&Pierre Baldi
Department of Computer Science
University of California, Irvine
<EMAIL_ADDRESS>

###### Abstract

In a physical neural system, backpropagation is faced with a number of
obstacles including: the need for labeled data, the violation of the locality
learning principle, the need for symmetric connections, and the lack of
modularity. Tourbillon is a new architecture that addresses all these
limitations. At its core, it consists of a stack of circular autoencoders
followed by an output layer. The circular autoencoders are trained in self-
supervised mode by recirculation algorithms and the top layer in supervised
mode by stochastic gradient descent, with the option of propagating error
information through the entire stack using non-symmetric connections. While
the Tourbillon architecture is meant primarily to address physical
constraints, and not to improve current engineering applications of deep
learning, we demonstrate its viability on standard benchmark datasets
including MNIST, Fashion MNIST, and CIFAR10. We show that Tourbillon can
achieve comparable performance to models trained with backpropagation and
outperform models that are trained with other physically plausible algorithms,
such as feedback alignment.

##  1 Introduction

In physical neural systems, whether biological or neuromorphic,
backpropagation is faced with a number of problems (e.g. Bengio et al. (2015);
Nøkland (2016); Baldi and Sadowski (2016); Guerguiev et al. (2017); Baldi and
Sadowski (2018); Lillicrap et al. (2020). Of greatest relevance here are the
following problems:

  1. 1.

Labeling: the need for large quantities of labeled data to compute gradients
for supervised learning.

  2. 2.

Locality: in a physical neural system, the learning rule for adjusting the
synaptic weights must be local, i.e. depend only on variables that are
available locally, both in space and time, at the synapse.

  3. 3.

Symmetry of Connections: backpropagation requires precisely symmetric
connections between the forward and backward passes that may be hard to
realize in a physical neural system.

  4. 4.

Distances: backpropagation requires propagating signals over significant
neural distances, which could lead to signal dilution, and distorted or
unstable gradients.

  5. 5.

Developmental Modularity: backpropagation in general requires having a
complete architecture in place before training can begin, which may not be
suitable under certain developmental constraints.

A number of solutions have been suggested to try to address these problems, in
isolation or in small combinations, but no approach addresses all of them at
once well. The Tourbillon architecture proposes to address all of them at once
by combining different ideas including stacked autoencoders, random
backpropagation, and recirculation. We stress that the goal here is to address
the problems above in physical neural systems, and not to derive a new
architecture or algorithm that is practically useful for current engineering
applications of deep learning.

##  2 Stacked Autoencoders

One well known approach for dealing with the Labeling problem is to use a
stack of autoencoders (Hinton et al., 2006; Baldi, 2012), where each
autoencoder in the stack is trained in self-supervised manner to reproduce the
hidden representation produced by the previous autoencoder in the stack. The
intuition is that such a stack can learn increasingly more abstract and
powerful representations of the input data without the need for labels. Labels
can be used at the top of the architecture to train the top layer in a
supervised manner by stochastic gradient descent, with the additional option
of backpropagating through all the layers to fine-tune the entire stack
(although there is some debate in the field as whether the latter is helpful
or not (Tschannen et al., 2018)). The stacked-autoencoder approach has the
potential for addressing two other problem: Distance and Developmental
Modularity. Because backpropagation occurs inside each autoencoder in the
stack, error gradients are potentially propagated over shorted distances
confined to the size of the individual autoencoders. Furthermore, individual
autoencoders can be trained at least as soon as all the previous autoencoders
in the stack are trained and possibly earlier, avoiding the need to wait for
the entire architecture to be wired before training can begin.

The stack of autoencoders approach however does not address the problems of
Locality and Symmetry of the forward weights. Each autoencoder is deep by
definition, in the sense that it has at least one hidden layer, and therefore
it requires applying the backpropagation algorithm across at least two
adaptive layers of weights. This in turn requires having a deep learning
channel (i.e. "wires") for transmitting the error signal and making it local
to the hidden layer(s). Furthermore, the weights on these wires have to be
symmetric to the forward weights in order to implement standard
backpropagation.

##  3 Random Backpropagation

Feedback alignment (FA) or random backpropagation (RBP) refers to a family of
algorithms (Lillicrap et al., 2016; Baldi et al., 2018) that address the
Symmetry problem by using non-symmetric random weights in the backward pass.
For a forward weight $w_{ij}$, the backpropagation learning equation can be
written as:

$\Delta w_{ij}=\eta B_{i}^{post}O_{j}^{pre}$ (1)

where $\eta$ denotes the learning rate, $B_{i}^{post}$ denotes the
postsynaptic backpropagated error, and $O_{j}^{pre}$ denotes the presynaptic
activity. In contrast, RBP can be expressed as:

$\Delta w_{ij}=\eta R_{i}^{post}O_{j}^{pre}$ (2)

where $R_{i}^{post}$ denotes the postsynaptic, randomly backpropagated, error.
Many flavors of RBP can be obtained by using different notions of randomness
to select the backward weights, by including various skipped backward
connections, by making the backward weights adaptive, and so forth (Baldi et
al., 2018, 2017). While RBP addresses the Symmetry problem, by itself it does
not address the other problems. Furthermore, while RBP simulations have
succeeded on the MNIST and CIFAR benchmark data sets, it has been noted that
RBP algorithms do not work well with convolutional layers (Bartunov et al.,
2018; Moskovitz et al., 2018; Refinetti et al., 2020). A few methods have been
proposed to address this apparent weakness of RBP algorithms, however, most of
them are not biologically plausible. For instance, Liao et al. (2016) proposes
to use uniform sign-concordant feedback (uSF), where each forward matrix $W$
in the forward pass is replaced by the matrix $sign(W)$ in the backward pass.
They also pointed that normalization methods such as batch normalization and
batch Manhattan can improve the performance of RBP when dealing with
convolution layers. Moskovitz et al. (2018) modifies the uSF method by putting
additional constraints on the backward weight matrices. In all these
approaches, the backward weight matrices must have some knowledge about the
forward matrices.

##  4 Circular Autoencoders and Recirculation Algorithms

In a standard feed forward autoencoder, the data itself provides the targets
(self-supervised learning). The data and hence the targets are available in
the input layer. However, they are not available in the output layer, in the
sense that they are not physically local to the output layer. This problem is
addressed by circular autoencoders where the output layer is physically equal
(or physically adjacent) to the input layer (Figure 1). With the circular
layout, targets and errors can be computed at the level of the input/output
layer. Standard backpropagation, or even standard RBP, of these targets would
require a channel (wires) running backward from the output layer to the hidden
layer. However, because of the circular layout, one may suspect that it may be
possible to use the forward connections to propagate target and error
information during learning. This is the fundamental idea behind
recirculation, a family of algorithms for training circular autoencoders that
do not require backward connections (Hinton and McClelland, 1988; O’Reilly,
1996; Baldi and Sadowski, 2018).

Figure 1: Circular autoencoder. The output layer (layer $k$) is equal to the
input layer (layer 0). For a given input, information can be propagated
(recirculated) multiple times through the autoencoder loop. The first pass is
indexed by 0, the last one by $T$. For instance, $O(0)^{k-1}$ denotes the
output of layer $k-1$ during the first circulation.

Consider a circular autoencoder with layers numbered from $0$ to $k$, where
$0$ and $k$ correspond to the input layer and use the index $t$ to denote
different feed forward passes through the autoencoder, with the first pass
indexed by $t=0$. After the first pass, one can locally compute the error
$T-O^{k}(0)$, where $T$ is the target, i.e. the original data in the case of
an autoencoder. This error could be used to train the top layer of the
circular autoencoder by gradient descent, and then train the other layers by
using a form of RBP where the error signal is obtained by propagating the
error $T-O^{k}(0)$ using the forward weights of the circular autoencoder. This
however requires propagating two different kinds of signals, activities, and
errors, through the circular autoencoder. Thus rather than recirculating the
error, a more uniform approach can be obtained by recirculating activities. If
$O^{i}(t)$ denotes the output of layer $i$ during the forward pass indexed by
$t$, the main idea behind the recirculation family of algorithms is to use
$O^{i}(t)$ as the target for the output $O^{i}(t^{\prime})$ taken at a later
time $t^{\prime}$. The intuition behind this is that the data may become
increasingly corrupted as it is being recycled. Several different algorithms
can be obtained, by varying, for instance, the choices of $t$ and $t^{\prime}$
and the manner in which outputs from different passes are combined (e.g.
convex combinations) (Baldi and Sadowski, 2018). Here we will use the simplest
form, which often works the best, based only on the plain activities of the
first two passes corresponding to $t=0$ and $t=1$. Thus, for any weight
$w_{ij}$ in the circular autoencoder, the corresponding learning-by-
recirculation equation becomes:

$\Delta w_{ij}=\eta[O_{i}(0)-O_{i}(1)]^{post}[O_{j}(0)]^{pre}$ (3)

This rule has a Hebbian-product form between a post-synaptic term and a pre-
synaptic term, and is similar to backpropagation, except that the postsynaptic
backpropagated error is replaced by the postsynaptic recirculation error
$[O_{i}(0)-O_{i}(1)]^{post}$. This error term is local in space and is assumed
to be also local in time. This requires the assumption that two consecutive
passes through the circular autoencoder fall within the time window that
defines locality in a particular physical system. For the input layer, the
vector $O^{0}(0)$ corresponds to the input data, hence also to the targets in
the case of an autoencoder. Thus the recirculation learning equation for the
top layer of weights is exactly equal to backpropagation (stochastic gradient
descent ). Having stochastic gradient descent applied to the top layer is also
essential for RBP to work. The recirculation error $[O_{i}(0)-O_{i}(1)]$ can
also be thought of as a derivative of the activity. Thus, in short,
recirculation learning rules rely on the product of the pre-synaptic activity
by some measure of change in the post-synaptic activity, which is used to
communicate error information. Although in this work we are not using spiking
neurons, such learning rules are closely related to the concept of spike time-
dependent synaptic plasticity (STDP). STDP Hebbian or anti-Hebbian learning
rules have been proposed using the temporal derivative of the activity of the
post-synaptic neuron (Xie and Seung, 2000) to encode error derivatives. In the
Appendix, some of the simulations also explore a slightly different anti-
Hebbian learning rule:

$\Delta w_{ij}=-\eta[O_{i}(0)-O_{i}(1)]^{post}[O_{j}(0)-O_{j}(1)]^{pre}$ (4)

where the pre- and post-synaptic terms are more similar in form. This rule is
applied to all the layers, but the top adaptive layer where the standard SGD
rule is used.

##  5 Tourbillon

We propose the Tourbillon architecture as a stack of circular autoencoders
followed by a partially or fully-connected layer between the hidden
representation of the top circular autoencoder and the output layer. Each
circular autoencoder is broken down into two components, the encoder, and the
decoder components. The hidden layer that is shared by the encoding and
decoding components of a circular autoencoder is called the hinge hidden
layer. In the stack, the hinge hidden layer for the circular autoencoder at
level $i$, becomes the input layer for the next circular autoencoder at level
$i+1$ in the stack.

The circular autoencoders are trained by recirculation (self supervised
manner) and the top layer can be trained in a supervised manner by stochastic
gradient descent, with the option of continuing the error propagation through
the entire stack, as a form of RBP using non-symmetric connections. In a
physical neural system, this requires that the targets be locally available at
the final output layer. This architecture addresses all five problems
mentioned in the introduction. Most of the training is unsupervised, all the
learning rules are local in space and time, the forward and backward
connections are not symmetric, information is propagated only over relatively
short distances, the development is modular, and also convolutions can be
incorporated into this framework. To prove its viability, we test this
architecture on several benchmark problems in the Experiments section.

Figure 2: Tourbillon architecture with a stack of circular autoencoders
trained by recirculation. Some layers are drawn with multiple sheets to
signify the possible presence of convolutions.

Although we simulate Tourbillon in its most basic version, there are a number
of possible variations on the basic Tourbillon idea and training algorithms.
In particular, in the basic version, the stack of autoencoders is trained
sequentially bottom-up starting from the initial input layer. It is possible
to consider more asynchronous modes, where for instance any circular
autoencoder in the stack is being trained as soon as it receives an input from
a lower autoencoder.

###  5.1 Fine-Tuning Using the Decoders’ Connections to Create a Deep Learning
Channel

Once the stack of circular autoencoders is in place, possibly after a
developmental phase, the connections of their decoder components provide a
deep learning channel, i.e. a physical channel through which error obtained at
the top of the architecture could be communicated to the lower layers. This
channel, whose weights are initially learned by recirculations, can be used to
fine-tune the entire Tourbillon architecture, using a form of random
backpropagation, since there is no imposed symmetry between the weights of the
encoder and decoder networks of each circular autoencoder. This produces a
kind of RBP error signal for the hinge hidden layer of each circular
autoencoder. If the encoder component of circular autoencoders has multiple
layers, different possibilities can be considered for the fine-tuning phase
when the error signal arrives at the hinge. In all the simulations, we use
circular autoencoders with a single hidden layer, and thus the RBP error that
arrives at the hinge is local to the corresponding encoder weights and these
can be fine-tuned by standard RBP (Equation 2).

###  5.2 Compositionality of the Tourbillon Architecture

Due to the modularity of the Tourbillon architecture, it can be composed in
many different ways. In particular, it can be utilized to build a physically
plausible twin architecture for every physically non-plausible neural network
trained with backpropagation or random backpropagation. Although the
physically plausible architecture does not outperform its twin, this idea
opens the door for further studies of physically plausible approaches. Here we
explain the process of building the twin architecture through two examples:
(1) Tourbillon U-Net; and (2) Tourbillon recursion. We also provide the
general algorithm of building the physically plausible twin architecture in
Algorithm 1.

The first example is the “tourbillonization” of a U-Net (Ronneberger et al.,
2015) architecture. The U-Net architecture is basically a feed forward
autoencoder with a multi-layer encoder, and a multi-layer decoder, as well as
additional skip connections (which we ignore for simplicity). This
architecture was originally developed for image segmentation problems, where
the segmented images are used as the targets. Each layer of a U-Net
architecture can be replaced by a circular autoencoder to build a Tourbillon
version of U-Net. The only non-local aspect of the Tourbillon U-Net is the
need for targets in the final output layer. This could be addressed by turning
U-Net itself into a circular autoencoder, which leads to the second example
where the Tourbillon approach can be used recursively, by replacing each
hidden layer of a circular autoencoder, with a circular autoencoder, a
recursion that in principle could be iterated several times. In the later
sections we experiment the tourbillonizaiton of U-Net, however, how a
Tourbillon of Tourbillons architectures can be trained efficiently is left out
for future work.

Input: A network $M$ with $m$ sequential layers $L=[L_{1},...,L_{m}]$, $data$:
training data, and recirculation ($c$) function which trains a circular
autoencoder $c$.

Output: The physically plausible twin $M^{\prime}$.

initialization; $L^{\prime}$: an empty list

for  _$i\leftarrow 1$ to $m$_ do

if  _$L_{i}$ has learnable parameters_ then

$l$ = train_circular_ae(_$[L1,...,L_{i}]$_)

$L^{\prime}$.append($l$)

else

$L^{\prime}$.append($L_{i}$)

end if

end for

$M^{\prime}$ = $[data]$

for  _$i\leftarrow 1$ to $m$_ do

$M^{\prime}$ = $L^{\prime}_{i}$($M^{\prime}$)

end for

Function  _train_circular_ae(_$[L_{1},...,L_{i}]$_)_:

input = $L_{i-1}(...(L_{1}(data)))$

circular_ae = $L_{i}^{-1}(L_{i}(\text{input}))$

recirculation (circular_ae)

return $L_{i}$

Algorithm 1 Tourbillonization

##  6 Experiments

In the experiments, we train circular autoencoders and Tourbillon
architectures capable of supporting feed forward fully-connected layers, 2D
convolutions, 2D max pooling, 2D up sampling, reshaping, and five different
non-linear activation functions.

###  6.1 Data

In the experiments, we use three well-known benchmark datasets: (1) MNIST,
comprising 70,000 gray-scale images of handwritten digits of size $28\times
28$ with each pixel normalized to the $[0,1]$ range. We used the normal 60000
and 10000 train-test split; (2) Fashion MNIST comprising 70,000 gray-scale
images of fashion accessories of size $28\times 28$ with each pixel normalized
to the $[0,1]$ range. We used the normal 60000 and 10000 train-test split; and
(3) CIFAR-10 comprising 60,000 images of size $32\times 32$ with three RGB
channels corresponding to 10 object classes with each pixel normalized to the
$[0,1]$ range. We used the normal 50000 and 10000 train-test split. For the
following experiments, the models are trained using a single NVidia Titan X
GPU.

###  6.2 Tourbillon Building Blocks

Since the building blocks of the Tourbillon architecture are circular
autoencoders, we train compressive circular autoencoders using the
recirculation algorithm (Equation 3) and compare the mean-squared
reconstruction loss (similar results are obtained with the relative entropy
loss) with that of the corresponding architecture trained with both
backpropagation and random backpropagation. For experiments with MNIST and
Fashion MNIST, the circular autoencoder consists of two feed forward fully-
connected layers, where the middle layer is of size 256. For experiments with
CIFAR10, the circular autoencoder consists of two 2D convolution layers with
kernels of size $(5\times 5)$ to construct a hidden representation of size
$(32\times 32\times 6)$ in the hinge layer. Both convolutional layers use
strides of size $(1\times 1)$ with zero padding to preserve the size of the
input. Figure 3 shows the training and test error curves for the MNIST and
Fashion MNIST datasets. Figure 8, first row, shows the training and test error
curves for the convolutional circular autoencoder trained using the CIFAR10
dataset. In all these experiments, recirculation leads to reconstruction
errors that are comparable, and possibly even better, than the corresponding
errors obtained using backpropagation and random backpropagation. In all these
experiments, the models are trained with a mini-batch of size 64 to minimize
the reconstruction error. Across all three models, all hidden layers use a
tanh activation function and the final layer uses a logistic activation
function (since the pixels are normalized to $[0,1]$). During our experiments,
we observed that the presence of non-linear activation functions in
intermediate layers is essential for learning, especially for the circular
autoencoder which is consistent with the results provided in Baldi et al.
(2016). For all models, a grid search was used to optimize the
hyperparameters. As a result, models trained with backpropagation and random
backpropagation across all three datasets use a starting learning rate of 0.01
for all the layers, with a momentum term of 0.8. On the other hand, for the
circular autoencoder traioned using MNIST and Fashion MNISt datasets, the grid
search resulted in starting learning rates of 0.01 and 0.001 for the first and
second layers respectively. For the circular autoencoder trained using CIFAR10
dataset, the grid search resulted in starting learning rate of 0.001 and
0.0001 for the first and second convolutional layers respectively. All these
learning rates are also decayed by a rate of $0.9$.

Figure 3: Comparison of the training and test mean-squared reconstruction loss
of three equivalent autoencoders trained using the MNIST and Fashion MNIST
datasets. The first row shows the performance of the models for the MNIST
dataset, and the second row shows the performance for the Fashion MNIST
dataset. REC corresponds to a circular autoencoder trained with recirculation,
FA corresponds to the same architecture trained with feedback alignment, and
BP corresponds to the same architecture trained with backpropagation.

###  6.3 Tourbillon

Here we stack trained circular autoencoders to build a Tourbillon
architecture. The input data to each of the circular autoencoders correspond
to the hidden representations produced in the hinge layer of the previous
circular autoencoder in the stack. We perform three classification experiments
using MNIST, Fashion MNIST, and CIFAR10 and compare the performance of
Tourbillon with two neural networks with an equivalent architecture trained
with backpropagation and random backpropagation. As shown in Figures 4 and the
second row of Figure 8, Tourbillon outperforms random backpropagation and is
comparable to standard backpropagation, especially when using feed forward
fully-connected architectures. For MNIST and Fashion MNIST, we use the same
architecture where the Tourbillon consists of two trainable circular
autoencoders, the first tasked with compressing the data size from 784 to 256,
and the second tasked with compressing the hinge hidden representation from
256 to 64. This is followed by a fully-connected layer with a softmax
activation function to perform the final classification. Each of the circular
autoencoders is trained by recirculation with the same hyperparameters used in
Section 6.2. The top layer is initialized with Glorot initialization (Glorot
and Bengio, 2010) and, after a hyperparameter grid search, its weights are
optimized with a learning rate of 0.01 and a momentum of 0.8. Similarly, for
the CIFAR10 dataset, the Tourbillon architecture comprises two trainable
circular autoencoders. The two circular autoencoders are similar to the
circular autoencoder in Section 6.2 for the CIFAR10 dataset. The hinge hidden
representation of the first circular autoencoder is of size $(32\times
32\times 6)$. The hinge hidden representation of the second circular
autoencoder is of size $(16,16,6)$. The encoder component of these two
circular autoencoders is provided by a 2D convolutional layer with tanh
activation functions and the decoder component consists of another 2D
convolutional layer with a logistic activation function. We use the same
hyperparameters as we used in Section 6.2 for training a circular autoencoder
using the CIFAR10 dataset.

Figure 4: Comparison of the training and test accuracy of three equivalent,
fully-connected, models trained using the MNIST and Fashion MNIST datasets.
Tourbillon corresponds to the Tourbillon approach. FA corresponds to the same
architecture trained with feedback alignment, and BP corresponds to same
architecture trained with backpropagation.

###  6.4 Tourbillonization

As described in Section 5.2, following the Algorithm 1, we build the
physically plausible twin architecture for a U-Net autoencoder architecture
that addresses all the problems mentioned in the Introduction. In addition, we
also “tourbillonize” a feed forward fully-connected network architecture.
Results obtained on MNIST and Fashion MNIST are very similar, therefore here
we report the results obtained on MNIST and CIFAR10. The U-Net architecture
for the MNIST dataset comprises a two-layer encoder and a two-layer decoder.
The encoder layers compress the data from 784 to 128, and then from 128 to 64.
The decoder layers expand the data from 64 to 128, and then from 128 to 784.
The U-Net architecture for the CIFAR10 dataset comprises two 2D convolutional
layers with kernels of size $(5\times 5)$. For both the MNIST and CIFAR10
U-Nets we use the same mini-batch size, learning rate, decay rate, momentum,
and initializations as described in Section 6.2. After building the twin
Tourbillon architectures and training their circular autoencoders by
recirculation, we apply fine-tuning as described in Section 5.1 to perform
random backpropagation using the channel provided by the decoders’
connections. Empirically, we found that it is necessary to have different
learning rate schedules for different layers during the fine-tuning phase to
ensure a form of asynchronous learning among layers(see also Refinetti et al.
(2020)). A summary of the results comparing the various approaches is given in
Table 1 (first row), and shows that the performance of the Tourbillon twin
approach is comparable to backpropagation.

For the feed forward architecture, we use a three-layer network with 256, 64,
and 10 hidden units. For this architecture and its Tourbillon twin, we use the
same hyperparameters as in Section 6.2. During the fine-tuning of the
Tourbillon, the learning rate schedules of each layer are similar to the rates
used for the U-Net Tourbillon. A summary of the results comparing the various
approaches is given in Table 1 (second row), and shows again that the
performance of the Tourbillon twin approach is comparable to backpropagation.

Table 1: Performance comparison of networks trained with backpropagation vs their physically plausible Tourbillon twin. For the U-Net experiment, the numbers represent the mean-squared reconstruction error. For the feed forward fully-connected architecture, the numbers represent the classification error on the test dataset. For every reported number, the standard deviation is also shown in the parenthesis. For the sake of brevity, FC stands for fully-connected network. Architecture |  | MNIST | CIFAR10
---|---|---|---
|  | Train | Test | Train | Test
U-Net | BP | 0.0031 (5.8 e-05) | 0.0031 (5.8 e-05) | 0.0024 (5.8 e-05) | 0.0026 (e-04)
Twin | 0.0111 (e-04) | 0.0113 (2e-04) | 0.0733 (3.2e-04) | 0.0783 (3.2e-04)
FC | BP | 1.04 (0.20) | 1.18 (0.20) | - | -
Twin | 5.21 (0.40) | 5.82 (0.40) | - | -

##  7 Discussion and Conclusion

In a physical neural system, implementation of backpropagation is faced with a
number of significant challenges described in the Introduction. Here we have
presented a systematic approach we call Tourbillon for deriving physically
plausible architectures addressing most if not all of them. At its core,
Tourbillon relies on a stack of circular autoencoders trained by
recirculation. Tourbillon architectures are modular and can be trained in
large part in self-supervised mode using learning rules that are local in both
space and time, without the need for weight transport.

The name Tourbillon captures the turbulent topology of the architecture.
Moreover, in horology, a tourbillon is an addition to the mechanics of a watch
escapement to increase its accuracy. While we do not claim to have increased
accuracy, we have shown that the Tourbillon approach achieves results that are
comparable to backpropagation, at least on MNIST, Fashion MNIST, and CIFAR10.

In the development of Tourbillon, several issues have been identified that
will require additional research. The first one is to study whether local
learning algorithms can be developed to improve the fine-tuning phase of
Tourbillon architectures where the encoder components of the circular
autoencoders have multiple hidden layers. Something along the lines of
recirculating the RBP error may be a possibility.

The second one is to develop local learning algorithms for the recursive
Tourbillon approach, starting with Tourbillions of Tourbillons. In particular,
one would like to know whether the fine-tuning phase is necessary and how to
carry it, perhaps using a fast form of recirculation in the smaller circular
autoencoders and a slower form of recirculation in the larger circular
autoencoder.

The third one is the issue of convolutions that were only incrementally
addressed here by showing that Tourbillon with convolutions does better than
RBP, but lags behind BP. When convolutions are used, neurons in a convolution
layer must have identical incoming weights, which may not be easy to realize
in a physical neural system. The most plausible approach to address the
convolution problem in a physical system is to first initialize the weights in
similar, but not identical, fashion across the entire convolutional layer,
using for instance normal weights with small standard deviation. This ensures
that all the weights are similar, without enforcing them to be identical.
Second, during training, each convolution neuron in the layer learns
independently of the other convolution neurons, without enforcing any kind of
rigid weight sharing. What is essential, however, is to ensure that all the
convolution neurons in the layer see exactly, or approximately, the same
training data in aggregate. This is easily achieved through data augmentation
by translating each training image in all possible directions, something that
may happen automatically in the real world due to moving objects, or head/eye
motions. With this data augmentation, the weights of the convolution neurons
remain similar throughout training, since they are trained on the same data,
without any exact weight sharing (Ott et al., 2020). This approach ought to be
tried in Tourbillon.

Tourbillon provides a framework for investigating these and other issues and
broadens the horizon of ongoing research into physically plausible deep
learning.

## References

  * Bengio et al. [2015] Yoshua Bengio, Dong-Hyun Lee, Jorg Bornschein, Thomas Mesnard, and Zhouhan Lin.  Towards biologically plausible deep learning.  _arXiv preprint arXiv:1502.04156_ , 2015.
  * Nøkland [2016] Arild Nøkland.  Direct feedback alignment provides learning in deep neural networks.  _arXiv preprint arXiv:1609.01596_ , 2016.
  * Baldi and Sadowski [2016] Pierre Baldi and Peter Sadowski.  A theory of local learning, the learning channel, and the optimality of backpropagation.  _Neural Networks_ , 83:61–74, 2016.
  * Guerguiev et al. [2017] Jordan Guerguiev, Timothy P Lillicrap, and Blake A Richards.  Towards deep learning with segregated dendrites.  _ELife_ , 6:e22901, 2017.
  * Baldi and Sadowski [2018] Pierre Baldi and Peter Sadowski.  Learning in the machine: Recirculation is random backpropagation.  _Neural Networks_ , 108:479–494, 2018.
  * Lillicrap et al. [2020] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton.  Backpropagation and the brain.  _Nature Reviews Neuroscience_ , 21(6):335–346, 2020.
  * Hinton et al. [2006] G.E. Hinton, S. Osindero, and Y.W. Teh.  A fast learning algorithm for deep belief nets.  _Neural Computation_ , 18(7):1527–1554, 2006\.
  * Baldi [2012] Pierre Baldi.  Autoencoders, unsupervised learning, and deep architectures.  In _Proceedings of ICML workshop on unsupervised and transfer learning_ , pages 37–49. JMLR Workshop and Conference Proceedings, 2012.
  * Tschannen et al. [2018] Michael Tschannen, Olivier Bachem, and Mario Lucic.  Recent advances in autoencoder-based representation learning.  _arXiv preprint arXiv:1812.05069_ , 2018.
  * Lillicrap et al. [2016] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman.  Random synaptic feedback weights support error backpropagation for deep learning.  _Nature communications_ , 7(1):1–10, 2016.
  * Baldi et al. [2018] P. Baldi, Z. Lu, and P. Sadowski.  Learning in the machine: Random backpropagation and the deep learning channel.  _Artificial Intelligence_ , 260:1–35, 2018.  Also: arXiv:1612.02734.
  * Baldi et al. [2017] P. Baldi, Z. Lu, and P. Sadowski.  Learning in the machine: the symmetries of the deep learning channel.  _Neural Networks_ , 95:110–133, 2017.
  * Bartunov et al. [2018] Sergey Bartunov, Adam Santoro, Blake A Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap.  Assessing the scalability of biologically-motivated deep learning algorithms and architectures.  _arXiv preprint arXiv:1807.04587_ , 2018.
  * Moskovitz et al. [2018] Theodore H Moskovitz, Ashok Litwin-Kumar, and LF Abbott.  Feedback alignment in deep convolutional networks.  _arXiv preprint arXiv:1812.06488_ , 2018.
  * Refinetti et al. [2020] Maria Refinetti, Stéphane d’Ascoli, Ruben Ohana, and Sebastian Goldt.  The dynamics of learning with feedback alignment.  _arXiv preprint arXiv:2011.12428_ , 2020.
  * Liao et al. [2016] Qianli Liao, Joel Leibo, and Tomaso Poggio.  How important is weight symmetry in backpropagation?  In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ , pages 1837–1844, 2016.  URL http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12325.
  * Hinton and McClelland [1988] Geoffrey E Hinton and James L McClelland.  Learning representations by recirculation.  In _Neural information processing systems_ , pages 358–366. New York: American Institute of Physics, 1988.
  * O’Reilly [1996] Randall C O’Reilly.  Biologically plausible error-driven learning using local activation differences: The generalized recirculation algorithm.  _Neural computation_ , 8(5):895–938, 1996.
  * Xie and Seung [2000] Xiaohui Xie and H. Sebastian Seung.  Spike-based learning rules and stabilization of persistent neural activity.  In S. A. Solla, T. K. Leen, and K. Müller, editors, _Advances in Neural Information Processing Systems 12_ , pages 199–208. MIT Press, 2000.  URL http://papers.nips.cc/paper/1658-spike-based-learning-rules-and-stabilization-of-persistent-neural-activity.pdf.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pages 234–241. Springer, 2015.
  * Baldi et al. [2016] Pierre Baldi, Peter Sadowski, and Zhiqin Lu.  Learning in the machine: Random backpropagation and the learning channel.  _arXiv preprint arXiv:1612.02734_ , 2016.
  * Glorot and Bengio [2010] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS10). Society for Artificial Intelligence and Statistics_ , 2010.
  * Ott et al. [2020] Jordan Ott, Erik Linstead, Nicholas LaHaye, and Pierre Baldi.  Learning in the machine: To share or not to share?  _Neural Networks_ , 126:235–249, 2020.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.

##  8 Appendix

In this section, we first provide more explanation on the new learning rule
introduced in Equation 4. Then we include evaluation plots corresponding to:
(1) The performance of the circular autoencoder and Tourbillon models trained
using the CIFAR10 dataset and their comparison to models trained using random
backpropagation and backpropagation; (2) The representation power of
Tourbillon models by visualizing the reconstructed images; and (3) The process
of tourbillonization.

Here we rewrite the symmetric learning rule we introduced earlier:

$\Delta w_{ij}=-\eta[O_{i}(0)-O_{i}(1)]^{post}[O_{j}(0)-O_{j}(1)]^{pre}$ (5)

This learning rule has symmetric terms for the pre- and post-synaptic
activation differences. Although the loss trajectory is smoother than when
using the main recirculation rule (Figure 5), the algorithm becomes trapped
into a mode-collapse state where the reconstructed images are the mean of the
entire dataset (Figure 6). Further studies of this learning rule and its mode
collapse are left for future work.

Figure 5: The different behavior of the two learning rules introduced in
Section 4. The symmetric learning rule corresponds to Equation 4 and the REC
learning rule corresponds to Equation 3. While both demonstrate similar
performance in training models using MNIST (first row) and Fashion MNIST
(second row), the symmetric learning rule has smoother trajectories. Figure 6:
Examples of reconstructed images from the MNIST and Fashion MINST datasets
using the circular autoencoder trained with the symmetric learning rule. There
is a mode-collapse effect and the model reconstructs the mean of the data.

The first row of Figure 8 represents the training and test performance of a
circular autoencoder trained by recirculation using the CIFAR10 dataset and
its comparison to an autoencoder with similar architecture trained with random
backpropagation and backpropagation.

Figure 9 shows the reconstructed images using a stack of two circular
autoencoders. We also use the tSNE [Van der Maaten and Hinton, 2008]
dimensionality reduction method in Figure 7 to visualize the output of
circular autoencoders used in Section 6.3. These two figures show the
representational power of the Tourbillon building blocks.

Figure 7: From left to right, the plots correspond to the 2D visualization of
2000 random samples from the MNIST, Fashion MNIST, and CIFAR10 datasets.
Figure 8: First row: Comparison of the mean-squared reconstruction loss
between circular autoencoder with recirculation, feedback alignment, and
backpropagation. Second row: Comparison of the training and test accuracy for
Tourbillon, feedback alignment, and backpropagation using the CIFAR10 dataset.
Figure 9: Examples of reconstructed images from the MNIST and the Fashion
MNIST datasets using Tourbillon. Figure 10: Schematic visualization of the
“tourbillonization” process. Left: A given neural network model where the blue
arrows represent the weights that are trained by backpropagation. Middle: The
Tourbillon equivalent of the given neural network. Each of the intermediate
layers of the given network are replaced with a circular autoencoder that has
a one-layer encoder and a one-layer decoder. Both red and green arrows show
the weights that are trained by recirculation. Right: The final tourbillonized
model where the forward weight (green arrows) can be fine-tuned using the
decoder’s channel (red arrows) by random backpropagation.


11institutetext:  Rensselaer Polytechnic Institute, Troy, NY, USA
11email: {charis, senevo}<EMAIL_ADDRESS>
22institutetext: IBM Research, Cambrdige, MA, USA
22email: {daniel_gruen, morgan.foreman<EMAIL_ADDRESS>
# Explanation Ontology: A Model of Explanations for User-Centered AI

Shruthi Chari 11 0000-0003-2946-7870 Oshani Seneviratne 11 0000-0001-8518-917X
Daniel M. Gruen 22 0000-0003-0155-3777 Morgan A. Foreman 22
0000-0002-2739-5853 Amar K. Das 22 0000-0003-3556-0844 Deborah L. McGuinness
11 0000-0001-7037-4567

###### Abstract

Explainability has been a goal for Artificial Intelligence (AI) systems since
their conception, with the need for explainability growing as more complex AI
models are increasingly used in critical, high-stakes settings such as
healthcare. Explanations have often added to an AI system in a non-principled,
post-hoc manner. With greater adoption of these systems and emphasis on user-
centric explainability, there is a need for a structured representation that
treats explainability as a primary consideration, mapping end user needs to
specific explanation types and the system’s AI capabilities. We design an
explanation ontology to model both the role of explanations, accounting for
the system and user attributes in the process, and the range of different
literature-derived explanation types. We indicate how the ontology can support
user requirements for explanations in the domain of healthcare. We evaluate
our ontology with a set of competency questions geared towards a system
designer who might use our ontology to decide which explanation types to
include, given a combination of users’ needs and a system’s capabilities, both
in system design settings and in real-time operations. Through the use of this
ontology, system designers will be able to make informed choices on which
explanations AI systems can and should provide.

Resource: https://tetherless-world.github.io/explanation-ontology

###### Keywords:

Explainable AI Explanation Ontology Modeling of Explanations and Explanation
Types Supporting Explainable AI in Clinical Decision Making and Decision
Support

##  1 Introduction

Explainability has been a key focus area of Artificial Intelligence (AI)
research, from expert systems, cognitive assistants, the Semantic Web, and
more recently, in the machine learning (ML) domain. In our recent work [5], we
show that advances in explainability have been coupled with advancements in
the sub-fields of AI. For example, explanations in second-generation expert
systems typically address What, Why, and How questions [23, 6]. With ML
methods, explainability has focused on interpreting the functioning of black-
box models, such as identifying the input features that are associated the
most with different outputs [15, 13]. However, while explanations of the
“simplified approximations of complex decision-making functions” [17] are
important, they do not account for “specific context and background knowledge”
[18] that users might possess, and hence, are often better suited for experts
or debugging purposes. Several researchers have written about this shortcoming
[17, 13], and the fallacy of associating explainability to be solely about
model transparency and interpretability [9]. Given this shift in focus of
explainable AI, due to the adoption of AI in critical and user-facing fields
such as healthcare and finance, researchers are drawing from adjacent
“explanation science” fields to make explainable AI more usable [16]. The term
“explanation sciences” was introduced by Mittlestadt et al. to collectively
refer to the fields of “law, cognitive science, philosophy, and the social
sciences.” [17]

Recent review papers [17, 3] point out that explainability is diverse, serving
and addressing different purposes, with user-specific questions and goals.
Doshi et al. [7] propose a set of questions beyond the What, Why, How
questions that need to be addressed by explanations: “What were the main
factors in a decision?”, “Would changing a certain factor have changed the
decision?” and “Why did two similar-looking cases get different decisions or
vice versa?” Other researchers, like Wang et al. [26], in their conceptual
framework linking human reasoning methods to explanations generated by
systems, support various explainability features, such as different
“intelligibility queries.” Lim and Dey observed these intelligibility queries
during their user study where they were studying for mechanisms to improve the
system’s intelligibility (comprehensibility) by looking to gain user’s trust
and seeking to avoid situations that could lead to a “mismatch between user
expectation and system behavior” [12]. Support for such targeted provisions
for explanations would enable the creation of “explainable knowledge-enabled
systems", a class of systems we defined in prior work [5] as, “AI systems that
include a representation of the domain knowledge in the field of application,
have mechanisms to incorporate the users’ context, are interpretable, and host
explanation facilities that generate user-comprehensible, context-aware, and
provenance-enabled explanations of the mechanistic functioning of the AI
system and the knowledge used.”

Currently, a single class of AI models, with their specific focus on
particular problems that tap into specific knowledge sources, cannot wholly
address these broad and diverse questions. The ability to address a range of
user questions points to the need for providing explanations as a service via
a framework that interacts with multiple AI models with varied strengths. To
achieve this flexibility in addressing a wide range of user questions, we see
a gap in semantic support for the generation of explanations that would allow
for explanations to be a core component of AI systems to meet users’
requirements. We believe an ontology, a machine-readable implementation, can
help system designers, and eventually a service, to identify and include
methods to generate a variety of explanations that suit users’ requirements
and questions. While there have been a few efforts to establish connections
between explanation types and the mechanisms that are capable of generating
them [26, 1], these efforts are either not made available in machine-readable
formats or not represented semantically. The lack of a semantic representation
that would offer support for modeling explanations makes it difficult for a
system designer to utilize their gathered user requirements as a means to
build in explanation facilities into their systems.

We first present related work on taxonomies of explanation (Section 2). We
then introduce the design of an explanation ontology (Section 3) that treats
explanations as a primary consideration of AI system design and that can
assist system designers to capture and structure the various components
necessary to enable the generation of user-centric explanations
computationally. These components and the attributes of explanations are
gathered from our literature review of AI [5, 4], associated “explanation
sciences” domains including social sciences and philosophy. We use the results
of a previously conducted user-centered design study, which used a prototype
decision-support system for diabetes management, to demonstrate the usage of
our ontology for the design and implementation of an AI system (Section 4).
Finally, we evaluate our ontology’s competency in assisting system designers
by our ontology’s ability to support answering a set of questions aimed at
helping designers build “explainable, knowledge-enabled,” AI systems (Section
5).

##  2 Related Work

While there have been several taxonomies proposed within explainable AI, we
limit our review to taxonomies that are closest to our focus, in that they
catalog AI models and the different types of explanations they achieve [2, 1],
or ones that capture user-centric aspects of explainability [26]. Recently,
Arya et al. [2] developed a taxonomy for explainability, organizing ML methods
and techniques that generate different levels of explanations, including post-
hoc (contain explanations about the results or model functioning), local
(about a single prediction), and general (describes behavior of entire model)
explanations, across various modalities (interactive/visual, etc.). Their
taxonomy has been used as a base for the AI Explainability 360 (AIX 360)
toolkit [20] to recommend applicable, explainable AI methods to users. Their
taxonomy only considers attributes of explanations from an implementation and
data perspective and does not account for end-user-specific requirements.
Further, their taxonomy, implemented as a decision tree, lacks a semantic
mapping of the terms involved, which makes it hard for system designers to
extend this taxonomy flexibly or to understand the interaction between the
various entities involved in the generation of explanations. In our ontology,
we provide a semantic representation that would help system designers support
and include different explanation types in their system, while accounting for
both system and user attributes.

Similarly, Arrieta et al. [1] have produced a taxonomy, mapping ML models
(primarily deep learning models) to the explanations they produced and the
features within these models that are responsible for generating these
explanations. Their taxonomy covers different types of explanations that are
produced by ML models, including simplification, explanation by examples,
local explanations, text explanations, visual explanations, feature relevance,
and explanations by transparent models. However, in their structural taxonomy,
due to the lack of a semantic representation, they often refer to explanation
types and the modalities in which they are presented interchangeably. In
addition, the explanations they cover are tightly coupled with the
capabilities of ML models and they do not explore other aspects that could be
included in explanations, such as different forms of knowledge, to make them
amenable to end users. Through our ontology, we address this gap by
incorporating diverse aspects of explanaibility that are relevant to
supporting the generation of user-centric explanation types (e.g.,
counterfactual, contrastive, scientific, trace based explanations, etc.) that
address different user goals beyond model interpretability.

Wang et al. have developed a “Conceptual Framework for Reasoned Explanations”
to describe “how human reasoning processes inform” explainable AI techniques
[26]. Besides supporting connections between how humans reason and how systems
generate an explanation, this conceptual framework also draws parallels
between various aspects of explainability, such as between explanation goals
and explanation types, human reasoning mechanisms and AI methods, explanation
types and intelligibility queries. While they cover some explanation types and
point to the need for “integrating multiple explanations,” we support a
broader set of literature-derived explanation types via our ontology. Also, it
remains unclear as to whether their framework is in a machine-readable format
that can be used to support system development. Within our ontology, we model
some of these explainability aspects that are captured in their framework,
including explanation types, explanation goals, and different modalities.

Tiddi et al. [24] created an ontology design pattern (ODP) to represent
explanations, and showcased the ability of the ODP to represent explanations
across “explanation sciences,” such as linguistics, neuroscience, computer
science, and sociology. In this ODP, they associate explanations with
attributes, including the situation, agents, theory, and events. Additionally,
they provide support for the association of explanations with explanandum (the
fact or event to be explained) and explanan (that which does the explaining)
to associate explanations with some premise to the explanandum. Their
contribution is a general-purpose ODP, however, it cannot be applied as is in
practice, due to the difficulty in condensing explanations to their suggested
form of  <explanans (A), posterior explanandum (P), theory (T), and
situational context (C)>, without the background understanding of how these
entities were generated in their field of application. In our ontology, we
reuse classes and properties from this ODP, where applicable, and expand on
their mappings to support the modeling of explanations generated via
computational processes and that address the users’ questions, situations, and
contexts.

##  3 Explanation Ontology

As we have discussed in Section 1 and 2, explainability serves different
purposes. Given this diversity, there is a need for a semantic representation
of explanations that models associations between entities and attributes
directly and indirectly related to explanations from the system as well as
user standpoints. In designing our “Explanation Ontology,” (EO) we have used
both bottom-up and top-down modeling approaches. We undertook a bottom-up
literature review to primarily identify different explanation types and their
definitions in the literature. We utilize our literature review as a base for
our modeling and use a top-down approach to refine the modeling by analyzing
the usage of different explanation types by clinicians during a requirements
gathering session we conducted. In Section 3.1, we describe our modeling,
then, in Section 3.2, we showcase our representation of literature-derived
explanation types using this modeling.

###  3.1 Ontology Composition

We design our ontology around the central explanation class (ep:Explanation)
and include entities and attributes that we see occurring often in the
literature as explanation components. In Fig. 1, we present a conceptual
overview of our ontology and depict associations necessary to understand its
coverage. In Table 1, we list ontology prefixes that we use to refer to
classes and properties.

Table 1: List of ontology prefixes used in the paper.

| Ontology
---
Prefix
Ontology | URI
sio |  | SemanticScience Integrated
---
Ontology
http://semanticscience.org/resource/
prov | Provenance Ontology | http://www.w3.org/ns/prov-o#
eo | Explanation Ontology | https://purl.org/heals/eo#
ep |  | Explanation Patterns
---
Ontology
http://linkedu.eu/dedalo/explanationPattern.owl#

Figure 1: A conceptual overview of our explanation ontology, capturing
attributes of explanations to allow them to be assembled by an AI Task, used
in a system interacting with a user. We depict user-attributes of explanations
in the upper portion (green highlight), system-attributes in the lower portion
(blue highlight), and attributes that would be visible in a user interface are
depicted in the middle portion in purple.

In our ontology, we build on the class and property hierarchies provided by
Tiddi et al.’s explanation ODP [24] and the general-purpose SemanticScience
Integrated Ontology (SIO) [8]. When referencing classes and properties from
the SemanticScience Integrated Ontology (SIO) that use numeric identifiers, we
follow the convention used in their paper [8] by referring to classes and
properties via their labels. E.g., sio:‘in relation to’.

We introduce classes and properties as necessary to construct a model of
explanations that supports property associations between explanations
(ep:Explanation), the AI Task (eo:AITask) that generated the recommendation
(eo:SystemRecommendation) upon which the explanation is based (ep:isBasedOn)
and the end-user (eo:User) who consumes them. In our modeling, we note that
explanations are dependent on (ep:isBasedOn) both system recommendations as
well as implicit/explicit knowledge (eo:knowledge) available to systems or
possessed by users. In addition, we also model that the knowledge available to
the system can be in relation to (sio:‘in relation to’) various entities, such
as the domain knowledge, situational knowledge of the end-user (eo:User), and
knowledge about a record of the central object (eo:ObjectRecord) fed into the
system (e.g., as we will see in Section 4, a patient is a central object in a
clinical decision support system). We also model that explanations address a
question (sio:‘question’) posed by end-users, and these questions are
implemented (eo:implements) by AI Tasks that generate recommendations.

AI Tasks (eo:AITask) can be thought of as analogous with different reasoning
types (i.e., inductive, deductive, abductive or a hybrid reasoning strategy)
and that are implemented by different AI methods (eo:AIMethod) (e.g.,
similarity algorithms, expert systems) to arrive at recommendations. This
decomposition of an AI Task to methods is inspired by Tu et al.’s research in
the problem-solving domain [25] of creating domain-independent AI systems that
can be instantiated for specific domains. Besides capturing the interplay
between an eo:AITask and its implementation, an eo:AIMethod, we also model
that an AI Task is implemented in a particular reasoning mode
(eo:ReasoningMode) of the system which dictates the overall execution
strategy. We believe our approach to supporting the different granularities of
work separation within an AI system can be valuable to building AI systems
with hybrid reasoning mechanisms capable of generating different explanation
types. In addition to capturing the situational context of the user, we also
support modeling their existing knowledge and preferences for different forms
for presentations of explanation (eo:ExplanationModality).

In the rest of this paper, we refer to ontology classes within single quotes
and italicize them (e.g., ‘knowledge’) to give readers an idea of the coverage
of our ontology. While, we only depict the top-level classes associated with
explanations in Fig. 1, through representations of different explanation types
and an example from the clinical requirements gathering session, we show how
system designers can associate explanations with more specific subclasses of
entities, such as with particular forms of ‘knowledge’, ‘AI Task’s, and ‘AI
methods’. Further, we maintain definitions and attributions for our classes
via the usage of terminology from the DublinCore [19] and Prov-O [10]
ontologies.

###  3.2 Modeling of Literature-derived Explanation Types

We previously created a taxonomy of literature-derived, explanation types [4]
with refined definitions of the nine explanation types. We leverage the
mappings provided within EO, and knowledge of explanation types from our
taxonomy, to represent each of these explanation types as subclasses of the
explanation class (ep:Explanation). These explanation types serve different
user-centric purposes, are differently suited for users’ ‘situations,’ context
and ‘knowledge,’ are generated by various ‘AI Task’ and ‘methods,’ and have
different informational needs. Utilizing the classes and properties supported
within our ontology, we represent the varied needs for the generation of each
explanation type, or the sufficiency conditions for each explanation type, as
OWL restrictions.

In Table 2, we present an overview of the different explanation types along
with their descriptions and sufficiency conditions. In Listing 1, we present
an RDF representation of a ‘contextual explanation,’ depicting the encoding of
sufficiency conditions on this class.

Listing 1: OWL expression of the representation of a ‘contextual explanation’
(whose sufficiency conditions can be referred to from Table 2) in Manchester
syntax. In this snippet, we show the syntax necessary to understand the
composition of the ‘contextual explanation’ class in reference to the classes
and properties introduced in Fig. 1.

⬇

1Class: eo:ContextualExplanation

2 EquivalentTo:

3 (isBasedOn some eo:‘System Recommendation’)

4 and (

5 (ep:isBasedOn some

6 (eo:’Contextual Knowledge’

7 and (sio:‘in relation to’ some ep:Situation))) or

8 (ep:isBasedOn some (‘Contextual Knowledge’

9 and (sio:‘in relation to’ some eo:‘Object Record’))))

10 SubClassOf:

11 ep:Explanation

12

13Class: ep:Explanation

14 SubClassOf:

15 sio:‘computational entity’,

16 ep:isBasedOn some eo:Knowledge,

17 ep:isBasedOn some eo:SystemRecommendation,

18 ep:isConceptualizedBy some eo:AITask

Table 2: An overview of explanation types against simplified descriptions of
their literature-synthesized definitions, and natural language descriptions of
sufficiency conditions. Within the explanation type description we also
include a general prototypical question that can be addressed by each
explanation type. Further, within the sufficiency conditions, we highlight
ontology classes using single quotes and italics.

Explanation Type | Description | Sufficiency Conditions
---|---|---
Case Based |  | Provides solutions that are based on actual prior cases
---
that can be presented to the user to provide compelling
support for the system’s conclusions, and may involve
analogical reasoning, relying on similarities between
features of the case and of the current situation.
“To what other situations has this
recommendation been applied?”
| Is there at least one other prior case (‘object record’)
---
similar to this situation that had an ‘explanation’?
Is there a similarity between this case, and
that other case?
Contextual |  | Refers to information about items other than the
---
explicit inputs and output, such as information about
the user, situation, and broader environment that
affected the computation.
“What broader information about the current
situation prompted the suggestion of this
recommendation?”
| Are there any other extra inputs that are not
---
contained in the ‘situation’ description itself?
And by including those, can better insights
be included in the ‘explanation’?
Contrastive |  | Answers the question “Why this output instead of that
---
output,” making a contrast between the given output
and the facts that led to it (inputs and other
considerations), and an alternate output of interest and
the foil (facts that would have led to it).
“Why choose option A
over option B that I typically choose?”
| Is there a ‘system recommendation’ that was made
---
(let’s call it A)? What facts led to it?
Is there another ‘system recommendation’ that
could have happened or did occur, (let’s call it B)?
What was the ‘foil’ that led to B?
Can A and B be compared?
Counterfactual |  | Addresses the question of what solutions would have
---
been obtained with a different set of inputs than
those used.
“What if input A was over 1000?”
| Is there a different set of inputs that can be
---
considered?
If so what is the alternate ‘system recommendation’?
Everyday |  | Uses accounts of the real world that appeal to the user,
---
given their general understanding and knowledge.
“Why does option A make sense”
| Can accounts of the real world be
---
simplified to appeal to the user based on
their general understanding and ‘knowledge’?
Scientific |  | References the results of rigorous scientific methods,
---
observations, and measurements.
“What studies have backed this
recommendation?”
| Are there results of rigorous ‘scientific
---
methods’ to explain the situation?
Is there ‘evidence’ from the literature to
explain this ‘situation’?
Simulation Based |  | Uses an imagined or implemented imitation of a
---
system or process and the results that emerge from
similar inputs.
“What would happen if this recommendation
is followed?”
| Is there an ‘implemented’
---
imitation of the ‘situation’ at hand?
Does that other scenario have inputs similar
to the current ‘situation’?
Statistical |  | Presents an account of the outcome based on data about
---
the occurrence of events under specified
(e.g., experimental) conditions. Statistical explanations
refer to numerical evidence on the likelihood of factors
or processes influencing the result.
“What percentage of people with
this condition have recovered?”
| Is there ‘numerical evidence’/likelihood
---
account of the ‘system recommendation’ based on
data about the occurrence of the outcome
described in the recommendation?
Trace Based |  | Provides the underlying sequence of steps used by the
---
system to arrive at a specific result, containing the line
of reasoning per case and addressing the question of
why and how the application did something.
“What steps were taken by the system to
generate this recommendation?”
| Is there a record of the underlying sequence
---
of steps (‘system trace’) used by the ‘system’ to
arrive at a specific ‘recommendation’?

##  4 Clinical Use Case

We demonstrate the use of EO in the design and operations of an AI system to
support treatment decisions in the care of patients with diabetes. We
previously conducted a two-part user-centered design study that focused on
determining which explanations types are needed within such a system. In the
first part of the study, we held an expert panel session with three diabetes
specialists to understand their decision-support needs when applying
guideline-based recommendations in diabetes care. We then used the
requirements gathered from this session to design a prototype AI system. In
the second part, we performed cognitive walk-throughs of the prototype to
understand what reasoning strategies clinicians used and which explanations
were needed when presented with a complex patient.

In modeling the reasoning strategies that need to be incorporated into this
system design, we found that the Select-Test (ST) model by Stefanoli and
Ramoni [22] mirrored the clinician’s approach. Applying their ST model, we can
organize the clinical reasoning strategy within the system design based on
types of ‘reasoning mode,’ such as differential diagnosis or treatment
planning. Each of these modes can be associated with AI tasks, such as ranking
that creates a preferential order of options like diagnoses or treatments,
deduction that predicts consequences from hypotheses, abstraction that
identifies relevant clinical findings from observations, and induction that
selects the best solution by matching observations to the options or requests
new information where necessary. Each of these AI tasks can generate system
recommendations that requires explanations from the clinicians.

We discovered that, of the types of explanations listed in Table 1, everyday
and contextual explanations were required more than half the time. We noted
that clinicians were using a special form of everyday explanations,
specifically their experiential knowledge or ‘clinical pearls’ [14] to explain
the patient’s case. We observed concrete examples of the explanation
components being used in the explanations provided by clinicians, such as
‘contextual knowledge’ of a patient’s condition being used for diagnosis and
drug ‘recommendations’. Other examples of explanation types needed within the
system design include trace-based explanations in a treatment planning mode,
to provide an algorithmic breakdown of the guideline steps that led to a drug
recommendation; ‘scientific explanations’ in a plan critiquing mode, to
provide references to studies that support the drug, as well as
‘counterfactual explanations’, to allow clinicians to add/edit information to
view a change in the recommendation; and ‘contrastive explanations’ in a
differential diagnosis mode, to provide an intuition about which drug is the
most recommended for the patient. The results of the user studies demonstrated
the need for a diverse set of explanation types and that modeling explanation
requires various components to support AI system design.

An example of our ontology being used to represent the generation process for
a ‘contrastive explanation’, while accounting for the ‘reasoning mode,’ ‘AI
Task’ involved, can be viewed in Listing 2. In the RDF representation of a
‘contrastive explanation’ used by a clinician, we depict how our ontology
would be useful to guide a system to provide an explanation in real-time to
the question, “Why Drug B over Drug A?” In our discussion of the Listing 2
hereafter, we include the entity IRIs from the listing in parantheses. Upon
identifying what explanation type would best suit this question, our ontology
would guide the system to access different forms of ‘knowledge’ and invoke the
corresponding ‘AI tasks’ that are suited to generate ‘contrastive
explanations’. In this example, a deductive AI task (:AITaskExample) is
summoned and generates a system recommendation (:SystemRecExampleA) that Drug
A is insufficient based on contextual knowledge of the patient record
(:ContextualKnowledgePatient). In addition, the deductive task is also fed
with guideline evidence that Drug B is a preferred drug, which results in the
generation of a recommendation (:SystemRecExampleB) in favor of Drug B.
Finally, our ontology would help guide a system to populate the components of
a ‘contrastive explanation’ from ‘facts’ that supported the hypothesis, “Why
Drug B?” and its ‘foil’, “Why not Drug A?,” or the facts that ruled out Drug
A. We note that the annotation of granular content, such as patient and drug
data within these explanations, would require the usage of domain-specific
ontologies, whose concepts would need to be inferred into classes supported
within our ontology. We defer the granular content annotation effort to future
work.

Listing 2: Turtle representation of the process a system would undergo to
generate a ‘contrastive explanation’, such as the one presented during our
cognitive walk-through to address, “Why drug B over drug A?”

⬇

1:ContrastiveQuestion

2 a sio:‘question’;

3 rdfs:label ‘‘Why Drug B over Drug A?’’ .

4

5:ContrastiveExpInstance

6 a eo:ContrastiveExplanation;

7 ep:isBasedOn :SystemRecExampleA, :SystemRecExampleB;

8 rdfs:label ‘‘Guidelines recommend Drug B for this patient’’;

9 :addresses :ContrastiveQuestion .

10

11:SystemRecExampleA

12 a eo:SystemRecommendation;

13 prov:used :ContextualKnowledgePatient;

14 rdfs:label ‘‘Drug A is not sufficient for the patient’’ .

15

16:SystemRecExampleB

17 a eo:SystemRecommendation;

18 prov:used :GuidelineEvidence;

19 rdfs:label ‘‘Drug B is recommended by the guidelines’’ .

20

21:AITaskExample

22 a eo:DeductiveTask;

23 sio:‘has output’ :SystemRecExampleA, :SystemRecExampleB;

24 ep:hasSetting [a eo:ReasoningMode; rdfs:label ‘‘Treatment Planning’’];

25 prov:used :ContextualKnowledgePatient, :GuidelineEvidence;

26 rdfs:label ‘‘Deductive task’’ .

27

28:ContextualKnowledgePatient

29 a eo:ContextualKnowledge, eo:Foil;

30 sio:‘in relation to’ [a sio:‘patient’];

31 sio:‘is input in’ :AITaskExample;

32 rdfs:label ‘‘patient has hyperglycemia’’ .

33

34:GuidelineEvidence

35 a eo:ScientificKnowledge, eo:Fact;

36 sio:‘is input in’ :AITaskExample;

37 rdfs:label ‘‘Drug B is the preferred drug’’ .

##  5 Evaluation

We evaluate our ontology via a set of competency questions posed from the
perspective of a system designer who may need to design a system that includes
appropriate explanation support and may hope to use our ontology. These
competency questions are designed to aid system designers in their planning of
resources to include for generating explanations that are suitable to the
expertise level of the end-user, the scenario for which the system is being
developed, etc. The resources that a system designer would need to consider
could include the ‘AI method’ and ‘tasks’ capable of generating the set of
explanations that best address the user’s ‘question’ [11], the reasoning
‘modes’ that need to be supported within the system, and the ‘knowledge’
sources. These competency questions would be ones that, for example, we, as
system designers looking to implement a clinical decision-support system, such
as the prototype described in Section 4, would ask ourselves upon analyzing
the requirements of clinicians gathered from a user study. Contrarily, if the
specifications were to live in documentation, versus an explanation ontology,
it would be cumbersome for a system designer to perform a lookup to regenerate
the explanation requirements for every new use case. Through EO we support
system designers to fill in what can be thought of as “slots” (i.e.,
instantiate classes) for explanation generation capabilities.

Table 3: A catalog of competency questions and candidate answers produced by
our ontology. These questions can be generalized to address queries about
other explanation types supported within our ontology.

Competency Question | Answer
---|---
| Q1. Which AI model (s) is capable of
---
generating this explanation type
(e.g. trace-based)?
| Knowledge-based systems,
---
Machine learning model: decision trees
| Q2. What example questions have been
---
identified for counterfactual explanations?
| What other factors about the patient does
---
the system know of?
What if the major problem was a fasting
plasma glucose?
| Q3. What are the components of a
---
scientific explanation?
| Generated by an AI Task, Based on
---
recommendation, and based on evidence from
study or basis from scientific method
| Q4. Given the system was performing abductive
---
reasoning and has ranked specific
recommendations by comparing different
medications, what explanations can be
provided for that recommendation?
Contrastive explanation
| Q5. Which explanation type best suits the
---
user question, “Which explanation type
can expose numerical evidence about
patients on this drug?,” and how
will the system generate the answer?
| Explanation type: statistical
---
System: run ‘Inductive’ AI task with
‘Clustering’ method to generate
numerical evidence

In Table 3, we present a list of competency questions along with answers. The
first three questions can be addressed before or during system development,
when the system designer has gathered user requirements, and the last two
questions need to be answered in real-time, based on the system and a user’s
current set of attributes. During system development and after the completion
of a requirements gathering session, if a system designer learns that a
certain set of explanation types would be suitable to include, through answers
to the first three questions, they can be made aware of the AI methods capable
of generating explanations of the type (Q1), the components to build into
their system to ensure that the explanations can be generated (Q3), and some
of the questions that have been addressed by the particular explanation type
(Q2). When a system is running in real-time, an answer to (Q4) would help a
system designer decide what pre-canned explanation type already supported by
the system would be the best current set of system, and user attributes and an
answer to (Q5) would help a system designer decide on whether their system,
with its current capabilities, can generate an explanation type that is most
suitable to the form of the question being asked. While we address these
competency questions, and specifically ask questions Q4 and Q5 in the setting
of our clinical requirements gathering session, we expect that these questions
can be easily adapted for other settings to be addressed with the aid of EO.
In addition to presenting natural-language answers to sample competency
questions, we depict a SPARQL query used to address the third question in
Listing 3.

Listing 3: A SPARQL query that retrieves the sufficiency conditions encoded on
an explanation type to answer a competency question of the kind, “What are the
components of a scientific explanation?”

⬇

1prefix rdfs:<http://www.w3.org/2000/01/rdf-schema#>

2prefix owl:<http://www.w3.org/2002/07/owl#>

3

4select ?class ?restriction

5where {

6 ?class (rdfs:subClassOf|owl:equivalentClass) ?restriction .

7 ?class rdfs:label ‘‘Scientific Explanation’’ .

8}

Table 4: Results of the SPARQL query to retrieve the sufficiency conditions
defined on the ‘scientific explanations’ class. These results depict the
flexibility that we allow so that ‘scientific explanations’ can either be
directly based on ‘scientific knowledge’ or on system recommendations that use
‘scientific knowledge’.

subject | restriction
---|---
Scientific Explanation |  | (ep:isBasedOn some (eo:‘Scientific Knowledge’ and ((prov:wasGeneratedBy
---
some ‘Study’) or (prov:wasAssociatedWith some eo:‘Scientific Method’))) and
(isBasedOn some eo:‘System Recommendation’)) or
(ep:isBasedOn some (eo:‘System Recommendation’ and
(prov:used some (eo:‘Scientific Knowledge’ and ((prov:wasGeneratedBy
some ‘Study’) or (prov:wasAssociatedWith some
eo:‘Scientific Method’))))))

##  6 Resource Contributions

We contribute the following publicly available artifacts: Explanation Ontology
with the logical formalizations of the different explanation types and SPARQL
queries to evaluate the competency questions, along with the applicable
documentation available on our resource website. These resources, listed in
Table 5, are useful for anyone interested in building explanation facilities
into their systems. The ontology has been made available as an open-source
artifact under the Apache 2.0 license [21] and we maintain an open source
Github repository for all our artifacts. We also maintain a persistent URL for
our ontology hosted on the PURL service.

Table 5: Links to resources we have released and refer to in the paper.

Resource | Link to resource
---|---
Resource Website | http://tetherless-world.github.io/explanation-ontology
EO PURL URL | https://purl.org/heals/eo
Github Repository | https://github.com/tetherless-world/explanation-ontology

##  7 Discussion and Future Work

To address the gap in a semantic representation that can be used to support
the generation of different explanation types, we designed an OWL ontology, an
explanation ontology, that can be used by system designers to incorporate
different explanation types into their AI-enabled systems. We leverage and
maintain compatibility with an existing explanation patterns ontology [24] and
we expand on it to include representational primitives needed for modeling
explanation types for system designers. We also leveraged the widely-used
general-purpose SIO ontology, and introduce the classes and properties
necessary for a system to generate a variety of user-centric explanations.
During our modeling, we make certain decisions that are inspired by our
literature-review and knowledge of the usage of explanation types in clinical
practice, to include classes (e.g., ‘system recommendation,’ ‘knowledge,’
‘user’) that we deem as central to generating explanations. We include other
classes that would indirectly be needed to generate explanations and reason
about them, hence, capturing the process involved in explanation generation.
However, through our ontology we do not generate natural language
explanations, and rather provide support to fill in “slots” that will be
included in them.

Our explanation ontology is comprehensive and flexible as it was designed from
requirements gathered from a relatively extensive literature review along with
a requirements gathering session in a clinical domain. In this paper, we have
described how the ontology can be used to represent literature-derived
explanation types and then how those explanation types address the questions
posed by clinicians. The ontology is also designed to be extensible, as with
all ontologies, representational needs may arise as applications arise and
evolve. Our competency questions provide guidance to system designers as they
make their plans for providing explanations within their decision support
systems.

In the future, we plan to build a middleware framework (such as the service we
alluded to in Section 1) that would interact with the system designer, take a
user’s ‘question’ as input, and apply learning techniques on a combination of
the user’s ‘question’ and the inputs available to the AI system, which could
include the user’s ‘situation’ and context, and the system’s ‘reasoning mode’
to decide on the most suitable ‘explanation’ type. Upon identifying the
appropriate ‘explanation’ type, the framework would leverage the sufficiency
conditions encoded in our ontology to gather different forms of ‘knowledge’ to
generate the suitable explanation type and summon the AI ‘tasks’ and ‘methods’
that are best suited to generating the explanation. Such a framework would
then be capable of working in tandem with hybrid AI reasoners to generate
hybrid explanations [4, 17] that serve the users’ requirements.

We have represented the explainability components that we deem necessary to
generate explanations with a user focus. However, there are other aspects of
user input that may be harder to capture. Wang et al. [26] have shown that
there is a parallel between one of these user aspects, a user’s reasoning
strategies, and the reasoning types that an ‘AI Task’ uses to support the
generation of explanations to address these situations. We are investigating
how to include classes, such as a user’s reasoning strategies, that are harder
to capture/infer from a system perspective and would be hard to operationalize
in a system’s model. The user-centric focus of AI has been emphasized recently
by Liao et al. in their question bank of various user questions around
explainability that suggests “user needs should be understood, prioritized and
addressed” [11]. As we start to build more user attributes into our ontology,
we believe that our model will evolve to support more human-in-the-loop AI
systems. We are using EO as a foundation for generating different explanation
types in designing a clinical decision support system, and we will publish
updates to our ontology as we make edits to support new terms.

##  8 Conclusion

We have built an ontology aimed at modeling explanation primitives that can
support user-centered AI system design. Our requirements came from a breadth
of literature and requirements gathered by prospective and actual users of
clinical decision support systems. We encode sufficiency conditions
encapsulating components necessary to compose hybrid explanation types that
address different goals and expose different forms of knowledge in our
ontology. Through a carefully crafted set of competency questions, we have
exposed and evaluated the coverage of our ontology in helping system designers
make decisions about explanations to include in their systems. We believe our
ontology can be a valuable resource for system designers as they plan for what
kinds of explanations their systems will need to support. We are continuing to
work towards supporting the generation of different explanation types and
designing a service that would use our explanation ontology as a base to
generate explanation types that address users’ questions.

## Acknowledgements

This work is done as part of the HEALS project, and is partially supported by
IBM Research AI through the AI Horizons Network. We thank our colleagues from
IBM Research, including Ching-Hua Chen, and from RPI, Sabbir M. Rashid,
Henrique Santos, James P. McCusker and Rebecca Cowan, who provided insight and
expertise that greatly assisted the research.

## References

  * [1] Arrieta, A.B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al.: Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58, 82–115 (2020)
  * [2] Arya, V., Bellamy, R.K., Chen, P.Y., Dhurandhar, A., Hind, M., Hoffman, S.C., Houde, S., Liao, Q.V., Luss, R., Mojsilović, A., et al.: One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques. arXiv preprint arXiv:1909.03012 (2019)
  * [3] Biran, O., Cotton, C.: Explanation and justification in machine learning: A survey. In: IJCAI-17 workshop on explainable AI (XAI). vol. 8, p. 1 (2017)
  * [4] Chari, S., Gruen, D.M., Seneviratne, O., McGuinness, D.L.: Directions for Explainable Knowledge-Enabled Systems. In: Ilaria Tiddi, Freddy Lecue, Pascal Hitzler (eds.), Knowledge Graphs for eXplainable AI – Foundations, Applications and Challenges. Studies on the Semantic Web. p. to appear. IOS Press (2020)
  * [5] Chari, S., Gruen, D.M., Seneviratne, O., McGuinness, D.L.: Foundations of Explainable Knowledge-Enabled Systems. In: Ilaria Tiddi, Freddy Lecue, Pascal Hitzler (eds.), Knowledge Graphs for eXplainable AI – Foundations, Applications and Challenges. Studies on the Semantic Web. p. to appear. IOS Press (2020)
  * [6] Dhaliwal, J.S., Benbasat, I.: The use and effects of knowledge-based system explanations: theoretical foundations and a framework for empirical evaluation. Information systems research 7(3), 342–362 (1996)
  * [7] Doshi-Velez, F., Kim, B.: Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017)
  * [8] Dumontier, M., Baker, C.J., Baran, J., Callahan, A., Chepelev, L., Cruz-Toledo, J., Del Rio, N.R., Duck, G., Furlong, L.I., Keath, N., et al.: The Semanticscience Integrated Ontology (SIO) for biomedical research and knowledge discovery. J. Biomed. Semantics 5(1), 14 (2014)
  * [9] Holzinger, A., Biemann, C., Pattichis, C.S., Kell, D.B.: What do we need to build explainable AI systems for the medical domain? arXiv preprint arXiv:1712.09923 (2017)
  * [10] Lebo, T., Sahoo, S., McGuinness, D., Belhajjame, K., Cheney, J., Corsar, D., Garijo, D., Soiland-Reyes, S., Zednik, S., Zhao, J.: Prov-o: The prov ontology. W3C recommendation (2013)
  * [11] Liao, Q.V., Gruen, D., Miller, S.: Questioning the AI: Informing Design Practices for Explainable AI User Experiences. arXiv preprint arXiv:2001.02478 (2020)
  * [12] Lim, B.Y., Dey, A.K., Avrahami, D.: Why and why not explanations improve the intelligibility of context-aware intelligent systems. In: Proc. of the SIGCHI Conf. on Human Factors in Computing Systems. pp. 2119–2128. ACM (2009)
  * [13] Lipton, Z.C.: The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016)
  * [14] Lorin, M.I., Palazzi, D.L., Turner, T.L., Ward, M.A.: What is a clinical pearl and what is its role in medical education? Medical teacher 30(9-10), 870–874 (2008)
  * [15] Lou, Y., Caruana, R., Gehrke, J.: Intelligible models for classification and regression. In: Proc. of the 18th ACM SIGKDD Int. Conf. on Knowledge discovery and data mining. pp. 150–158. ACM (2012)
  * [16] Miller, T.: Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267, 1–38 (2019)
  * [17] Mittelstadt, B., Russell, C., Wachter, S.: Explaining explanations in AI. In: Proc. of the Conf. on fairness, accountability, and transparency. pp. 279–288. ACM (2019)
  * [18] Páez, A.: The pragmatic turn in explainable artificial intelligence (xai). Minds and Machines 29(3), 441–459 (2019)
  * [19] Dublin Core Metadata Initiative: Dublin Core Metadata Terms. Resource Page. https://www.dublincore.org/specifications/dublin-core/dcmi-terms/2006-12-18/ (accessed Aug. 18, 2020
  * [20] IBM Research Trusted AI: AI Explainability 360 Open Source Toolkit. Resource Page. https://aix360.mybluemix.net/ (accessed Aug. 18, 2020)
  * [21] The Apache Software Foundation: Apache 2.0 License. License Description Page. https://www.apache.org/licenses/LICENSE-2.0 (accessed Aug 18, 2020)
  * [22] Stefanelli, M., Ramoni, M.: Epistemological constraints on medical knowledge-based systems. In: Advanced models of cognition for medical training and practice, pp. 3–20. Springer (1992)
  * [23] Swartout, W.R., Moore, J.D.: Explanation in second generation expert systems. In: Second generation expert systems, pp. 543–585. Springer (1993)
  * [24] Tiddi, I., d’Aquin, M., Motta, E.: An ontology design pattern to define explanations. In: Proceedings of the 8th Int. Conf. on Knowledge Capture. pp. 1–8 (2015)
  * [25] Tu, S.W., Eriksson, H., Gennari, J.H., Shahar, Y., Musen, M.A.: Ontology-based configuration of problem-solving methods and generation of knowledge-acquisition tools: application of PROTEGE-II to protocol-based decision support. Artificial intelligence in medicine 7(3), 257–289 (1995)
  * [26] Wang, D., Yang, Q., Abdul, A., Lim, B.Y.: Designing theory-driven user-centric explainable AI. In: Proceedings of the 2019 CHI Conf. on Human Factors in Computing Systems. pp. 1–15 (2019)