# Interpreting Differentiable Latent States for Healthcare Time-series Data

Yu Chen  Nivedita Bijlani  Samaneh Kouchaki  Payam Barnaghi

###### Abstract

Machine learning enables extracting clinical insights from large temporal
datasets. The applications of such machine learning models include identifying
disease patterns and predicting patient outcomes. However, limited
interpretability poses challenges for deploying advanced machine learning in
digital healthcare. Understanding the meaning of latent states is crucial for
interpreting machine learning models, assuming they capture underlying
patterns. In this paper, we present a concise algorithm that allows for i)
interpreting latent states using highly related input features; ii)
interpreting predictions using subsets of input features via latent states;
and iii) interpreting changes in latent states over time. The proposed
algorithm is feasible for any model that is differentiable. We demonstrate
that this approach enables the identification of a daytime behavioral pattern
for predicting nocturnal behavior in a real-world healthcare dataset.

Healthcare,Machine Learning,Interpretability, ICML

##  1 Introduction

Digital healthcare is a rapidly growing field empowered by the internet,
computers, and mobile devices. It offers promising prospects for enhancing
healthcare quality and accessibility. Machine learning techniques have the
potential to unlock valuable insights from extensive temporal datasets,
enabling healthcare providers to identify disease patterns and predict patient
outcomes. However, deploying sophisticated machine learning technologies in
digital healthcare poses challenges due to their limited interpretability. The
interpretability of machine learning models is crucial in healthcare
applications because it enables professionals to understand how a model
arrived at a particular prediction or decision. This is especially important
when making decisions that can have significant consequences for patients,
such as in disease diagnosis or treatment recommendation.

Explainable Artificial Intelligence (XAI) has drawn increasing attention in
recent years due to the growing use of deep learning models in critical
decision-making applications. Various approaches have been developed for
explaining black-box models, such as feature importance explanations (Datta et
al., 2016; Ribeiro et al., 2016; Lundberg & Lee, 2017; Sundararajan et al.,
2017) and example-based explanations (Li et al., 2018; Chen et al., 2019;
Gurumoorthy et al., 2019; Crabbé et al., 2021). However, when it comes to
healthcare applications, obtaining only the importance of each feature or
example is not enough for interpreting the underlying patterns. Understanding
the meaning of latent states (representations) learned by a black-box model is
key to this issue, assuming they capture underlying patterns. We posit that a
latent state can be translated to a comprehensible pattern that is formed by a
subset of features.

In this paper, we introduce a simple, yet efficient algorithm that aims to: i)
interpret latent states using highly related input features; ii) interpret
predictions using subsets of input features via latent states; and iii)
interpret changes in latent states over time. Once the first two goals are
achieved, the algorithm can readily fulfill the third one as long as the model
is able to learn latent states over time. This algorithm is feasible for any
differentiable latent state with respect to input features. Hence, it can work
as a readily available plugin for most deep-learning models for temporal data,
such as transformers (Vaswani et al., 2017), recurrent neural networks
(Tealab, 2018), Long Short-Term Memory (LSTM) networks (Hochreiter &
Schmidhuber, 1997), and advanced state space models (Gu et al., 2021). To
demonstrate the portability of our method, we additionally provide a neat
solution for applying it to models using Neural  (NCDE) (Kidger et al., 2020).
NCDE models are capable of i) fitting the dynamics of latent states (i.e.
first-order derivatives w.r.t. time), ii) capturing long-term dependencies in
latent states, iii) dealing with irregularly sampled time series, and iv)
working in an online fashion. In short, NCDE is an attractive approach for
modeling healthcare time-series data. We demonstrate that the proposed
algorithm enables the identification of a daytime behavioral pattern for
predicting nocturnal behavior in a real-world healthcare dataset using a NCDE
model.

##  2 Related work

In this section, we introduce several prior methods in XAI that are related to
our work.

###  2.1 Feature importance to predictions

The most popular methods in XAI focus on identifying important features for
the prediction of a given sample, such as SHAP (Lundberg & Lee, 2017) and LIME
(Ribeiro et al., 2016). These methods do not consider the factor of time.
Crabbé et al. (2021) proposed Dynamask to identify feature importance over
time, and Crabbé & van der Schaar (2022) proposed for measuring feature
importance without labels. However, we aim to detect salient patterns rather
than individual features, which could be more helpful in understanding disease
progression or predicting certain symptoms.

###  2.2 Interpreting latent states

We can view a latent state as a compact code of an underlying pattern of
observed features. It is often referred to as a latent representation. Esser
et al. (2020) introduced an approach to transform a latent state into an
interpretable one. The interpretability comes from a decomposition of semantic
concepts, which requires prior knowledge to find associated semantic concepts
in the data. Accessing this information is rather difficult in healthcare
data.

The integrated Jacobian (also known as integrated gradients) (Sundararajan et
al., 2017; Crabbé et al., 2021) can quantify the impact of an input feature on
the shift of a latent state. Although this is a one-to-one impact measure, we
can utilize it to identify subsets of key features for each latent state.

Assume we have a model $\mathcal{G}$ to infer the latent state $z\in R^{H}$ of
an observation $x\in R^{D}$, i.e. $z=\mathcal{G}(x):R^{D}\rightarrow R^{H},\ \
H<D$. Let $x_{i}$ denote the $i$-th feature of a baseline sample $x$,
$\hat{x}_{i}$ denotes the $i$-th feature of a test sample $\hat{x}$. We can
compute the integrated Jacobian of the $i$-th feature $\mathbf{j}_{i}\in
R^{H}$ regarding $(x,\hat{x})$ as below:

$\begin{split}\mathbf{j}_{i}&=(x_{i}-\hat{x}_{i})\int_{0}^{1}\frac{\partial\mathcal{G}(x)}{\partial
x_{i}}\Bigr{|}_{\gamma_{i}(\lambda)}d\lambda,\end{split}$ (1)

Here $\gamma_{i}(\lambda)=\hat{x}_{i}+\lambda(x_{i}-\hat{x}_{i}),\ \
\forall\lambda\in[0,1]$, represents a point between $x_{i}$ and $\hat{x}_{i}$.

##  3 Methods

In this section, we first define a measurement that quantifies the impact of
an input feature on a latent state. We then propose an algorithm for
identifying the most influential features that affect each latent state. With
this algorithm, we can interpret each latent state by two sets of influential
features, which likely result in the most positive and negative magnitude
shift of a latent state. We can further analyze how these feature sets affect
predictions by combining them with other methods (such as SHAP, LIME).

###  3.1 Impact of an input feature to a latent state

We quantify the impact of $i$-th input feature on $s$-th latent state in terms
of the shift between two samples $(x,\hat{x})$ as the following equation:

$p_{i,s}=\frac{\mathbf{j}_{i,s}}{|z_{s}-\hat{z}_{s}|},\ \ z=\mathcal{G}(x),\ \
\hat{z}=\mathcal{G}(\hat{x})$ (2)

$\mathbf{j}_{i,s}$ is the $s$-th element in $\mathbf{j}_{i}$ which can be
obtained by Equation 1. $z_{s},\hat{z}_{s}$ are the $s$-th element in $z$ and
$\hat{z}$ respectively.

The absolute value of $p_{i,s}$ is the ratio of the shift caused by the
specific feature compared to the total shift caused by all features for a
latent state. The impact can be in the positive (increasing) or negative
(decreasing) direction which enables us to analyze the shifts more precisely.
Crabbé et al. (2021) provides a measurement called projected Jacobian which
quantifies the total impact of an input feature on all latent states. Using
Equation 2, we can measure the impact of an input feature on individual latent
states.

Figure 1: The two heat maps for interpreting latent state $z_{1}$ over time
$t$ learned by the NCDE model. The y-axis is the latent state along the time
axis, $z_{1}(t_{n})$ means $z_{1}$ at the $n$-th time step. The x-axis is the
input features which are named by the locations of sensors and the aggregate
methods, e.g. ”kitchen” is the number of activities in the kitchen during one
hour, ”kitchen_ctime” is the cumulative number of hours that ”kitchen” has
non-zero values, ”kitchen_cmax” is the maximum value of ”kitchen” among all
passed hours, etc.

###  3.2 Interpreting latent states by influential features

Algorithm 1 Generate Feature Heat Map of Latent States

Input: The direction of impact $d\in\\{True,False\\}$, training set
$\\{X,y\\}$, model $\mathcal{G}$, baseline set $\hat{X}$, the direction of
impact $d\in\\{True,False\\}$, number of latent states $H$, number of input
features $D$, size of a subset of training samples $m$, number of top samples
$k$, number of latent states $h$, and number of top features $l$.

$X_{c}\leftarrow\text{balanced subset}(X,y,m)$

$Z_{c}\leftarrow\text{latent states}(X_{c},\mathcal{G})$

Initialize the heat map matrix $\mathbf{M}$ with zeros: $\forall\ \ 0\leq
i<H,0\leq j<D,\mathbf{M}_{i,j}=0$

for $\hat{x}\in\hat{X}$ do

$\hat{z}\leftarrow\text{latent states}(\hat{x},\mathcal{G})$

$\mathbf{Z}_{id}\leftarrow\text{TopDissimilarSamples}(Z_{c},\hat{z},k)$

$\mathbf{F}\leftarrow\text{TopImpactfulFeatures}(d,\mathcal{G},\hat{x},X_{c}[\mathbf{Z}_{id}],l)$

for $k_{i}\in\\{1,\dots,k\\}$ do

for $h_{i}\in\\{1,\dots,h\\}$ do

$i=h_{i},\ \ \forall j\in\mathbf{F}_{k_{i},h_{i}},\ \ \mathbf{M}[i,j]+=1$

end for

end for

end for

Return: $\mathbf{M}$

Our proposed approach for interpreting latent states is based on contrastive
methods, i.e. at least one pair of different samples is required to compute
the similarities and shifts in latent and input spaces.

More specifically, we identify the most different samples $\\{x|x\in
X_{dif}\\}$ compared to a baseline sample $\hat{x}$ and locate features that
contribute most to the positive/negative shift of each latent state between
each pair of $(x,\hat{x})$. To obtain an overall result, we iterate this
process on a small set of baseline samples and generate a heat map that
indicates how frequently an input feature has been identified as one of the
most impactful features of a latent state. Please refer to Algorithms 1, 2 and
3 for more details. By visualizing this heat map, we can readily observe the
relation between latent states and input features over time. Figure 1 shows an
example of this heat map obtained by real-world data.

In order to identify influential features, we focus on the most dissimilar
pairs of training samples. These pairs are likely to exhibit significant
shifts in their latent states. On the other hand, the most similar pairs tend
to have minimal shifts in most latent states, making it challenging to
determine which features contribute the most to their similarity.

Algorithm 2 TopDissimilarSamples

Input: Latent states of the subset samples $Z$, latent states of a baseline
sample $z$, number of top samples $k$.

$S_{cos}\leftarrow$ Cosine Similarities between $z$ and $Z$

$S_{ids}\leftarrow$ Argsort($S_{cos}$, ascending=True)

Return: $S_{ids}[:k]$

Algorithm 3 TopImpactfulFeatures

Input: The direction of impact $d\in\\{True,False\\}$, a differentiable model
$\mathcal{G}$, input features of a baseline sample $\hat{x}\in R^{D}$, input
features of a subset samples $X\in R^{k\times D}$, number of top features $l$.

$P\leftarrow$ ImpactMeasure($d$,$X$,$\hat{x}$,$\mathcal{G}$) # apply Equation
2 on all pairs of $\\{(\hat{x},x)|x\in X\\}$ for each latent state.

$P_{id}\leftarrow$ Argsort($P$, descending=$d$, axis=-1)

Return: $P_{id}[:,:,:l]$

Figure 2: A demonstration of identifying a behavioral pattern that is likely
related to the model output. We first identify the most important latent state
$z_{2}(t_{11})$ of the model output by SHAP value (the top one in the left
figure); we then get the top influential features of the top latent state (the
third row in the table) by the proposed method; finally, we can analyze the
correlations between the specific latent state and its top features, e.g.
using scatter plot to show how the top two features with positive impact
correlate to the value of the latent state (the right figure). The example
shows that it is more likely to get higher values of $z_{2}(t_{11})$ when
_lounge cum_ (the cumulative sum of activities in the lounge) is low and
_front door ctime_ (the cumulative active hours at the front door) is high.
Higher values of $z_{2}(t_{11})$ are likely to give a negative impact on the
model output, which indicates less likely being awake during night time.

###  3.3 Interpreting latent states of NCDE models

Assuming that there are latent states representing the underlying patterns of
time series, which depend on the cumulative influence from all previous
states, differential equations are often applied to find the dynamics of such
latent states and forecast future states.

####  3.3.1 Priliminary: Neural  (NCDE)

Neural  (NODE) (Chen et al., 2018) is a deep learning technique that can use
neural networks to approximate a latent state that has no explicit formulation
with its dynamics. For instance:
$z(t_{1})=z(t_{0})+\int_{t_{0}}^{t_{1}}f_{\theta}(z(t),t)dt$. Here $z(t)$
denotes latent states at time $t$, $f_{\theta}(\cdot)$ represents an arbitrary
neural network. NCDE (Kidger et al., 2020) uses a controlled different
equation, in which the dynamics of $z(t)$ are assumed to be controlled or
driven by an input signal $x(t)$.

$\begin{split}z(t_{1})&=z(t_{0})+\int_{t_{0}}^{t_{1}}\tilde{f}_{\theta}(z(t),t)dx(t)\\\
&=z(t_{0})+\int_{t_{0}}^{t_{1}}\tilde{f}_{\theta}(z(t),t)\frac{dx}{dt}dt,\\\
\end{split}$ (3)

where $\tilde{f}_{\theta}:R^{H}\rightarrow R^{H\times D}$, $H$ is the
dimension of $z$, $D$ is the dimension of $x$. The NODE is a special case of
NCDE when $dx/dt=I$. By treating $f_{\theta}(z(t),t)\frac{dx}{dt}$ as one
function $g_{\theta}(z(t),t,x)$, one can solve NCDE by regular methods for
solving NODE. The trajectory of control signals ($x(t)$) in NCDE can be fitted
independently with $\tilde{f}_{\theta}$, which enables NCDE to deal with
irregular sampled time series and work with online streaming data.

####  3.3.2 Integrated Jacobian of NCDE models

We cannot compute the integrated Jacobian of NCDE models readily by Equation 1
because $z\neq\mathcal{G}(x)$ in this case. Interestingly, we can obtain the
Jacobian ${\partial z}/{\partial x}$ easily when we i) use observations as
control signals; ii) $x(t)$ is approximated by an invertible function.
According to the chain rule:

$\begin{split}\frac{\partial z}{\partial x}=\frac{\partial z}{\partial
t}\frac{dt}{dx}=\left(\tilde{f}_{\theta}(z(t),t)\frac{dx}{dt}\right)\frac{dt}{dx}=\tilde{f}_{\theta}(z(t),t)\end{split}$
(4)

This way only needs a forward computation instead of a back-propagation to
obtain the Jacobian ${\partial z}/{\partial x}$.

##  4 Experiments

We conducted experiments on two different datasets: 1). one uses data from an
ongoing study of dementia care; 2) the other is the Sepsis dataset (Reyna et
al., 2019) from PhysioNet
111https://physionet.org/content/challenge-2019/1.0.0/. We use NCDE models for
both datasets. We also conducted feature augmentation by cumulative operations
on both datasets. For example: ”feature_ctime” is the cumulative number of
time steps that a specific feature has non-zero values, ”feature_cmax” is the
maximum value of a specific feature among all passed time steps,
”feature_csum” is the value sum of a specific feature over passed time steps.
Each time step represents one-hour information for both datasets.

###  4.1 Experiments with dementia care data

The data were collected from sensors deployed in 91 participants’ homes,
including Passive Infra-Red (PIR) sensors installed in multiple locations,
door sensors, and an under-the-mattress sleep mat. The dataset is not public
but a smaller dataset with similar properties is available on Zenodo
repository 222https://zenodo.org/record/7622128. We train an NCDE model using
household sensor data between 6:00 AM and 6:00 PM daily to predict if a
participant will be awake more than half of the time when he/she is in bed
during the night. Each data sample includes 12 time steps each of which has 36
features that are generated from sensor activations and feature augmentation.

The NCDE model learns four latent states at each time step and we concatenate
all latent states as the input to the final linear layer. Figure 1
demonstrates the heat maps for interpreting one of these latent states at each
time step. It illustrates how this latent state evolves over time. Combined
with SHAP values (Lundberg & Lee, 2017) of top 10 important latent states (the
left figure in Figure 2), we can discover subsets of features that jointly
affect the model prediction through latent states. In Figure 2, we demonstrate
how to identify a behavioral pattern using the aforementioned dataset by the
proposed method. The figure shows that it is more likely to get higher values
of $z_{2}(t_{11})$ when _lounge cum_ (the cumulative sum of activities in the
lounge) is low and _front door ctime_ (the cumulative active hours at the
front door) is high. Higher values of $z_{2}(t_{11})$ are likely to give a
negative impact on the model output, which indicates less likely being awake
during night time.

###  4.2 Experiments with Sepsis dataset

The Sepsis dataset (Reyna et al., 2019), applied in the PhysioNet/Computing in
Cardiology Challenge 2019, is a public dataset consisting of hourly vital
signs and lab data, along with demographic data, for 40366 patients obtained
from three distinct U.S. hospital systems. The objective is to predict sepsis
within 72 hours from the time a patient was admitted to the ICU. We use a
subset of this dataset which consists of data from 12000 patients. The task is
a binary classification of predicting the onset of sepsis. Therefore, each
data sample includes 72 time steps (hours) each of which has 136 features that
are augmented by 34 original features (vital signs and lab data). For patients
who have no records of the full 72 hours, we treated features of missing hours
as missing values filled by 0. All original features were preprocessed by min-
max normalization and shifted from the range [0,1] to the range [1,2].

The NCDE model learns two latent states at each time step. The SHAP values
plot in Figure 3(a) shows the 10 most impactful latent states on the sepsis
outcome. $z_{0}(t_{35})$ is the most influential latent state according to the
figure, which is the first latent state at the 36-th hour. Using our method to
link this latent state to the most influential input features, the top three
on $z_{0}(t_{35})$ are _SBP_csum_ , _O2sat_csum_ , and _Resp_csum_. We see
from Figures 3(b) and 3(c) that when values of _SBP_csum_ (cumulative sum of
Systolic BP), _O2sat_csum_ (cumulative sum of Pulse oximetry), and _Resp_csum_
(cumulative sum of Respiration rate) are all in the lower side,
$z_{0}(t_{35})$ is likely having a larger value and thus likely having a
positive impact on the prediction, i.e., increasing the likelihood of sepsis.

##  5 Conclusion and future work

We propose a method that can identify the most influential features of a
latent state, which may or may not be linearly correlated. Our method allows
linking model outcomes to the most contributing features, which brings
insights into understanding the knowledge that learned by the model. However,
after discovering influential features, we still need to manually analyze the
relations between those features, latent states, and model outputs. In order
to get more readily interpretable results, we consider developing algorithms
that can automatically discover interpretable patterns using top influential
features in the future. More future work can be done in multiple directions,
such as causal analysis of identified features, or probabilistic modeling on
joint distributions of those features.

(a) SHAP values of top 10 latent states on Sepsis dataset

(b) Scatter plot using values of _SBP_csum_ , _O2sat_csum_ at $t_{35}$, the
color indicates values of the top latent state $z_{0}(t_{35})$

(c) Scatter plot using values of _SBP_csum_ , _Resp_csum_ at $t_{35}$, the
color indicates values of the top latent state $z_{0}(t_{35})$

Figure 3: Interpreting the top latent state in Sepsis dataset

## Acknowledgment

This project was supported by the Engineering and Physical Sciences Research
Council (EPSRC) PROTECT Project (grant number: EP/W031892/1) and the UK DRI
Care Research and Technology Centre funded by the Medical Research Council
(MRC) and Alzheimer’s Society (grant number: UKDRI-7002).

## References

  * Chen et al. (2019) Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., and Su, J. K.  This looks like that: deep learning for interpretable image recognition.  _Advances in neural information processing systems_ , 32, 2019.
  * Chen et al. (2018) Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K.  Neural ordinary differential equations.  _Advances in neural information processing systems_ , 31, 2018.
  * Crabbé & van der Schaar (2022) Crabbé, J. and van der Schaar, M.  Label-free explainability for unsupervised models.  In _International Conference on Machine Learning_ , pp. 4391–4420. PMLR, 2022.
  * Crabbé et al. (2021) Crabbé, J., Qian, Z., Imrie, F., and van der Schaar, M.  Explaining latent representations with a corpus of examples.  _Advances in Neural Information Processing Systems_ , 34:12154–12166, 2021.
  * Datta et al. (2016) Datta, A., Sen, S., and Zick, Y.  Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems.  In _2016 IEEE symposium on security and privacy (SP)_ , pp. 598–617. IEEE, 2016.
  * Esser et al. (2020) Esser, P., Rombach, R., and Ommer, B.  A disentangling invertible interpretation network for explaining latent representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9223–9232, 2020.
  * Gu et al. (2021) Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and Ré, C.  Combining recurrent, convolutional, and continuous-time models with linear state space layers.  _Advances in neural information processing systems_ , 34:572–585, 2021.
  * Gurumoorthy et al. (2019) Gurumoorthy, K. S., Dhurandhar, A., Cecchi, G., and Aggarwal, C.  Efficient data representation by selecting prototypes with importance weights.  In _2019 IEEE International Conference on Data Mining (ICDM)_ , pp. 260–269. IEEE, 2019.
  * Hochreiter & Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Kidger et al. (2020) Kidger, P., Morrill, J., Foster, J., and Lyons, T.  Neural controlled differential equations for irregular time series.  _Advances in Neural Information Processing Systems_ , 33:6696–6707, 2020.
  * Li et al. (2018) Li, O., Liu, H., Chen, C., and Rudin, C.  Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Lundberg & Lee (2017) Lundberg, S. M. and Lee, S.-I.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30, 2017.
  * Reyna et al. (2019) Reyna, M. A., Josef, C., Seyedi, S., Jeter, R., Shashikumar, S. P., Westover, M. B., Sharma, A., Nemati, S., and Clifford, G. D.  Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019.  In _2019 Computing in Cardiology (CinC)_ , pp. Page–1. IEEE, 2019\.
  * Ribeiro et al. (2016) Ribeiro, M. T., Singh, S., and Guestrin, C.  ” why should i trust you?” explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pp. 1135–1144, 2016.
  * Sundararajan et al. (2017) Sundararajan, M., Taly, A., and Yan, Q.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , pp. 3319–3328. PMLR, 2017.
  * Tealab (2018) Tealab, A.  Time series forecasting using artificial neural networks methodologies: A systematic review.  _Future Computing and Informatics Journal_ , 3(2):334–340, 2018.  ISSN 2314-7288.  doi: https://doi.org/10.1016/j.fcij.2018.10.003.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.

  *[XAI]: Explainable Artificial Intelligence
  *[LSTM]: Long Short-Term Memory
  *[NCDE]: Neural
  *[NODE]: Neural


11institutetext: Department of Information Engineering and Computer Science
(DISI),
University of Trento, Italy
11email:
<EMAIL_ADDRESS>22institutetext: Conceptual and Cognitive Modeling Research Group (CORE),
Free University of Bozen-Bolzano, Bolzano, Italy
22email<EMAIL_ADDRESS>
#  iTelos \- Purpose Driven Knowledge Graph Generation

Fausto Giunchiglia  0000-0002-5903-6150 11 Simone Bocca  0000-0002-5951-4589
11 Mattia Fumagalli  0000-0003-3385-4769 22 Mayukh Bagchi  0000-0002-2946-5018
11 Alessio Zamboni  0000-0002-4435-1748 11

(March 2021)

###### Abstract

When building a new application we are more and more confronted with the need
of reusing and integrating pre-existing knowledge, e.g., ontologies, schemas,
data of any kind, from multiple sources. Nevertheless, it is a fact that this
prior knowledge is virtually impossible to reuse as-is. This difficulty is the
cause of high costs, with the further drawback that the resulting application
will again be hardly reusable. It is a negative loop which consistently
reinforces itself. iTelos111iTelos shares with Telos[14] the focus of the
purpose (as from the name), with the latter being a language focused on how to
represent it, the former being a methodology focused on how to exploit it in
order to build suitable applications. is a general purpose methodology aiming
at minimizing as much as possible the effects of this loop. iTelos is based on
the intuition that the data level and the schema level of an application
should be developed independently, thus allowing for maximum flexibility in
the reuse of the prior knowledge, but under the overall guidance of the needs
to be satisfied, formalized as competence queries. This intuition is
implemented by codifying all the requirements, including those concerning
reuse, as part of an a-priori defined purpose, which is then used to drive a
middle-out development process where the application schema and data are
continuously aligned.

###### Keywords:

Knowledge and data reuse Knowledge Graphs.

##  1 Introduction

Once upon a time, one would design an application top-down starting from the
requirements down to implementation, without thinking of the data: they would
be generated by the system in production with, at most, the need of
initializing them with data from the legacy systems being substituted.
Nowadays, more and more, we are designing systems which, at the beginning but
also when in production, must be integrated with data coming from other
systems, possibly from third parties. Some examples are the health systems
which integrate personal data coming from multiple institutions and B2C
applications exploiting the big data available on the Web, e.g., open data, or
streaming data.

The key aspect of this reuse problem is how to handle the semantic
heterogeneity which arises any time there is the need to perform data
integration across multiple sources. This problem has been extensively studied
in the past and two main approaches have been proposed. The first is using
ontologies to agree on a fixed language or schema to be shared across
applications [4]. The second is the use of Knowledge Graphs (KGs) and the
exploitation of the intrinsic flexibility and extensibility they provide [12],
as the means for facilitating the adaptation and integration of pre-existing
heterogeneous data. However the problem is far from being solved. When
developing an application, no matter whether one exploits ontologies or KGs,
it is impossible to reuse the pre-existing knowledge as-is. There is always
some specificity which makes the current problem in need of dedicated
development, with the further drawback that the resulting application is,
again, hardly reusable.

In this paper, we propose iTelos, a general purpose methodology whose main
goal is to minimize as much as possible the negative effects and the high
costs deriving from this loop. iTelos exploits all the previous results, in
particular, it is crucially based on the use of ontologies and KGs. At the
same time, iTelos takes a step ahead by providing a precise specification of
the process by which an application should be developed, focusing on how to
effectively reuse data from multiple sources. iTelos is based on three key
assumptions:

  * •

the data level and the schema level of an application should be developed
independently, thus allowing for maximum flexibility in the reuse of the prior
data and schemas, e.g., ontologies, but under the overall guidance of the
needs to be satisfied, formalized as competence queries;

  * •

Data and schemas to be reused, as well as competence queries, should be
decided before starting the development, as precisely as possible, and defined
a priori as part of an application purpose. Additionally, the purpose is
assumed to specify a set of constraints specifying how much the satisfaction
of each of its three elements is allowed to influence the satisfaction of the
other two;

  * •

the purpose should be used to drive a middle-out development process where the
successive evolutions of the application schema and data, both modeled as KGs,
are continuously aligned upwards (bottom-up) with the reference schemas to be
reused and downwards (top-down) with the data to be reused.

This paper is organized as follows. In Section 2 we provide a description of
the purpose structure. In Section 3 we describe the iTelos middle-out process.
In Section 4 we provide a detailed description of how iTelos enhances the
reusability of the available data. In Section 5 we describe how iTelos
enhances the sharability and future reusability of the application KG. In
Section 6 we provide the main alignment metrics used to ensure that the iTelos
middle-out process stays within the constraints specified by the purpose.
Section 7 describes a set of case studies by which iTelos has been
progressively evaluated and refined. Finally, Section 8 closes the paper with
the conclusions.

Figure 1: The iTelos approach.

##  2 The purpose

A very high level view of the iTelos process is depicted in Figure 1. The
iTelos logical sequence is represented by the dashed lines, where the User
provides an informal specification of the problem she wants to solve, the
Purpose, and receives in output a KG, called, in Figure 1, the Entity Graph.
The concrete factual process is represented by the four solid lines which
indicate the reuse of the prior knowledge, represented at the data lavel as
Datasets and at the schema level as Ontologies.

The purpose contains four main elements, as follows:

  * •

the functional requirements of the application, i.e., the needs to be
satisfied, that we assume to be ultimately formalized as competence queries
(CQ) [11].

  * •

a set of datasets to be reused, as encoded in pre-existing applications, that
we assume that, as part of the overall iTelos strategy, are progressively
developed as KGs, thus facilitating their future reuse.

  * •

a set of pre-existing reference schemas, ontologies but not only, whose reuse
will facilitate the development of common schemas shared by future
applications. It is important to notice that many such repositories are
already available; for instance, LOV, LOV4IoT, and DATAHUB,222See,
respectively: https://lov.linkeddata.es/, http://lov4iot.appspot.com/,
https://old.datahub.io/. are three among the most relevant repositories On top
of this, in order to facilitate the adoption of iTelos, a new repository,
called LiveSchema,333See: http://liveschema.eu/. is under construction, where
reference schemas are annotated by a very rich set of metadata, see, e.g., [3,
9], with the goal of automating as much as possible the iTelos process.

  * •

a set of metrics, whose goal is to define the boundaries within which any of
the three key elements of the purpose, viz. competence queries, reference
schemas, data to be reused, can be revised, in terms of use and reuse.

A crucial design decision in the structure of the purpose, which reflects into
the overall iTelos process, as depicted in Figure 1, is that the data level
and the schema level are kept distinct and independent, and that they are
represented via two different types of KG. This assumption is key, as it
allows to split the problem of reusing existing data from the problem of
facilitating the sharability and future reuse of the KG being developed. Data
level KGs, that we call Entity Graphs (EGs) are graphs where nodes are
entities (e.g., my cat Garfield), decorated with data property values
describing them, and where links are object properties which describe the
relations holding between any two such entities. Schema level KGs, that we
call entity type (etype) Graphs (ETGs), namely KGs which define the schema of
EGs. In other words, for each EG there is a corresponding ETG which defines
its schema. In ETGs nodes are etypes, namely classes of entities (e.g., the
class cat), each described by a set of data properties and by a set of object
properties which define the range of links allowed among the nodes of the EG
defined by the ETG.

Datasets and Ontologies in Figure 1 are, respectively examples of EGs and
ETGs. In iTelos EGs are used to represent: (i) the entity graph produced at
the end of the process (see Figure 1) and, optionally, depending on their
availability, (ii) the input datasets. ETGs and etypes are used to represent
(i) the schema of the output EG, (ii) the reference schemas and ontologies
and, also, ultimately, when fully formalized, (iii) Competency queries. This
uniformity of representation provides two major advantages. The first is that,
in practice, ETGs become syntactic sugar for a Description Logic (DL) TBOX
while, dually, EGs become (syntactic sugar for) a (DL) ABOX, thus allowing for
the usual DL reasoning. In particular, this allows us to see CQs as actual
queries to the EG. The second is that, this uniformity of representation at
the data and at the knowledge level can be used to implement, via an encoding
of ETGs and EGs in Formal Concept Analysis (FCA) [6], various Machine Learning
techniques which simplify the development of the output KG [9].

Figure 2: The iTelos Process.

##  3 The Process

Figure 2 instantiates the dashed line that in Figure 1 produces the Entity
Graph from the Purpose, into an overall process articulated in four phases.
The purpose of each such phase can be synthetically described as follows:

  * •

Inception: takes in input the user’s purpose and formalizes it, to the extent
that it is needed, into a set of CQs and a first validated collection of input
datasets and input ontologies;

  * •

Modeling: builds the ETG model by taking into account CQs and datasets;

  * •

Knowledge Alignment: builds a shareable ETG based on the model designed
previously, via the reuse of the selected reference ontologies;

  * •

Data Integration: builds the output EG by integrating the input datasets into
the ETG.

Let us analyze these phases in detail. The Inception phase takes as input the
purpose and collects the ontologies and datasets needed to build the target
EG. The main goal is to finalize the functional requirements while enabling
the reuse of existing resources. Because of this, the purpose is formalized,
starting from its initial natural language description, into a list of CQs.
CQs are then matched to the input datasets and ontologies thus implementing
the activities of Ontology Collection and Dataset Collection, as from Figure
2. Depending on the process, the final list of collected datasets and
ontologies may extend and also partially overlap with the input list. The key
observation is that matching CQs and (the schemas of the) datasets is crucial
for the success of the project. A little coverage would mean that there are
not enough data for the implementation of the EG and that there should be a
revision of the CQs; whenever this is the case, the process we follow is
inspired by the work in [10].

The Modeling phase receives in input the ontologies and datasets previously
collected, as well as the CQ list. The main objective of this phase is to
build the most suitable model of the ETG to be used as the schema of the final
EG, which in Figure 2 is called the ETG model. In practice, the ETG model
includes all the etypes and properties needed to represent the information
required by each CQ, possibly extended by extra etypes and properties
suggested by the datasets. This extension is optional but suggested to allow
for future expansions, given that the availability of data would make this
step low cost, in particular, if taken into account since the early stages.
The main objective of ETG Modeling activity is to compose such etypes and
properties into an EER model, essentially transforming the CQ list into an
ETG. In parallel, the Dataset Selection activity finalizes the selection of
those datasets which are effectively required, pruning away useless data
resources.

The Knowledge Alignment phase takes as input the ETG model previously
generated, plus the selected set of datasets and reference ontologies. The
main objective of this phase is to enhance the sharability of the final EG,
building in turn a shareable ETG that fits in the best possible way the
datasets to be integrated. The input ETG model is itself a possible solution.
However the set of reference ontologies provides more possibilities, in terms
of etypes and properties available, that can be adopted to implement a final
ETG (called ETG in Figure 2) easier to share and reuse. The ETG Alignment
activity is implemented via the Machine Learning algorithm described in [9].
Such an algorithm takes as input the set of reference ontologies and the ETG
model and it outputs the final version of the ETG. The resulting ETG is
verified for compliance with the input datasets before the final approval. In
parallel the Dataset Cleaning activity performs the final cleaning of the
datasets consistently with the ETG. Here, the main objective is the alignment
of the data types and formats of the datasets, with the etypes and properties
of the ETG.

The last phase is Data Integration. Its input consists of the ETG and the
cleaned datasets. The objective is to build the EG integrating the schema and
data resources, as adapted to the purpose in the previous phases. Unlike the
previous ones, the data integration phase consists of a single activity, that
we call EG Generation, that merges together the schema and data layers
creating the final EG. To do that, the ETG and datasets are provided in input
to a specific data mapping tool, called KarmaLinker, which consists of the
Karma data integration tool [13] extended to perform Natural Language
Processing on short sentences (i.e., what we usually call the language of
data) [1]. The first activity in this phase maps the data to the etypes and
properties of the ETG. The following step is the generation of the entities
that are then matched and, whenever they are discovered to be different
representations of the same real world entity, merged. These activities are
fully supported by Karmalinker. The above process is iteratively executed over
the list of datasets selected in the previous phase, processed sequentially.
The process concludes with the export of the EG into an RDF file.

The key observation is that the desired middle-out convergence, as described
in the introduction, has been implemented in two separate sub-processes,
executed in parallel within each phase, one operating and the schema level,
the other at the data level (blue and green boxes in Figure 2). During this
process, the initial purpose keeps evolving building the bridge between CQs,
datasets and ontologies. To enforce the convergence of this process, and also
to avoid making costly mistakes, each phase ends with an evaluation activity
(Eval boxes in Figure 2). The specifics of this activity are described in
Section 6. Here it is worth making two observations. The first is that this
evaluation is driven by the non-functional requirements provided by the
purpose. The second is that, within each phase, the evaluation aims to verify
that the target of that phase is met, namely: aligning CQs with datasets and
ontologies in phase 1, thus maximizing reusability; aligning the ETG model
with the datasets in phase 2, thus guaranteeing the success of the project;
and aligning ETG and ontologies in phase 3, thus maximizing sharability. The
evaluation in phase 4 has the goal of checking that the final EG satisfies the
requirements specified by the purpose. As from Figure 2, a failure of the
evaluation in any of the steps causes the process to go back to the evaluation
step of the previous phase. In the extreme case of a major early design
mistake it is possible to go back from phase 4 to phase 1.

##  4 Enhancing data reuse

Reusability is enhanced during the phases of inception and modeling, whose
main goal is to progressively transform the specifications from the purpose
into the ETG Model. This process happens according to the following steps:

  1. 1.

generation of a list of natural language sentences, each informally defining a
CQ, as implicitly or explicitly implied by the purpose;

  2. 2.

generation of a list of relevant etypes and corresponding properties, which
formalize the informal content of CQs, as from the previous step;

  3. 3.

selection of the datasets whose schema informally matches the CQs, as from the
previous step;

  4. 4.

generation of a list of etypes with associated properties, from the selected
datasets, which match the etypes and properties from the CQs;

  5. 5.

construction of the ETG model;

Steps 1-4 happen during inception, while step 5 happens during the modeling
phase. Most of the work is definitely done during inception, while modeling
just exploits and selects from the choices made during the previous phase
with, possibly, the opportunity of backtracking in case the evaluation does
not produce the desired results. The key observation is that, in both phases,
the three types of resources involved (i.e., CQs, ontologies, datasets) are
handled through a series of three iterative executions, each corresponding to
a specific category, following a decreasing level of reusability. The
categories are defined as follows:

Common: this category involves resources associated with aspects that are
common to all domains, also outside the domain of interest. Usually, these
resources correspond to abstract etypes specified in upper level ontologies
[5], e.g., person, organization, event, location, and/or to etypes from very
common domains, usually needed in most applications, e.g., Space and Time.
These latter types of etypes and properties correspond to what in knowledge
organization are called Common Isolates [15]. The data that are found in Open
Data sites as well the ontologies which can be found in the repositories
mentioned above are examples of common resources.

Core: this category involves resources associated with the more core aspects
of the domain under consideration. They carry information about the most
important aspects considered by the purpose, information without which it
would be impossible to develop the EG. Consider for instance the following
purpose:

”There is a need to monitor Covid-19 data in the Trentino region (Italy), to
understand the diffusion of the virus and the social restriction caused by the
virus, with the possibility to identify new outbreaks.”444This example, as
well as all the follow-up material, as described below in the paper, has been
extracted from the project with title ”Integration of medical data on
Covid-19” developed by Antony, N., Gotca, D., Jyate, M., Donini, L. The
complete material and description of this project can be found at the URL
https://github.com/UNITN-KDI-2020/COVID-data-integration.

In this example, core resources could be those data values reporting the
number of Covid-19 infections in the specified region. Examples of common
resources are the data of certain domains, e.g., public sector facilities
(e.g., hospitals, transportation, education), domain specific ontologies that
can be found again in the repositories above, as well as domain specific
standards (e.g., Health, interoperability standards of various types). In
general, data are harder to find than ontologies, in particular when they are
about the private sector, where they carry economic value and are often
collected by private bodies.

Contextual: this last category involves resources that carry specific,
possibly unique, information from the domain of interest. These are the
resources whose main goal is to create added value. If core resources are
necessary for a meaningful application, contextual resources are the ones
which can make the difference with respect to the competitors. In the above
example, examples of contextual resources can be those data describing the
type of social restrictions adopted to contrast the virus. At the schema
level, contextual etypes and properties are those which differentiate the
ontologies which, while covering the same domain, actually present major
differences. [8] presents a detailed quantitative analysis of how to compare
these ontologies. Data level contextual resources are usually not trivial to
find, given their specificity and intrinsic. In various applications we have
developed in the past, this type of data have turned out to be a new set of
resources those had to be generated on purpose for the application under
development, in some cases while in production.

The overall conclusive observation is that the availability of resources, and
of data in particular, decreases from top to bottom. In parallel to
reusability and sharability as well as, to a large extent, also to a decrease
in cost, going from more infrastructural data to more application specific
data. As performed during the evaluation, the consequent considerations on
feasibility are among the drivers that guide the development of the EG.

Table 1: CQs categorized in the three reusability categories.

Number Question Action Category 1 How many cases in schools in Trentino?
Return the number of school cases in Trentino region Contextual 2 Are there
schools that are closing? Return whether schools are closed in Trentino region
Contextual 3 Will the number of cases increase in Italy? Return the infection
prediction information in Italy Core 4 How many cases are in the
RSA555Residenza Sanitaria Assistenziale, Italian equivalent of extended care
facilityin Trentino region? Return number of RSA cases in Trentino region
Contextual

Let us see an example of how the six elaboration steps listed at the beginning
of the section intermix with the three categories above. Table 1 shows some
CQs, extracted from the case study introduced above (step 1). Notice how,
already in this step, CQs are categorized as being common, core or contextual
(last column) and how this is done after translating the text from the purpose
(left column) into something much closer to a requirement for the EG (central
column). Table 2 then reports the etypes and properties extracted from the CQs
(step 2). While this is not reported in the table for lack of space, these
etypes and properties inherit the category from the CQs. Notice that it is
possible to have an etype that is core or common or contextual with properties
in all three categories; this being a consequence of the fact that an etype
can be mentioned in multiple (types of) CQs.

Table 2: Etypes and properties from the CQs.

CQ Number Etypes Properties 1 Covid_status, Location date,
total_number_of_cases, number_of_active_cases, number_of_new_positive_cases,
number_of_deaths, number_of_recovered_cases 2 Restriction, Location
location_type, restriction_type, closure_start, closure_end 3
Case_projections, Case_information, Location location_type, date,
mean_of_est.infections, lower_bound_of_est.infections,
upper_bound_of_est.infections 4 RSA_cases date, number_of_RSA_case,
number_of_home_care_cases

The next step is to select the datasets (step 3). Let us assume that the
dataset used to (partially) answer the queries in Table 1 generates the
properties matching the CQs as in Table 3 (step 4). As an example of matching,
compare the attribute cases in Table 3 with the property total_number_of_cases
of CQ n°1 in Table 2. Notice how the ordering of the analysis (from common to
core to contextual) creates dependencies that may drive the choice of one
dataset over another. As an example, School might be needed as a core etype,
but to properly define it, we also need the common etype Location. The final
ETG, as constructed, during steps 5, is reported in Figure 3, containing
etypes and properties from the tables above.

Table 3: Dataset’s attributes classified according to the reusability
categories.

Attributes Description Type Category cases Number of new cases on the date of
the report int Core deaths Number of deaths on the date of the report int Core
countriesAndTerritories Country and territory name related to the records
string Contextual year Year from the date of report int Common month Month
from the date of report int Common day Day from the date of report int Common

##  5 Enhancing data sharability

The knowledge alignment phase aims to enhance sharability, by aligning and
possibly modifying the ETG model to take into account the etypes and
properties coming from the reference ontologies. The key observation is that
the alignment mainly concerns the common and, possibly, the core types with
much smaller expectations on contextual etypes. This is where the repositories
listed in Section 2 can make the difference. Notice that, in retrospect, the
alignment with the most suitable ontology can enable the reuse of data, at
least for what concerns common and sometimes core etypes. As an example, the
selection of Google GTFS or FOAF as reference ontologies ensures the
availability of a huge amount of data, a lot of which are open data. This type
of decision should be made during inception; if discovered here, it might
generate backtracking.

We align the ETG model with the reference ontologies, by adapting the Entity
Type Recognition (ETR) process proposed in [9]. This process happens in three
steps as follows:

Step 1: ontologies selection. This step aims at selecting the set of reference
ontologies that best fit the ETG model. As from [9], this selection step
occurs by measuring each reference ontology according to two metrics, which
allow:

  * •

to identify how many etypes of the reference ontologies are in common with
those defined in the ETG model, and

  * •

to measure a property sharability value for each ontology etype, indicating
how many properties are shared with the ETG model etypes.

The output of this first step is a set of selected ontologies, which best
cover the ETG model, and that have been verified fitting the dataset’s schema,
at both etypes and etype properties levels.

Figure 3: The example ETG.

Step 2: Entity Type Recognition(ETR). The main goal here is to predict, for
each etype of the ETG model, which etype of the input ontologies, analyzed one
at the time, best fits the ETG. In practice, the ETG model’s etypes are used
as labels of the classification task and, as mentioned in [9], the execution
exploits techniques that are very similar to those used in ontology matching
(see, e.g., [7]). The final result is a vector of prediction values, returning
a similarity score between the ETG model’s etypes and the selected ontology
etypes.

Figure 4: Portion of the Codo reference ontology.

Step 3: ETG generation. This step identifies, by using the prediction vector
produced in the previous step, those etypes and properties from the ontologies
which will be added to the final version of the ETG. Notice how this must be
done while preserving the mapping with the datasets’ schemas; whenever this is
problematic, this becomes a possible source of backtracking. The distinction
among common, core and contextual etypes and properties plays an important
role in this phase and can be articulated as follows:

  * •

The common etypes should be adopted from the reference ontology, in percentage
as close as possible to 100%. This usually results in an enrichment of the top
level of the ETG model by adding those top level etypes (e.g., thing, product,
event, location) that usually no developer considers, because too abstract,
but which are fundamental for building a highly shareable ETG where all
properties are positioned in the right place. This also allows for an
alignment of those common isolates (see Section 4) for which usually a lot of
(open) data are publicly available (e.g., street);

  * •

The core etypes are tentatively treated in the same way as common etypes, but
the results highly depend on the ontologies available. Think for instance of
the GTFS example above;

  * •

Contextual etypes and, in particular, contextual properties are mainly used to
select among ontologies, the reason being that, the contextual properties
allow distinguishing the most suitable among a set of ontologies about the
same domain [9].

As an example, compare the ETG as from Figure 3, with the portion of the
reference ontology Codo [2] in Figure 4.666 In this project, the matching
between CODO and the ETG model has been done manually. As it can be noticed,
the two Codo etypes Place and Statistics can be matched to the ETG etypes
Location and CasesInformation via the ETG’s etype LocationType. In this
project the ETG model in Figure 3 was adopted as the final ETG, as the authors
preferred their original terminology and organization, over that of the
selected reference ontology, which was allowed by the purpose they were
provided.

##  6 Alignment Evaluation

The evaluation is based on a set of metrics that are applied to CQs, ETGs,
datasets, and dataset schemas, where the latter - when they are not KGs - are
treated as sets of properties associated with one or more selected etypes of
the ETG. These metrics are applied in the four phases of the iTelos process.
For lack of space, below we describe only the application of three metrics to
a specific phase.

Let $\alpha$ and $\beta$ be two generic element sets, where an element can be
an etype or a property. We have the following:

Coverage. This metric allows to ascertain the extent to which an element set
is appropriate for the purpose, and it is computed as the ratio between the
intersection of $\alpha$ and $\beta$ and the whole $\alpha$.

$Cov=(\alpha\cap\beta)/\alpha$ (1)

For instance, during the inception phase (Eval(a) in Figure 2), $Cov$ plays a
central role in evaluating the _reusability_ of potential datasets (via
dataset schemas) with respect to the CQs. For each dataset, a high value of
$Cov$, both applied to etypes and properties, implies that the dataset is
highly appropriate for the purpose. A low value of $Cov$ implies minimal
overlap between the purpose and the dataset, the consequence being non-
consideration of the dataset for reuse, and possible modification of the
(underspecified) CQs.

Extensiveness. This metric quantifies the proportional amount of knowledge
provided by any element set (such as $\beta$), in terms of set of etypes or of
sets of properties, with respect to the entire knowledge considered (here
$\alpha$ and $\beta$)

$Ext=(\beta-(\alpha\cap\beta))/((\alpha+\beta)-(\alpha\cap\beta))$ (2)

During the Modeling phase (Eval(b) in Figure 2), the evaluation utilizes $Ext$
to ensure that the ETG model maximally extends the set of CQs, which makes the
model easily amenable for further evolution. To that end, a high value of
$Ext$ is evaluative of the fact that the ETG extends the scope of the CQs, by
indicating a limited contribution of the CQs in generating the ETG model. On
the other hand, low values of $Ext$ are evaluative of the fact that CQs have
contributed significantly towards enhancing the ETG model.

Sparsity. This metric quantifies the element-level difference between any
number of similar element sets, and is defined as the sum of the percentage of
$\alpha$ not in $\beta$, and vice versa.

$Spr=((\alpha+\beta)-2(\alpha\cap\beta))/((\alpha+\beta)-(\alpha\cap\beta))$
(3)

In the knowledge alignment phase (Eval(c) in Figure 2), our principal focus is
to utilize $Spr$ for ensuring the _sharability_ of the ETG. We incrementally
enforce sharability by ensuring a required threshold of $Spr$ between the ETG
and each of the reference ontology, which indicates that the ETG contains
axioms reflective of _contextual knowledge_.

All the above metrics operate at the schema level. But they do not say
anything about the results of integrating the datasets, as caused by the
semantic heterogeneity existing among them. The situation is as follows.
Assume that $D1$ and $D2$ are two datasets. Let us assume that, using
KarmaLinker, we analyze first $D1$ and then $D2$. Then we have the following
possible situations:

  * •

$D2$ contains an etype which is also contained by $D1$. In this case the EG is
not extended; only the number of links across nodes is increased. We have two
situations depending on whether an entity populates both datasets or only one.
In the first case the entity will be enriched with a new set of property
values with the possibility of contradictory information. In the second case,
the two different sets of entities will values only for the properties shared
between $D1$ and $D2$. This case will result in an EG with a lot of missing
links.

  * •

$D2$ contains a new etype with respect $D1$. In this case, the EG is extended
with new nodes and new links. We have again two situations depending in
whether an entity populates both datasets or only one. In the first case the
EG is nicely extended by attaching new links to the entities in $D1$. In the
second case the two parts of the EG resulting from the import of the two
datasets will be not connected.

The data driven metrics briefly introduced above are crucial for the
evaluation of the quality of the final EG. At the moment, these
characteristics of the EG have been evaluated with a set of techniques ad hoc.
A set of metrics for the evaluation of the quality of the final EG are under
definition.

##  7 Case studies

We have extensively evaluated and revised iTelos during the past three years
(2018, 2019, 2020) as part of the Knowledge and Data Integration (KDI) class,
a six credit course of the Master Degree in Computer Science of the University
of Trento.777See https://unitn-kdi-2020.github.io/unitn-kdi-2020/ for more
details. This site contains the material used during the 2020 edition of the
course and it consists of theoretical and practical lectures, as well as demos
of the tools to be used, some of which have been mentioned above. Table 4
reports the information regarding the population involved and a number of
projects. During this class, 2-5 students per group, must generate an EG using
the pipeline above starting from a high level problem specification. Part of
the task is also to identify the most suitable datasets and pre-process them.
Datasets are usually found in open data repositories, but some are also
scraped from the Web. The overall project has an elapsed time of 14 weeks
during which students have to work intensely, even not full time. We estimate
the overall effort each group puts into building an EG in around 4-8 person-
months, depending on the case. As of today we have piloted 24 projects and 75
evaluations.

Table 4: Evaluation’s subjects in KDI class - 2018, 2019, 2020. | 2018 | 2019 | 2020 | Tot
---|---|---|---|---
# Students | 29 | 20 | 26 | 75
# Project teams | 14 | 4 | 6 | 24
% Male | 69% | 75% | 95% | 59
% Female | 31% | 25% | 5% | 16
% Undergraduate | 20.7% | 20% | 25% | 16.5
% Post graduate | 79.3% | 80% | 75% | 59

The summary of the quantitative evaluation is reported in Figure 5. We show
three heat maps containing the answers to the seven quantitative questions
submitted to users. All questions foresaw a scaled rating within the range
$[1,7]$ where $1$ represents the most positive score and $7$ is the most
negative. Our aim was to observe if, throughout the entire three-years period,
the overall evaluation of the methodology revealed an increasing appreciation
from the users. This point is validated by observing the heat maps related to
the years 2018 and 2019. Indeed, the reader can appreciate the scores shifted
to the left part of the heat map. Differently, from the evaluation collected
in 2020, we observed a worsening of the scores (i.e. shift to the right). By
combining the quantitative evaluation with the analysis of the qualitative
evaluation, we concluded that during the first two years, we applied the
iTelos methodology within an offline context where the face-to-face
interaction was predominant. Instead in the last year, given the pandemic, we
applied entirely the iTelos within an online context for the first time. This
was an important test-bed since one of the long term objectives of iTelos is
to support the distributed collaboration between people in creating and
deploying high quality knowledge graphs. Hence, on the one hand, we observed
how the refinement process of the methodology we setup for finding the best
trade-off between effectiveness and efficiency in building knowledge graphs is
proper. On the other hand, the deployment of the methodology within a complete
online context triggered the need for further refinements for increasing its
overall usability.

Figure 5: Usability of the methodology: (a) 2018, (b) 2019, (c) 2020

iTelos users have also been asked to answer a questionnaire containing a set
of qualitative queries to collect more fine-grained feedback. Below is a short
description of the most important results, organized in strengths and
weaknesses:

  * •

(Strength) the step by step, precisely articulated, iTelos process is easy to
follow;

  * •

(Strength) somewhat opposite to our expectations, the stepwise iterative
evaluation process was perceived positively as a way for improving the overall
outcome of the project as well as a support for the refinement of the Entity
Graph;

  * •

(Strength) Two projects, carried out by PhD students, were also developed as
part of EC projects, and required interaction with third party data providers.
It turned out that the iTelos process adapts well to the cooperative
specification among multiple parties, in particular in the case of data
providers and KG developers;

  * •

(Weakness) A wrong decision made in the early phases is quite difficult to
remedy, with this possibility being very high during the inception phase;

  * •

(Weakness) The work between the schema and the data layer is unbalanced in
favour of the second, in particular during the informal modeling phase. This
complicates the synchronization of the work with the possibility of
misalignments, mainly because of misunderstandings, that have to be handled
very carefully by the project manager.

##  8 Conclusions

In this paper we have introduced iTelos, a novel methodology whose ultimate
goal is to implement a circular development process. By this we mean that the
goal of iTelos is to enable the development of EGs via the reuse of already
existing EGs and ETGs, while being simultaneously developed to be later easily
reused by other applications to come.

Acknowledgements The research conducted by Fausto Giunchiglia, Mayukh Bagchi
and Simone Bocca has received funding from the _“DELPhi - DiscovEring Life
Patterns”_ project funded by the MIUR (PRIN) 2017. The research conducted by
Alessio Zamboni was supported by the _InteropEHRate_ project, EC Horizon 2020
programme under grant number 826106.

## References

  * [1] Bella, G., Zamboni, A., Giunchiglia, F.: Domain-based sense disambiguation in multilingual structured data. In: DIVERSITY Workshop at ECAI 2016 (2016)
  * [2] Dutta, B., DeBellis, M.: Codo: an ontology for collection and analysis of covid-19 data. (2020), in Proc. of 12th Int. Conf. on Knowledge Engineering and Ontology Development (KEOD), 2-4 November 2020
  * [3] Dutta, B., Nandini, D., Shahi, G.K.: Mod: metadata for ontology description and publication. In: International Conference on Dublin Core and Metadata Applications. pp. 1–9 (2015)
  * [4] Ekaputra, F., Sabou, M., Serral Asensio, E., Kiesling, E., Biffl, S.: Ontology-based data integration in multi-disciplinary engineering environments: A review. Open Journal of Information Systems 4(1), 1–26 (2017)
  * [5] Gangemi, A., Guarino, N., Masolo, C., Oltramari, A., Schneider, L.: Sweetening ontologies with dolce. In: International Conference on Knowledge Engineering and Knowledge Management. pp. 166–181. Springer (2002)
  * [6] Ganter, B., Wille, R.: Formal concept analysis: mathematical foundations. Springer (2012)
  * [7] Giunchiglia, F., Autayeu, A., Pane, J.: S-match: an open source framework for matching lightweight ontologies. Semantic Web 3(3), 307–317 (2012)
  * [8] Giunchiglia, F., Fumagalli, M.: Teleologies: Objects, actions and functions. In: ER - International Conference on Conceptual Modeling. pp. 520–534 (2017)
  * [9] Giunchiglia, F., Fumagalli, M.: Entity type recognition – dealing with the diversity of knowledge. In: Knowledge Representation Conference (KR). Rhodes, Greece (2020)
  * [10] Grüninger, M.: Ontology validation as dialogue (2019)
  * [11] Grüninger, M., Fox, M.S.: The role of competency questions in enterprise engineering. In: Benchmarking—Theory and practice, pp. 22–31. Springer (1995)
  * [12] Kejriwal, M.: Domain-specific knowledge graph construction. Springer (2019)
  * [13] Knoblock, C.A., Szekely, P.: Exploiting semantics for big data integration. Ai Magazine 36(1), 25–38 (2015)
  * [14] Mylopoulos, J., Borgida, A., Jarke, M., Koubarakis, M.: Telos: Representing knowledge about information systems. ACM Transactions on Information Systems (TOIS) 8(4), 325–362 (1990)
  * [15] Ranganathan, S.R.: Prolegomena to library classification. Asia Publishing House London, 3rd ed. edn. (1967)