ANNETR | 0.712 | 0.507 | 0.732 | 0.659 | 0.613 | 0.528 | 0.702 | 0.491

####  5.5.4 Ablation Study

In this section, we conduct ablation studies to substantiate the efficacy of
specific components incorporated within our entity type recognition framework.

##### Effect of Similarity Metrics

The first ablation study is to evaluate if each of the proposed property-based
similarity metrics is effective. In this experiment, we test the
backbones777trained by all three metrics ($Sim_{V},Sim_{H},Sim_{I}$) (B) which
were used in the schema-level and instance-level recognition tasks,
respectively. Based on the backbones, we also design a controlled group that
includes models trained without one of the property-based similarity metrics
(i.e. B-SimV, B-SimH and B-SimI) and models trained without all metrics (i.e.
B-L). If the backbones perform better than the corresponding models in the
controlled group, we can quantitatively conclude that each of the property-
based similarity metrics ($Sim_{V},Sim_{H},Sim_{I}$) contributes to the entity
type alignment and recognition tasks. Table 5.4 demonstrates the
$F_{1}$-measure of each group. We apply ConfTrack and EnTypeGen for the
recognition cases. Note that we select two models for each case as Table 5.4
shows. We find that backbones perform better than models in the controlled
group, especially for models trained without all metrics. Thus, we consider
all property-based similarity metrics to contribute to better recognition
performance. Particularly, SimV and SimH significantly affect the performance
of schema alignment cases, and SimI affects instance-level cases more.

Table 5.4: Ablation study on property-based similarity metrics. Dataset | Model | Backbone |  B-SimV |  B-SimH |  B-SimI | B-L
---|---|---|---|---|---|---
ConfTrack |  ANNETR | 0.713 | 0.635 | 0.639 | 0.660 | 0.618
XGBoostETR | 0.740 | 0.648 | 0.655 | 0.694 | 0.632
EnTypeGen |  ANNETR | 0.537 | 0.327 | 0.391 | 0.309 | -
XGBoostETR | 0.559 | 0.413 | 0.402 | 0.385 | -

##### Effect of Constraint Factors

In section 4.4.2, we introduced a constraint factor, denoted as $\lambda$, for
the computation of the metric $Sim_{H}$. The objective of this study is to
statistically ascertain the optimal value for $\lambda$. Utilizing the
ConfTrack dataset and its two most effective models, we set the value of
$\lambda$ at regular intervals ranging from 0.1 to 1, represented by discrete
points. Our evaluation focuses on determining whether this pre-defined factor
influences the ultimate performance in entity type recognition and deducing
the most suitable value of $\lambda$ for generic entity type recognition
tasks. Table 5.5 exhibits these results, with a special emphasis on both the
best and second-best outcomes. It is observed that varying values of $\lambda$
indeed impact the final performance in entity type recognition. Both models
display a similar trend, indicating that the optimal value of $\lambda$
approximates 0.5. Consequently, we set $\lambda=0.5$ for the calculation of
the metric $Sim_{H}$ in our experimental framework.

Table 5.5: Ablation study on the constraint factor $\lambda$. The best and second-best results are highlighted in  red and  blue, respectively. Factor | Model | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9
---|---|---|---|---|---|---|---|---|---|---
$\lambda$ |  ANNETR | 0.613 | 0.638 | 0.670 | 0.678 | 0.712 | 0.685 | 0.679 | 0.653 | 0.630
XGBoostETR | 0.662 | 0.677 | 0.714 | 0.727 | 0.729 | 0.716 | 0.654 | 0.711 | 0.705

###  5.6 Summary

In this chapter, we introduce a machine learning-based framework for achieving
entity type recognition. Initially, we outline the comprehensive structure of
this framework, detailing its constituent components: the knowledge graph
parser, the formalization module, the similarity calculation module, the
property matcher, and the entity type recognizer. Subsequently, we explore a
variety of machine learning algorithms aimed at enhancing knowledge extraction
performance, encompassing both several classical algorithms and neural
networks. Following this, we discuss the training strategy for the critical
entity type recognizer, as well as the data processing pipeline. Finally, we
assess the effectiveness of our framework across multiple scenarios using
various datasets. When compared to state-of-the-art methods, the experimental
outcomes affirm the validity of our similarity metrics and the superiority of
our proposed recognition framework, both quantitatively and qualitatively.
Additionally, ablation studies are conducted to further validate the efficacy
of the proposed similarity metrics and to ascertain the optimal parameter
settings for the recognition framework.

##  Chapter 6 Knowledge Graph Assessment

###  6.1 Introduction

In this chapter, we introduce the knowledge graph assessment methods,
contributing to the evaluation step of our proposed knowledge graph extension
framework. This study is based on our work presented in [50].

Knowledge graph assessment is a pivotal procedure in semantic computing and
data science, focusing on evaluating the accuracy, completeness, and
reliability of knowledge graphs. Such assessments are fundamental for
applications including search engines, recommendation systems, and natural
language processing tasks. The assessment methods are also applied to reduce
the effort in constructing knowledge graphs and to promote their reuse [85].

Significant efforts have been made in the functional, or purpose-driven,
evaluation of knowledge graphs, which especially pertains to their purpose-
based ranking [22]. The primary challenge here is to quantify the purpose-
driven features of a knowledge graph, which are often heavily reliant on
context and application [51]. For example, consider the critical significance
of the term “core” in describing entity types and their varied interpretations
[42]. With the growing abundance of knowledge graphs, the necessity to define
methodologies for their reuse intensifies [34], especially in emerging
application domains, such as relational learning [93] and transfer learning
[48]. This increasing proliferation of knowledge graphs calls for refined
approaches to ensure their efficient and effective reuse in diverse contexts.

Following contemporary psychology, the purpose of categorization can be
reduced to “…a means of simplifying the environment, of reducing the load on
memory, and of helping us to store and retrieve information efficiently” [112,
63]. This process involves grouping entities based on shared features or
similarities, where categorization typically employs well-defined and
effective constructs such as knowledge graphs. Accordingly, we introduce a
quantifiable and deterministic way to assess knowledge graphs according to
their categorization purpose. We measure the categorization purpose as a novel
perspective of knowledge graph assessment, via metrics that we ground on the
notion of categorization defined in cognitive psychology [86].

Thus, we propose a set of metrics called focus to assess knowledge graphs from
their categorization purpose. We will introduce the details of how focus can
be used to rank: i). the concepts inside a knowledge graph which are more/less
informative; ii). the concepts across multiple knowledge graphs which are
more/less informative; and iii). the knowledge graphs which are more/less
informative. We also discuss existing assessment metrics that are used to
evaluate knowledge graphs from dimensions including completeness, consistency,
and data quality. Lastly, to test the utility of the focus measures, we show
how focus metrics can be used to support engineers in measuring the relevance
of knowledge graphs. That is, a). we verify how the knowledge graphs ranking
provided by the focus metrics reflects the ranking of the knowledge graphs
provided by a group of knowledge engineers, according to guidelines inspired
by a well-known experiment in cognitive psychology; b). we verify how focus
can help scientists in selecting better knowledge graphs for supporting the
training process in machine learning tasks.

###  6.2 Existing Assessment Metrics

Effective assessment of knowledge graphs is crucial for ensuring that they
serve their intended purpose accurately and efficiently in various application
scenarios. Assessment methodologies often employ a mix of automated tools and
manual inspection, especially when comparing the graph’s data against a gold
standard or benchmark dataset. We introduce several existing ranking metrics
as baseline measurements, namely class match measure (CMM) [1], density
measure (DEM) [1], Okapi best matching measure (BM25) [21] and term frequency-
inverse document frequency measure (TF-IDF) [117].

  1. 1.

CMM: The class match measure is designed to assess the extent to which a
knowledge graph encompasses given search terms by identifying entity types
with labels that either exactly match (where the label is identical to the
search term) or partially match (where the label contains the search term) the
search term. This measure posits that a knowledge graph encompassing all the
specified search terms will attain a higher score compared to others, and
prioritizes exact matches over partial ones. For example, when searching for
student a knowledge graph with an entity type labeled precisely as Student
will be rated higher in this measure compared to other knowledge graphs that
include partial matches, such as PhDstudent. The CMM measure can be
mathematically represented by:

$CMM(K,Q)=\alpha score_{EMM}(K,Q)+\beta score_{PMM}(K,Q)$ (6.1)

where $score_{EMM}(K,Q)$ and $score_{PMM}(K,Q)$ represent the scores
attributed to exact alignments and partial alignments within the knowledge
graph $K$ concerning the query $Q$, variables $\alpha$ and $\beta$ serve as
weight factors for these two distinct aligning states. Given that exact
aligning is preferred over partial aligning, the typical configuration sets
$\alpha$ at 0.6 and $\beta$ at 0.4.

  2. 2.

DEM: In the context of evaluating an effective representation of a particular
concept, people would expect to find a certain degree of detail in the
representation of the knowledge pertaining to that concept. The density
measure is conceived to estimate the representational density or the
information content inherent in classes, thereby reflecting the depth of
knowledge details. Currently, the calculations for density are confined to
aspects such as the number of subclasses, the number of properties associated
with the concept, the count of sibling concepts, and similar elements. The DEM
measure can be represented as:

$DEM(K)=\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{p}w*M(S_{j}(c_{i}),q)$ (6.2)

where $M(S_{j}(c_{i}),q)$ represents the aligning state between the query $q$
and the aspect $S_{j}(c)$ of the concept $c$, notice that $S_{j}$ is selected
from $S=\\{relations(c),superclasses(c),subclasses(c),siblings(c)\\}$, and the
factor $p$ is the number of items in set $S$, $n$ refers to the number of
alignments, and $w$ is the weight factor set to a default value of 1.

  3. 3.

TF-IDF: The term frequency inverse document frequency measure is an
information retrieval statistic that reflects the importance of a word to a
document in a collection or corpus. It is also widely used for knowledge graph
ranking by calculating relevance between potential knowledge graphs with an
automatically gathered concept that describes the domain of interest. TF-IDF
can be modeled as:

$TF-IDF(K,Q)=TF(K,Q)*IDF(K,Q)$ (6.3)

where $TF(K,Q)$ denotes the term frequency for query $Q$ within the knowledge
graph $K$, and $IDF(K,Q)$ pertains to the prevalence of a resource throughout
the resources, which is derived by dividing the aggregate count of knowledge
graphs by the number of resources that include the query $Q$, followed by
calculating the logarithm of the resulting quotient.

  4. 4.

BM25: The Okapi BM25 algorithm is a widely used information retrieval
technique. Exploiting the probabilistic information retrieval framework, BM25
modifies and extends the TF-IDF measure. BM25 can effectively differentiate
between the occurrence of a term in a document relative to its frequency in
the corpus, while also preventing overemphasis on longer documents. It excels
in scenarios where relevant documents vary in length, as it can adeptly
balance the frequency of terms with the length of the document. BM25 can be
presented as:

$BM25(K,Q)=\sum_{i=1}^{n}IDF(K,q_{i})\frac{TF(K,q_{i})*m+1}{TF(K,q_{i})+m*(1-b+b*\frac{|d|}{avg})}$
(6.4)

where $|d|$ is the length of the resources $d$ in words, and $avg$ is the
average document length in the text collection from which the resources are
drawn, two key parameters $m$ and $b$ control the saturation of term frequency
and the length normalization, respectively.

The above-mentioned assessment metrics are widely applied to measure the
knowledge graphs from dimensions including completeness, consistency, and data
quality. However, these metrics lack the consideration of the categorization,
which is one of the main purposes of the construction of knowledge graphs.
Thus, we are attempting to improve the current knowledge graph assessment by
proposing a novel set of assessment metrics from the categorization
perspective.

###  6.3 Measuring Categorization Purpose by Focus

####  6.3.1 Intuition

Envision a scenario where an individual specifies “the book on my desk” to
have another person retrieve that specific book based on her description. Such
object descriptions are intended to facilitate the retrieval of the designated
objects. The essence of this process is to precisely describe the cluster of
objects, which underpins the categorization purpose of an object description.
These object descriptions, also termed types or categories, form the
foundation of the organization of our mental life [86].

In line with contemporary psychological insights, particularly the seminal
contributions of Eleanor Rosch [112], the purpose of categorization can be
understood through two principal dimensions: i). maximizing the number of
features that characterize the members of a specific category, and ii).
minimizing the number of features shared with other categories. To assess
these dimensions, Eleanor Rosch introduced the concept of cue validity [113],
defined as “the conditional probability $p(c_{j}|f_{j})$ that an object falls
in a category $c_{j}$ given a feature $f_{j}$”. Cue validity was instrumental
in identifying basic-level categories, which are those that maximize intra-
category feature commonality while minimizing inter-category feature overlap.

Rosch’s formulations were initially applied in experiments where individuals
were asked to classify objects into specified categories. We adapt Rosch’s
original approach to the realm of knowledge graph engineering. In our context,
each available knowledge graph, such as schema.org or DBpedia, is
conceptualized as a categorization, modeled as a collection of entity types
associated with properties. The primary role of these categorizations is to
“draw sharp lines around the types of entities they contain, ensuring each
member definitively falls within or outside each entity type” [56, 57].
Knowledge engineers assume a role similar to the participants in Rosch’s
study. Every knowledge graph schema offers a rich set of categorization
instances. Each entity type functions as a category and all entity type
properties act as features. The categorization purpose of a knowledge graph,
which we term focus, is modeled as “the state or quality of being relevant in
storing and retrieving information”. We quantify the degree of this pertinence
by adapting cue validity in the following manner:

  * •

We take each property to have the same cue validity;

  * •

For each knowledge graph we equally divide the property cue validity across
the entity types the properties are associated with;

  * •

We quantify the relevance of the knowledge graph and entity types in storing
and retrieving information by checking the spreading of cue validity.

Thus, the focus can be then calculated based on the above assumptions and, in
turn, it can be functionally articulated in:

  * •

Entity Types Focus: This aspect pertains to the capability to identify those
entity types that constitute the most informative categories. These are the
categories with heightened relevance for categorization, or more specifically,
those that maximize the number of unique properties and minimize the overlap
of properties with other categories. To a certain extent, these entity types
correspond to what expert users deem as “core entity types” within a specific
domain.

  * •

Knowledge Graphs Focus: This refers to the ability to pinpoint knowledge
graphs that encapsulate the greatest number of highly informative (focused)
entity types. Such knowledge graphs are characterized, to a certain degree, as
“clean” or “not-noisy” and are associated with what expert users regard as
well-structured knowledge graphs [102].

####  6.3.2 Metric Focus of Entity Types

According to the notion of entity type focus we introduced above, we model the
metric focus of entity type $Focus_{e}$ as:

$Focus_{e}(e)=Cue_{e}(e)+\eta
Cue_{er}(e)=Cue_{e}(e)(1+\frac{\eta}{|prop(e)|})$ (6.5)

where $e$ represents an entity type, the $Focus_{e}$ results from the
summation of $Cue_{e}$ and $Cue_{er}$. $Cue_{e}$ represents the cue validity
of the entity type. $Cue_{er}$ represents a normalization of $Cue_{e}$. $\eta$
represents a constraint factor to be applied over $Cue_{er}$. The constraint
factor $\eta$ is used to manipulate the weight of the $Cue_{er}$, where
$\eta>0$, thereby affecting the value of the metric $Focus_{e}$. Specifically,
two parts of the function can also be combined, in which $|prop(e)|$ is the
number of properties associated with the specific entity type $e$.

To model $Cue_{e}$ and $Cue_{er}$ we mainly adapted the work presented in
[55]. In order to calculate $Cue_{e}$, firstly, we define the cue validity of
a property p associated with an entity type $e$, also called $Cue_{p}$, as:

$Cue_{p}(p,e)=\frac{PoE(p,e)}{|dom(p)|}\in[0,1]$ (6.6)

where $|dom(p)|$ presents the cardinality of entity types that are the domain
of the specific property $p$. $PoE(p,e)$ is defined as:

$PoE(p,e)=\left\\{\begin{matrix}1,ife\in dom(p)\\\ 0,ife\notin
dom(p)\end{matrix}\right.$ (6.7)

$Cue_{p}(p,e)$ returns 0 if $p$ is not associated with $e$. Otherwise returns
$1/n$, where $n$ is the number of entity types in the domain of $p$. In
particular, $Cue_{p}$ takes the maximum value 1 if $p$ is associated with only
one entity type.

Given the notion of $Cue_{p}$ we provide the notion of cue validity of an
entity type. $Cue_{e}$ is related to the sum of the cue validity of the
properties associated with the specific entity type $e$ and is modeled as
follows:

$Cue_{e}(e)=\sum_{i=1}^{|prop(e)|}Cue_{p}(p_{i},e)\in[0,|prop(e)|]$ (6.8)

where $Cue_{e}$ provides the centrality of an entity type in a given knowledge
graph, by summing all its properties $Cue_{p}$. $Cue_{e}$ refers to the
maximization of the properties associated with entity type $e$ with the
members of its categories.

Given the notion of $Cue_{e}$, we capture the minimization level of the number
of properties shared with other entity types, inside a knowledge graph with
the notion of $Cue_{er}$, which we define as:

$Cue_{er}(e)=\frac{Cue_{e}(e)}{|prop(e)|}\in[0,1]$ (6.9)

After deriving $Cue_{e}$ and $Cue_{er}$ it is possible to calculate
$Focus_{e}$. Notice that, to normalize the range of the metrics, we applied
log normalization [18] on $Cue_{e}$ since $|prop(e)|$ can be significantly
unbalanced between entity types and min-max normalization [66] on $Cue_{er}$.

####  6.3.3 Metric Focus of Knowledge Graphs

Following the knowledge graphs focus notion we introduced, we model the metric
focus $Focus_{k}$ as follows:

$Focus_{k}(K)=Cue_{k}(K)+\mu Cue_{kr}(K)=Cue_{k}(K)(1+\frac{\mu}{|prop(K)|})$
(6.10)

where we take $K$ as an input knowledge graph and we take $Focus_{k}$ as a
summation of $Cue_{k}$ and $Cue_{kr}$. $Cue_{k}$ represents the cue validity
of the knowledge graph. $Cue_{kr}$ represents a normalization of $Cue_{k}$.
$\mu$ represents a constraint factor for $Cue_{kr}$, which we assume $\mu>0$,
as for $Cue_{er}$ above. $|prop(K)|$ refers to the number of the properties in
$K$.

The notions and terminology used for entity types, i.e., the notions of
$Cue_{e}$ and $Cue_{er}$, can be straightforwardly generalized to knowledge
graphs, generating the following metrics:

$Cue_{k}(K)=\sum_{i=1}^{|E_{K}|}Cue_{e}(e_{i})\in[0,|prop(K)|]$ (6.11)

The $Cue_{k}(K)$ is calculated as a summation of the cue validity of all the
entity types in a given knowledge graph, which in the function is represented
by $E_{K}$. $|prop(K)|$ refers to the number of the properties in the
knowledge graph, as the maximization of $Cue_{k}$.

Following the formalization of $Cue_{er}$ we capture the minimization level of
the number of properties shared across the entity types inside the graph with
the notion of $Cue_{kr}$, which we define as:

$Cue_{kr}(K)=Cue_{k}(K)/\sum_{i=1}^{|E_{K}|}prop(e_{i})\in[0,1]$ (6.12)

where $Cue_{k}$ and $Cue_{kr}$ can be used to assess the focus of a whole
knowledge graph. Notice that to normalize the metric $Focus_{k}$, we applied
log normalization on $Cue_{k}$, since $|prop(K)|$ may be significantly higher
in some knowledge graphs than others knowledge graphs and min-max
normalization on $Cue_{kr}$.

###  6.4 Metrics Validation

####  6.4.1 Analysis of Knowledge Graph Rankings

In this section, we discuss the validity of the proposed focus metrics via a
set of analyses that assess the collected knowledge graphs. We collected a
data set of 700 knowledge graphs, expressed in the Terse RDF Triple Language
(Turtle)111 https://www.w3.org/TR/turtle/ format. Most of these resources have
been taken from the LOV catalog222https://lov.linkeddata.es/dataset/lov. The
remaining ones, see for instance
freebase333https://developers.google.com/freebase and
SUMO444http://www.adampease.org/OP/ have been added to extend the validation
dataset.

Figure 6.1: Knowledge graphs selected for the analysis.

For analytical purposes, we adopt the framework of FCA delineated in section
4.3, wherein all datasets are transformed into a collection of sets of triples
(one set for each entity type), with each triple encoding information about
“entity type-property” associations. Furthermore, to generate the final output
datasets, property labels were processed through an NLP pipeline which
undertook various steps, including: i). splitting a string at the occurrence
of every capital letter; ii). converting all characters to lower case; iii).
removing stop-words. This processing enabled us to conduct a more precise
analysis.

From the initially processed dataset, we selected a subset by excluding all
knowledge graphs containing fewer than 30 entity types. Fig.6.1 offers a
comprehensive overview of the final output dataset, illustrating for each of
the remaining 44 knowledge graphs the number of properties, the number of
entity types, and their balance. The balance metric reflects the distribution
of a knowledge graph’s properties across its entity types, which is calculated
as follows:

$Balance(K)=\frac{|prop(K)|}{|prop(e)|_{max}\ast|etype(K)|}$ (6.13)

where $|prop(K)|$ presents the cardinality of the set of properties in the
knowledge graph, $|prop(e)|_{max}$ being the cardinality of the maximum set of
properties associated with the entity type $e\in K$, and $etype(K)$ refers to
the cardinality of the set of entity types in the knowledge graph.

Figure 6.2: Entity types categorization relevance for example knowledge
graphs.

By applying the Cue metric $Cue_{e}(e)$ and $Cue_{er}(e)$ to the knowledge
graphs of the resulting list, we obtained the scores to evaluate the
categorization relevance of the entity types for each knowledge graph. Details
are provided in Figure 6.2, where we randomly selected eight knowledge graphs
and listed them according to the number of entity types. The selected
knowledge graphs are: Freebase, OpenCyc555
https://pythonhosted.org/ordf/ordf_vocab_opencyc.html, DBpedia, SUMO,
schema.org, md666http://def.seegrid.csiro.au/isotc211/iso19115/2003/metadata,
pext777http://www.ontotext.com/proton/protonext.html and ludo-
gm888http://ns.inria.fr/ludo/v1/docs/gamemodel.html. The accompanying scatter
plots display the correlations between the min-max normalized $Cue_{e}(e)$ and
$Cue_{er}(e)$ for each entity type. Entity types positioned in the top-right
quadrant are those with higher categorization relevance. For instance, in the
SUMO knowledge graph, prominent entity types include GeopoliticalArea and
GeographicalArea, whereas in DBpedia, notable examples are Person and Place.
Thus, these observations validate that the metrics $Cue_{e}(e)$ and
$Cue_{er}(e)$ indeed capture the informative categories from different
knowledge graphs.

Table 6.1: Ranking knowledge graphs by the metric $Focus_{k}(K)$. Knowledge Graphs | $Cue_{k}(K)$ | $Cue_{kr}(K)$ | $Focus_{k}(K)$
---|---|---|---
Freebase | 8981 | 0,21 | 1,15
cal | 46 | 0,98 | 0,92
bibo | 71 | 0,97 | 0,92
OpenCyc-l | 6266 | 0,26 | 0,90
swpo | 87 | 0,88 | 0,83
cwmo | 107 | 0,85 | 0,80
eli | 62 | 0,84 | 0,78
ncal | 103 | 0,80 | 0,75
mo | 124 | 0,79 | 0,74
akt | 106 | 0,79 | 0,74
Table 6.2: Ranking entity types by the metric $Focus_{e}(e)$. Knowledge graph | Entity type | $Cue_{e}(e)$ | $Cue_{er}(e)$ | $Focus_{e}(e)$
---|---|---|---|---
DBpedia | person | 169,02 | 0,69 | 1,42
OpenCyc-l | firstordercoll | 230,59 | 0,30 | 1,30
Freebase | statisticalreg | 161,53 | 0,48 | 1,17
Opencyc-l | class | 194,95 | 0,31 | 1,15
dicom | ieimage | 158,90 | 0,44 | 1,13
DBpedia | place | 116,97 | 0,63 | 1,13

Meanwhile, we conducted several statistics to observe the performance of focus
metrics. Firstly, we generated a ranking of these graphs by implementing the
metric $Focus_{k}(K)$ across the 44 chosen knowledge graphs. The top 11
knowledge graphs in this ranking are presented in Table 6.1. Additionally, by
applying $Focus_{e}(e)$ to these knowledge graphs, we derived a ranking of
entity types, with the top 6 entity types, in terms of categorization
relevance, displayed in Table 6.2. Furthermore, by selecting a specific entity
type and applying $Focus_{e}(e)$, it is feasible to identify the most suitable
knowledge graph for that particular entity type. Table 6.3 offers an
illustration for the entity type Person. The statistics demonstrate the
feasibility of ranking knowledge graphs and entity types by focus metrics.
Compared to the ranking results by cue metrics, focus metrics consider
balanced features to capture categorization relevance.

Table 6.3: Ranking the entity type person from different knowledge graphs Knowledge graph | Entity type | $Cue_{e}(e)$ | $Cue_{er}(e)$ | $Focus_{e}(e)$
---|---|---|---|---
DBpedia | person | 169,02 | 0,69 | 1,42
akt | person | 8,00 | 1,00 | 1,03
OpenCyc-l | person | 122,14 | 0,43 | 0,95
vivo | person | 10,60 | 0,88 | 0,92
swpo | person | 3,50 | 0,88 | 0,88
cwmo | person | 5,83 | 0,83 | 0,85

####  6.4.2 Validation of $Focus_{e}$

In this section, we validate the efficacy of the $Focus_{e}(e)$ metric in
measuring the categorization relevance of entity types, specifically their
ability of information maximization. This analysis involves applying our
metrics and corresponding state-of-the-art ranking algorithms to a set of
knowledge graphs. We compare the results with a reference dataset formulated
by 5 knowledge engineers999All knowledge engineers are certified Ph.D.
candidates and joined the experiments voluntarily.. These engineers were given
a set of instructions for ranking the entity types, drawing inspiration from
Rosch’s experiment [112].

To evaluate our metric, we first select a subset of the knowledge graphs
discussed in the previous section, including akt101010
https://lov.linkeddata.es/dataset/lov/vocabs/akt, cwmo111111https://gabriel-
alex.github.io/cwmo/, ncal, pext, schema.org,
spt121212https://github.com/dbpedia/ontology-
tracker/tree/master/ontologies/spitfire-project.eu and SUMO. These particular
knowledge graphs were chosen due to their diverse representation in terms of
the number of properties and entity types, with almost all their entity type
labels being comprehensible to humans131313Many knowledge graphs encode their
entity type labels with an ID.. As a second step, we apply four introduced
ranking metrics: TF-IDF [117], BM25 [111], Class Match Measure (CMM) and
Density Measure (DEM) [1]. The performance of these algorithms served as a
baseline, selecting the top 10 entity types for each given knowledge graph,
and then compared with the rankings generated by $Focus_{e}(e)$. The validity
of our approach was quantified in terms of accuracy (ranging from 0 to 1),
determined by how many entity types in the metrics ranking results aligned
those in the ranking lists provided by the knowledge engineers. The outcomes
of this experiment are depicted in Figure 6.3, the blue line represents the
accuracy of the ranking trend provided by $Focus_{e}(e)$. Each bar represents
the accuracy of the ranking for the corresponding metric.

Figure 6.3: The experimental results on evaluating $Focus_{e}(e)$.

The primary observation is that all referenced metrics exhibit a remarkably
similar trend, demonstrating higher accuracy for knowledge graphs such as akt,
cwmo, ncal, and spt while showing lower accuracy for schema.org and SUMO.
However, this pattern is not mirrored by $Focus_{e}(e)$. Our metric, while not
uniformly superior across all knowledge graphs, is notably effective with
large and very noisy knowledge graphs (characterized by lower entity types
$Cue_{er}$), as exemplified by schema.org and SUMO. This outcome aligns with
our expectations and can be attributed to the significant emphasis we placed
on minimizing the number of overlapping properties. The $Cue_{er}$ for each
entity type indeed provides crucial insights into categorization relevance
that may not be adequately captured if undue importance is given to the sheer
number of an entity type’s properties. Consequently, for smaller and less
noisy (or “clean” in terms of overlapping properties) knowledge graphs, other
approaches that focus heavily on the quantity of entity type properties
perform admirably (as evidenced by the strong performance of the TF-IDF
algorithm). In contrast, knowledge graphs with a vast array of entity types
and low $Focus_{e}(e)$ more effectively discern categorization relevance.

The second key observation is that TF-IDF and $Focus_{e}(e)$ emerge as the top
metrics in terms of average performance, with both achieving a mean accuracy
of 0.52, compared to 0.47 for BM25 and CMM, and 0.44 for DEM. This result is
underpinned by the fact that TF-IDF consistently ranks as the best metric for
small and not-noisy knowledge graphs, while $Focus_{e}(e)$ balances standard
performance in small and clean knowledge graphs with high performance in large
and noisy knowledge graphs.

####  6.4.3 Validation of $Focus_{k}$

In this section, we analyze how focus can assist scientists in repurposing
knowledge graphs for new application domains, such as statistical relational
learning. Considering the absence of reference gold standards for assessing a
knowledge graph’s overall score, the primary objective of this assessment is
to demonstrate the ability of the metric $Focus_{k}(K)$ on the predictive
performance in a relational classification task. In this experiment, we
utilized the same knowledge graphs selected in the previous experiment for
$Focus_{e}(e)$, addressing relational classification, where the objective is
to predict the labels of entity types. Note that we introduce a specific type
of relational classification, as defined in [57]. The experimental setup was
as follows: i). training machine learning models by knowledge graphs selected
based on their $Focus_{k}(K)$ values, where we choose the decision tree [71]
and K-NN [33] algorithms; ii). reporting the relative performance of the
models in terms of accuracy differences and comparing these performances with
the $Focus_{k}(K)$ for each knowledge graph.

Figure 6.4: The experimental results on evaluating $Focus_{k}(K)$.

As depicted in Figure 6.4, the accuracy is represented as the proportion of
correct predictions, within the range of [0%, 100%]. The $Focus_{k}(K)$ is
indicated by the values of the line. The knowledge graph cwmo scores highest
in terms of accuracy (for both trained algorithms) and $Focus_{k}(K)$, while
schema.org ranks as the lowest. The key observation is that, as anticipated,
the accuracy trend for both models generally aligns with the $Focus_{k}(K)$
ranking for most knowledge graphs. However, an exception is noted with K-NN
for the knowledge graph pext, performing better than akt despite a lower
$Focus_{k}(K)$. Further analysis reveals that this phenomenon may be explained
by the interplay between the number of properties and entity types,
particularly the balance of the knowledge graph, which can influence the
model’s predictive performance. The more balanced the knowledge graph, the
higher the likelihood of having entity types with low focus. This effect is
particularly noticeable when comparing two knowledge graphs with very similar
$Focus_{k}(K)$ but different balances.

This experiment, while illustrating $Focus_{k}(K)$ as a measurement of the
categorization relevance of a knowledge graph, suggests its practical
application in assessing the potential performance of a knowledge graph, or a
set of knowledge graphs, in a relational prediction task. These findings could
be utilized, for example, to fine-tune knowledge graphs in an open-world data
extension scenario.

###  6.5 Summary

In this chapter, we focus on discussing various assessment methods for the
knowledge graphs. Firstly, we introduce several state-of-the-art metrics for
evaluating knowledge graphs, considering various dimensions. Then, we propose
a formal method to evaluate knowledge graphs according to their focus, namely,
what cognitive psychologists call categorization purpose. This in turn has
allowed us to describe how this assessment plays an important role in
supporting an accurate level of knowledge graphs understanding and reuse. In
this regard, as preliminary validation of the proposed metrics we are showing:
a). how focus knowledge graphs ranking reflects the ranking of the knowledge
graphs provided by a group of knowledge engineers, following the guidelines
inspired by a well-known experiment in cognitive psychology; b). how focus can
help to select better knowledge graphs to support down-stream tasks.

##  Chapter 7 Knowledge Graph Extension: The Platform

###  7.1 Introduction

In this chapter, we introduce an online system designed to collect, process,
and analyze existing knowledge graphs, aiming to achieve the goal of knowledge
graph extension. This study is based on our publications [122, 28, 49, 138].

The endeavor of identifying, processing, and transforming resourcing knowledge
graphs can be a time-consuming and labor-intensive process. The researchers
and knowledge engineers undertake an extensive exploration across available
resources and their associated catalogs, often necessitating the examination
of various data versions and formats. Furthermore, upon identifying potential
candidate resources, it becomes imperative to conduct a thorough analysis of
the data to ascertain their reliability in relation to the specific task at
hand. This phase not only entails a detailed examination but also pre-supposes
that the processing setup is established and executed directly by the
researcher. If the necessary data are successfully located, a transformation
procedure can be initiated, purposing the relational data to facilitate the
evolution of the resourcing knowledge graphs.

The conception of the LiveSchema platform arose from the pivotal challenge of
consolidating all aforementioned operations into a singular, comprehensive
system, supported by an array of ready-to-use built-in features. Primarily,
the system is designed to assist scientists in more efficiently locating the
relational data they require. Utilizing updates from some of the most advanced
catalogs, LiveSchema aspires to provide an aggregation service that enables
the monitoring of the evolving knowledge representation development community
from a unified location. Furthermore, by integrating various essential
libraries, LiveSchema is positioned to streamline the processes of data
preparation and analysis. Generally, these libraries necessitate an ad-hoc
setup and may require the integration of multiple components and environments,
demanding some extent of coding and development expertise that may not be
inherent to all data scientists. Consequently, LiveSchema is envisioned as a
platform that integrates data analysis, data processing, and machine learning
model deployment. This integration is intended to render these processes more
accessible, reusable, and efficient, thereby reducing the time and complexity
typically associated with knowledge representation tasks.

As a result, we present the LiveSchema initiative, a pioneering gateway
designed primarily to harness the wealth of data encapsulated within knowledge
graphs. Presently, LiveSchema integrates $\sim 1000$ datasets from four
diverse data sources, offering a range of essential facilities. All these
facilities are grouped into two main layers: the data layer and the service
layer. Thus, we first introduce the data layer architecture and the process of
data collection for the LiveSchema platform. Then, we describe various
services developed for utilization, particularly targeting the extension of
knowledge graphs. These capabilities enable users to: i) query and process the
aggregated knowledge graphs; ii) encode each knowledge graph for analysis and
visualization purposes; iii) transform knowledge graphs to aid downstream
tasks; iv) continuously evolve and expand a catalog of knowledge graphs by
aggregating from other source catalogs and repositories. Additionally, we
detail the algorithm designed for extending knowledge graphs. Case studies are
then presented to prove the efficacy of the LiveSchema platform and the
performance of the extensions. LiveSchema is accessible at
http://liveschema.eu/ and ready for demonstration.

###  7.2 Data Layer

The data layer enables a community of users to maintain whole applications and
to suggest and upload new resources or edit some already existing resources.
It is mainly managed by a group of expert knowledge engineers, software
engineers, and data scientists who contribute to the development and evolution
of the whole catalog. Moreover, a group of guest users can also be involved in
the collaborative development of the storage, by uploading and editing some
new datasets and creating new input reference catalogs following well-founded
guidelines. In this section, we are introducing how we design the architecture
of the data layer, and the details of the data collecting and processing
steps.

####  7.2.1 Data Architecture

The current version of LiveSchema is built upon the
CKAN111https://docs.ckan.org/en/2.9/user-guide.html#what-is-ckan open-source
data management system, renowned for being one of the most dependable tools
for open data management. In defining the data architecture of LiveSchema, we
focus on a critical feature made within CKAN, namely the distinction between
datasets and resources222https://docs.ckan.org/en/538-package-install-
docs/publishing-datasets.html. A dataset is conceptualized as a collection of
data (for example, the BBC Sport Ontology), which can encompass multiple
resources. These resources represent the tangible forms of the dataset in
various downloadable formats (such as the BBC Sport Ontology in Turtle or FCA
formats). This distinction allows us, as a major advance from mainstream
catalogs such as Linked Open Vocabularies [134], to exploit fine-grained
metadata attributes from the Application Profile for European Data Portals
(DCAT-AP)333https://ec.europa.eu/isa2/solutions/dcat-application-profile-data-
portals-europe_en/, which mirrors a conceptually identical distinction between
dataset and distribution. The additional benefit of employing DCAT-AP lies in
that it organizes attributes into categories: mandatory, recommended, and
optional. This classification is deemed pivotal in facilitating varying levels
of semantic interoperability amongst data catalogs. By adopting this approach,
LiveSchema not only aligns with the sophisticated metadata management of DCAT-
AP but also enhances the potential for semantic connectivity and data
integration across diverse data sources.

Here we demonstrate the metadata specification, i.e. the selected metadata
attributes for datasets and distributions considered in the current version:

  * i.

Dataset:

    * –

Mandatory: description, title;

    * –

Recommended: dataset distribution, keyword, publisher, category;

    * –

Optional: other identifier, version notes, landing page, access rights,
creator, has version, is version of, identifier, release date, update,
language, provenance, documentation, was generated by, version.

  * ii.

Distribution:

    * –

Mandatory: access url;

    * –

Recommended: description, format, license;

    * –

Optional: status, access service, byte size, download url, release date,
language, update, title, documentation.

It is imperative to recognize that the metadata distinction between dataset
and distribution is non-trivial, particularly considering that metadata
attributes such as format, license, byte size, and download URL are pertinent
to the distribution, rather than the dataset itself. This demarcation
highlights the specific allocation of metadata, ensuring that each attribute
is correctly ascribed to its relevant component within the data management
structure.

Our initial observation pertains to the two primary benefits that the
aforementioned distinction and metadata specification confer upon LiveSchema.
First and foremost, the application of metadata facilitates the
“Fairification”444The FAIR catalog is accessible at
https://github.com/OntoUML/ontouml-models. [141] of the knowledge schemas.
This process effectively makes the schemas findable, accessible,
interoperable, and reusable, aligning them with the objectives of downstream
tasks targeted by LiveSchema. Secondly, the incorporation of enhanced metadata
into the knowledge schemas plays a crucial role in initiating, augmenting, and
ensuring reproducibility. This observation addresses the potential for future
expansion of the metadata specification. A key advantage of incorporating
knowledge graph-specific metadata in LiveSchema lies in the ability it grants
users to conduct highly customized searches, which enhances the user
experience and ensures the retrieval of schemas that are optimally aligned
with specific task requirements.

####  7.2.2 Data Collection

At the current state, LiveSchema exploits four principal state-of-the-art
catalogs: LOV, DERI,555http://vocab.deri.ie/ FINTO666http://finto.fi/en/, and
a custom catalog, names KnowDive777http://liveschema.eu/organization/knowdive.
The dataset statistics from each of these catalogs are depicted in Figure
7.1(a)888Note that the statistics are illustrative since the catalog is
currently evolving and may change in the future.. As it stands, LOV
contributes the majority of datasets, establishing itself as one of the most
extensively utilized vocabulary catalogs within the semantic web sphere. Each
dataset within LiveSchema is linked to a specific provider, which refers to
the individual or organization responsible for uploading and maintaining the
dataset in its originating catalog. These datasets have been individually
scraped and uploaded from each catalog in an automated manner. Figure 7.1(b)
showcases the information regarding these providers, where we find that W3C is
the most productive organization in terms of dataset provision.

Figure 7.1: Overall statistics of LiveSchema datasets.

Figure 7.1(c) showcases the statistics of several critical pieces of
information. From the selected catalogs, 26 types of metadata attributes have
been scraped, which include: i). catalog attributes, such as id, title and
description; ii). dataset attributes, like language, publisher, license,
version, tags and source; iii). provider attributes, such as id, name, title
and uri. During data scraping, we ensured that no license agreements were
breached. Presently, five types of licenses are permitted given their
restrictions (all being part of the Creative
Commons999https://creativecommons.org/ initiative).

When parsing these datasets in various
formats101010https://rdflib.readthedocs.io/en/stable/plugin_parsers.html, we
serialize them into the most common formats, i.e., RDF and Turtle. More
sophisticated output formats can be generated through the processing
operations facilitated by LiveSchema services, such as CSV (where all triples
and metadata of the input relational data are stored in a datasheet format),
MEX (providing all the assessment metrics), FCA (representing the FCA
transformation matrix result), VIS (the format enabling visualization services
functionalities), and EMB (used for generating a statistical model based on a
knowledge embedding process).

####  7.2.3 Data Pre-processing

LiveSchema is managed by a group of knowledge experts who contribute to the
development and evolution of the whole system. This group, referred to as
LiveSchema administrators or simply admins, not only addresses maintenance
issues but also manages the application of functionalities related to the
system’s evolution, which are crucial in populating the catalog. Two evolution
operations can be executed by the LiveSchema admin. Firstly, LiveSchema
encompasses an automated evolution process, consisting of parsing and scraping
phases. Admins are provided with a few checkpoints to monitor the outcomes of
these automated procedures. Secondly, manual operations such as dataset
uploading, reviewing, and managing are also facilitated through LiveSchema
services. Some of these datasets are not directly available online and require
downloading, unzipping, or editing before being uploaded to GitHub for
collection via a URL link. Following data scraping, a secondary key semi-
automated parsing process is initiated.

This parsing process, executed iteratively aims to output a set of serialized
datasets that are generated by scanning the dataset list and parsing it with
the RDFlib Python library111111https://github.com/RDFLib/rdflib, a tool used
for processing RDF resources. The generated outputs are then used to create
more standardized reference formats, currently represented by RDF and Turtle.
The generation of a (.xlsx or .csv) file encoding all dataset information
(e.g., triples) is also supported, thereby facilitating other applications
provided by the catalog. In this phase, the admin’s key responsibility is to
edit the dataset list to exclude unrequired datasets. Then, the dataset will
be filtered and transformed into a set of triples as the output of the parsing
process.

###  7.3 Service Layer

The LiveSchema service layer offers the connection between users and collected
datasets, where all datasets acquired from source catalogs and uploaded to
LiveSchema can subsequently be transformed and utilized as input for the
platform’s available functionalities. The current version of LiveSchema offers
several functionalities, including:

  * •

Data querying: the basic function of a knowledge graph repository, enabling
the execution of full-text search, fuzzy-matching, and SPARQL
queries121212https://rdflib.readthedocs.io/en/stable/intro_to_sparql.html,
where all datasets and attributes are searchable for providing customized
service to the users.

  * •

FCA generation: the process of transforming the input knowledge graph format
into the FCA format which is designed in section 4.3;

  * •

Relational data analyzing: a set of functions that facilitate both visual and
parametric analysis of the target knowledge graph, providing knowledge
engineers with comprehensive measures for various analysis scenarios.

  * •

Metric generation: the process that produces the metrics we introduced in the
previous chapters, including similarity measurements for entity type
recognition, and Cue validity and Focus metrics for assessing knowledge
graphs.

  * •

Knowledge graph embedding: the function for encoding the target knowledge
graph into high-dimensional deep features using graph neural networks [137],
aiming to support deep analysis and machine learning pipelines in downstream
tasks.

  * •

Knowledge graph visualization: an application for viewing concepts and their
associated properties within the target knowledge graphs, via the
implementation of the WebVOWL
library131313http://vowl.visualdataweb.org/webvowl.html.

These functionalities are readily accessible and reusable through API services
and can be easily extended. Figure 7.2 presents an example of dataset
management, where all functionalities for managing, analyzing, and
transforming are displayed141414Current reference resources rdf, ttl, and csv
are ready for download.. Metadata, tags, and information about the reference
source catalog are also provided to users on the top left of the link. All the
new formats can be accessed on the corresponding functionality page. As some
functions, such as FCA generation, have been discussed in earlier chapters,
the remainder of this section will focus on detailing the data analysis and
knowledge embedding functions, as well as exploring several additional
functions anticipated to be available in the forthcoming update.

Figure 7.2: LiveSchema dataset page from an admin perspective.

####  7.3.1 Relational Data Analyzing

Each LiveSchema dataset is allocated a specific page that presents its
information, and from which various processing functionalities can be
accessed, as depicted in Figure 7.2. A user of LiveSchema has the capability
to conduct a straightforward search among the datasets available within the
system, subsequently executing analyses to identify the necessary datasets.
These analysis functions are accessible via the corresponding tab and
facilitate the creation of a corresponding matrix for each knowledge graph.
For instance, users have the option to tailor the matrix generation by
filtering the target predicates.

In an example scenario, let us consider the task of selecting a reference
knowledge graph for recognizing entities of the type ‘Person’ within a
collection of tabular data, e.g., originating from an open data repository.
This task may entail identifying a knowledge graph that possesses i). the
target entity type and its corresponding label; ii). a substantial number of
properties for the target entity type, enhancing the likelihood of aligning
properties in the input test data; iii). a relatively low count of overlapping
properties, aimed at reducing the incidence of false negatives/positives.
Suppose the user identifies three potential resources, namely
Schema.org151515https://schema.org/ (a reference standard for web document
indexing), FOAF161616http://xmlns.com/foaf/spec/ (a vocabulary commonly used
in social network contexts), and the BBC sport
ontology171717https://www.bbc.co.uk/ontologies/sport (used by the BBC for
modeling sports events and roles). By downloading and comparing all the cue
validity and focus values for the entity type Person representations across
these three sources, as illustrated in Table 7.1, a preliminary benchmarking
can be conducted. From this analysis, Schema.org emerges as the clear choice,
given the high overall evaluation score for the specified entity type.

Entity type | Knowledge graph | $Cue_{e}$ | $Cue_{er}$ | $Focus_{e}$
---|---|---|---|---
Person | Schema.org | 23 | 0.81 | 0.73
Person | FOAF | 3 | 0.82 | 0.54
Person | BBC | 0.73 | 0.75 | 0.23
Table 7.1: Analysis of the entity type person based on its generated metrics.
Figure 7.3: A generated knowledge lotus of the entity type person.

In addition to quantifying the generated metrics, further analysis can be
conducted by visualizing the intersection of several top entity types from the
provided resources. Figure 7.3 depicts an instance of a knowledge lotus
created by LiveSchema, illustrating the diversity of entity types in terms of
their shared properties. The yellow petals in the lotus represent the number
of properties that are distinctive for the given entity types. In the
illustrated example, the entity type ‘Person’ has 35 unique properties, and
the red-highlighted petal signifies a property that is shared across all
entity types. This kind of visual analysis aids in comprehending the overlap
and uniqueness of properties among different entity types within the knowledge
graph.

Figure 7.4: Intersection between entity types and properties by UpSet
visualization

Further analysis can be conducted through the application of UpSet (multiple
set) analysis facilities. LiveSchema facilitates both knowledge lotuses and
UpSet visualizations by incorporating the capabilities of the intervention
visualization environment181818https://intervene.shinyapps.io/intervene/. The
primary aim of these visualization tools is to simplify the analysis and
interpretation of the input resource. An exemplary depiction of a resource
using the UpSet module is presented in Figure 7.4. In this example, eight
types are selected. The blue bars on the left side indicate the size of these
based on the number of properties. The black dots represent the intersections,
while the red bars at the top of the figure denote the size of the
intersecting properties set. This approach offers a clear and intuitive
understanding of the relational dynamics and commonalities among different
resources.

####  7.3.2 Knowledge Graph Embedding

The knowledge embedding function in LiveSchema encodes the targeted knowledge
graph into high-dimensional deep features using graph neural networks. This
function is designed to facilitate in-depth analysis, such as graph-based
entity similarities, or to support machine learning pipelines in downstream
tasks. It is important to note that the current release of LiveSchema
incorporates distributional embedding techniques. LiveSchema utilizes the
PyKEEN library [5], a recent collection encompassing state-of-the-art
techniques for graph embedding. PyKEEN is a popular choice for creating custom
embedding models. It offers a broad selection of training approaches with
various parameters, resulting in a .pkl file that can be directly integrated
into machine learning pipelines.

Figure 7.5: The knowledge embedding interface in LiveSchema platform.

Figure 7.5 displays a screenshot of the knowledge graph embedding page, where
various parameters can be adjusted to achieve specific learning objectives.
Users can select the “embedding model” from a range of state-of-the-art
algorithms, such as TransE, RESCAL, or DistMult [137]. Additionally, settings
like the “loss function” are available. The loss function is typically
employed to minimize model error and can condense multiple model features into
a single scalar value. This scalar value enables the ranking and comparison of
candidate solutions [118], thereby enhancing the efficacy and applicability of
the embedding process.

####  7.3.3 Path Recommendation

The knowledge graph path recommendation task is an advanced analytical
operation in the realm of semantic computing and artificial intelligence. This
task involves identifying and predicting optimal paths within a knowledge
graph, where a path refers to a sequence of relations connecting a set of
entities (types) [120]. The path recommendation task aims to uncover
significant paths based on specific criteria or objectives. These objectives
could include finding the shortest path, the path toward the most influential
concept based on specific properties, or paths that reveal hidden patterns or
relationships not immediately apparent. Applications of path recommendation
encompass areas such as recommendation systems, semantic search, and data
mining. For instance, semantic search can enhance query understanding and
result relevance by tracing conceptual links between query terms and
information stored in the graph [139]. Similarly, recommendation systems can
suggest products, services, or content by navigating the relations between
user preferences and item attributes.

Recognizing the significance of the knowledge graph path recommendation task,
our research in this domain addresses both methodological and practical
aspects. From an application perspective, we have explored the application of
path recommendation in online learning environments [116]. Effectively
sequencing learning objects to create a well-defined learning path can
significantly assist e-learners in achieving their learning objectives
efficiently and systematically, particularly when dealing with a vast array of
fragmented learning content. In response to this need, we developed a
knowledge graph-based learning path recommendation system [122]. This system
utilizes both entity-level and schema-level knowledge to more effectively
organize and plan the learning path. Concurrently, we identified six main
semantic relations between learning objects within the knowledge graph.
Furthermore, we designed a learning path recommendation model that caters to
diverse learning requirements and is capable of generating and recommending
customized learning paths tailored to the e-learner’s specific learning
objectives. Our experimental results suggest that the proposed model can
successfully generate and recommend personalized learning paths, thereby
enhancing the learning experience for e-learners. The implementation of this
system significantly improves the depth and accuracy of learning content
available to learners, thereby contributing to more efficient and customized
educational experiences.

Meanwhile, current knowledge graph path recommendation methods predominantly
concentrate on the different selection of deep learning-based knowledge
representation models191919Embedding networks like TransE and RESCAL, which
encode $<$entity-property-entity$>$ triples into high-dimensional spaces..
However, these methods generally do not account for the diverse semantic
relations between entities as expressed in multi-step relations, which can
limit the effectiveness of path recommendation. To address this issue, we
introduced a novel framework, namely MRP2Rec [138], designed to execute path
recommendations using latent features derived from both single-step and
multiple-step relation paths. More specifically, our approach begins by
transforming independent triples into a unified global space to generate
multi-step relations. Subsequently, all relations are encoded through a path
semantic embedding network, where the encoding facilitates the generation of
prediction lists via the inner product in top-K recommendations. Note that the
embedding network can be selected from the LiveSchema knowledge embedding
functionality. The experimental results demonstrate that the MRP2Rec framework
outperforms state-of-the-art methods. This enhanced performance is attributed
to the framework’s ability to learn more semantic relations from paths
involving multiple-step relations. Consequently, our framework can recommend
items with implicit features indicative of user collaborative behavior,
providing a more nuanced and effective recommendation process.

###  7.4 Knowledge Graph Extension

In this section, we exhibit the primary application of the proposed platform,
specifically focusing on the task of knowledge graph extension, which also
constitutes the main objective of this dissertation. Firstly, we will
elaborate on the details of the knowledge graph extension algorithm.
Subsequently, we will present several case studies to validate the
effectiveness of our method.

####  7.4.1 The Algorithm

In the task of knowledge graph extension, we proceed with the presumption that
the input data includes a reference knowledge graph and one or more candidate
knowledge graphs. The reference knowledge graph is introduced to selectively
integrate content from these candidate knowledge graphs for its expansion.
These input knowledge graphs can be identified using the data analysis
functionalities available on the LiveSchema platform. Additionally, local
knowledge graphs can also be utilized as input sources.

Algorithm 2 Extending reference knowledge graph by integrating a candidate
knowledge graph. $KG_{ext}=KGExtension(KG_{ref},KG_{cand},EM_{ali})$

0: Reference and candidate knowledge graphs $KG_{ref}$, $KG_{cand}$;Aligned
entity type pairs $EM_{ali}$;

0: The extended reference knowledge graph $KG_{ext}$;

1: for all $(E_{a},E_{b})\in EM_{ali}$ do

2: $E_{a}.addProperty(E_{b}.property)$; {Merging the properties of entity type
$E_{b}$ into $E_{a}$, where $E_{a}\in KG_{ref},E_{b}\in KG_{cand}$. }

3: $E_{a}.addInstances(E_{b}.instances)$; {Merging the entities of $E_{b}$
into $E_{a}$. }

4: for all $E_{sub}\in E_{b}.subClass$ do

5: $E_{a}.addSubClass(E_{sub})$; {Merging the entity types $E_{sub}$ and their
entities into $E_{a}$.}

6: end for

7: $KG_{ext}.update(E_{a})$;

8: end for

9: $E_{ref}=listEtypes(KG_{ref})$;

10: $I_{cand}=listInstances(KG_{cand})$;

11: for all $I_{i}\in I_{cand}$ do

12: if $findClass(I_{i})\notin EM_{ali}$ then

13: $E_{n}=EtypeRecognizer(I_{i},E_{ref})$; {Recognizing the entity type
$E_{n}$ of candidate entities $I_{i}$, where $E_{n},E_{ref}\in KG_{ref}$.}

14: $E_{n}.addInstances(I_{i})$;

15: end if

16: $KG_{ext}.update(E_{n})$

17: end for

18: return $KG_{ext}$

Consequently, let us assume that we are already given the reference and
candidate knowledge graphs ($KG_{ref}$ and $KG_{cand}$ respectively), which
serve as inputs for the extension algorithm. Let us detail the extension
algorithm in Algorithm 2. Assume we also have obtained the entity type
alignment results by the entity type recognition method presented in Chapter
5. Thus, for each entity type $E_{b}$ that is aligned with entity type $E_{a}$
from $KG_{cand}$, we directly add its properties and entities into $E_{a}$ by
$addProperty(\cdot)$ and $addInstances(\cdot)$, respectively. We also consider
all subclasses of the aligned entity type $E_{b}$ since the subclass inherits
the properties of the entity type and will bring new entities. If the subclass
$E_{sub}$ of $E_{b}$ is not aligned with any other entity types in $KG_{ref}$,
$E_{sub}$ and its associated properties and entities will be merged into
$E_{a}$. Thus, we integrate candidate entities with $KG_{ref}$ when their
entity types are able to align with $KG_{ref}$.

Then, the proposed property-based similarity metrics are applied to align the
rest of the candidate entities with entity type $E_{ref}$ from the reference
knowledge graph, namely function $EtypeRecognizer(\cdot)$, which is also
defined in section5.2 as the instance-level entity type recognition. If we
successfully align an entity type $E_{n}$ with the entity $I_{i}$, $I_{i}$
will be merged into $E_{n}$, if not, $I_{i}$ will be discarded. Regarding the
real-world application scenario and topic, the knowledge engineer in our
platform will decide if it is needed to integrate the not-aligned entity types
with their corresponding entities into $KG_{ref}$. Note that such a strategy
for handling the conflict between the $KG_{ref}$ and $KG_{cand}$ is a specific
case to the KG extension task since the extension task will consider the usage
of the knowledge in the reference KG primarily. However, the conflict of
entity types, entities, and properties widely appears between the $KG_{ref}$
and $KG_{cand}$, while the conflict handling strategy can be changed when
applying this method to different scenarios and tasks. Finally, all changes
will be updated to $KG_{ref}$ and we will obtain an extended knowledge graph
$KG_{ext}$ as the final result of our method.

After we run the extension algorithm, an evaluation process will be conducted
to validate the quality of the extended reference knowledge graph, where the
metrics introduced in sections 6.2 and 6.3 will be exploited. We compare the
metrics on the original reference knowledge graph with the extended version to
see if the new-coming concepts from candidate knowledge graphs affect the
quality of the reference knowledge graph.

####  7.4.2 Case Study

We aim to qualitatively assess the extension results by several use cases. In
the first case, we demonstrate the process of extending a knowledge graph
within a specific domain, focusing on the extension of three datasets that
contain Chinese characters from different historical periods, as detailed in
the work presented in [28]. Due to the evolutionary nature of characters,
there are both semantic continuities and variations among characters from
different periods, with some characters maintaining their represented concepts
over time while others change or become obsolete. Therefore, it is crucial to
properly integrate the consistent characters and to import the new-coming
characters as required. The three datasets encompass oracle bone inscriptions,
bronze inscriptions, and warring states inscriptions, the statistics of the
entity types and entities are illustrated in Table 7.2. Each entity type is
linked with properties such as sense, glyph, composition structure,
deciphering status, and related characters.

Utilizing the information from these properties, we applied our proposed
entity type recognition approach, which facilitated the alignment of 1556
entity types and the identification of the types of 3251 entities. Then, by
employing the extension algorithm, we used the warring states inscriptions
knowledge graph as the reference to integrate the other candidate knowledge
graphs, according to the entity type recognition results. The final extended
knowledge graph comprises 6942 entity types and 42151 entities. The
correctness of new-coming entities was validated through data verification
conducted by three linguists202020The group of linguists consists of PhD
candidates and a professor in archeology., ultimately confirming the validity
of the extended knowledge graph.

Table 7.2: The statistics of character knowledge graphs.  Knowledge graph | Oracle KG | Bronze KG | Warring States KG | Extended KG
---|---|---|---|---
Entity Types | 2543 | 2319 | 5632 | 6942
Entities | 15175 | 14289 | 28421 | 42151

In the second case study, we endeavor to simulate a real-world situation where
individuals extend a general-purpose knowledge graph with domain-specific
knowledge graphs to improve usability. We select the widely utilized
schema.org as the reference knowledge graph. As candidates, we introduce two
domain-specific knowledge graphs
Transportation212121https://carlocorradini.github.io/Trentino_Transportation
and Educationtrentino222222https://alihamzaunitn.github.io/kdi-
educationtrentino that are specifically designed to support local
transportation and education systems, respectively. We chose these knowledge
graphs because they offer diverse examples in terms of the number of
properties and entity types. Furthermore, the clarity and human-readability of
most of their labels facilitate qualitative analysis. We follow the same
procedure as in the first case study to recognize the entity types and then
proceed to extend the reference knowledge graph. After the extension, the
reference knowledge graph encompasses 826 entity types and 1525 properties,
reflecting an increase of 23 entity types and 73 properties, respectively.
This expansion demonstrates the efficacy of incorporating domain-specific
knowledge into a broader knowledge graph, enriching its detail and
applicability.

To further evaluate the effectiveness of our knowledge graph extension
results, we conducted an additional experiment designed to assess the quality
of the extended entity types. This experiment is predicated on the idea that
our focus is predominantly on the extension of aligned entity types when
extracting information from domain-specific resources. For instance, we focus
more on aligned entity types such as University and Course from
Educationtrentino, Trip and Location from Transportation, or characters that
consistently represent the same concepts across different periods. The
objective of the experiment is to quantitatively evaluate these aligned entity
types using several assessment metrics. Specifically, we compare the metric
values of the original and extended entity types to determine whether these
entity types have been positively influenced by the extension process. We
employ three metrics in this study: CMM, DEM, and Focus, as previously
described in section 6.2. These metrics are designed to offer a comprehensive
evaluation, encompassing aspects such as coverage, level of knowledge details,
and categorization relevance. This experiment encompasses datasets from both
case studies, and the results are depicted in Figure 7.6. Each value
represents the average measure of all aligned entity types from the respective
case study. It is important to note that the results of all metrics have been
normalized to a range of [0,1] to facilitate easy comparison.

Figure 7.6: Comparison of the entity types before and after the knowledge
graph extension via assessment metrics.

The primary outcome of the results indicates that the extension process
demonstrates promising performance across all metrics in both cases. In the
first case, the objective was to extend a domain-specific knowledge graph,
leading to the inclusion of a substantial number of new concepts within the
extended entities. The experimental data show significant increases across all
three metrics, suggesting considerable enhancement in the informational
content. In the second case, a similar increasing trend in metrics was
observed, with the extended entity types surpassing their original version in
performance. However, the growth across the three metrics was less marked. We
attribute this to the fact that the reference schema.org is a general-purpose
knowledge graph encompassing a diverse array of concepts, where the limited
extension information might not have a significant impact on the metrics.

The second notable observation is that the extended entity types generally
demonstrate superior performance across all metrics compared to their original
versions. The rise in the CMM score indicates that the integration of new
concepts has broadened the coverage of entity types. The improvement in the
DEM score suggests enhanced detail in the entity types of the reference
knowledge graph, achieved through the addition of properties and siblings.
Moreover, the rising Focus score also reveals a trend towards better
categorization. Overall, the results from these case studies substantiate the
effectiveness of our method in aligning and expanding knowledge graphs.

###  7.5 Summary

In this chapter, we introduced the LiveSchema platform to collect, process,
and analyze existing knowledge graphs, and discussed how to achieve the goal
of automatic knowledge graph extension utilizing the services provided by
LiveSchema. Initially, we delve into the data layer of the proposed platform,
encompassing the unified format and metadata of the collected catalogs,
details of data collection, and data pre-processing procedures. We then
shifted our focus to the six primary services offered by LiveSchema, detailing
the competencies in data analysis, knowledge graph embedding, and path
recommendation services. Following this, we elaborate on the knowledge graph
extension algorithm, which represents the core purpose of the platform. This
includes an explanation of how to leverage the platform’s services to
facilitate knowledge graph extension, such as similarity calculation, entity
type recognition process, and knowledge graph assessment. Finally, we validate
the efficacy of the extension through two case studies. These case studies
demonstrate the effectiveness of the knowledge graph extension both
quantitatively and qualitatively.

##  Chapter 8 Conclusion

###  8.1 Main Conclusion

The construction of knowledge graphs offers critical support for systems
reliant on structured data, particularly for contemporary research in
artificial intelligence. Within this context, knowledge graph extension
emerges as a pivotal technology, focused on augmenting both the depth and
scope of the knowledge graph via the reuse of existing knowledge. Firstly, we
discuss and define several primary challenges, i.e., (1) the necessity of a
standardized framework to model the task of automatic knowledge graph
extension; (2) the need for precise aligning of knowledge graph concepts to
address the description diversity arising from semantic heterogeneity; (3) the
requirement of suitable assessment methods to verify the effectiveness of the
extended knowledge graph; (4) the demand for a unified platform to customize
knowledge graph extension services with consistent processing. In response to
these challenges, our main contributions are as follows: (1) we introduce a
framework that organizes functionalities into four stages for achieving
automatic knowledge graph extension; (2) we propose a method for entity type
recognition that employs machine learning models and property-based similarity
metrics to improve the accuracy of knowledge extraction; (3) we suggest a set
of assessing metrics to validate the quality of the extended knowledge graph;
(4) we develop an online platform, namely LiveSchema, which integrates
functionalities for the acquisition, management, and extension services of
knowledge graphs to benefit knowledge engineers. These principal contributions
are then introduced and described in detail in the subsequent chapters.

Therefore, we introduced the modeling of the automatic knowledge graph
extension task in Chapter 3. We elaborate on the overall extension framework
and its associated functionalities to guide the subsequent contents. At the
same time, we also define the utilized data and terminologies in symbolic
language.

In Chapter 4, our focus is primarily on the preparation steps required for
entity type recognition. We start by analyzing existing knowledge graphs and
discuss the rationale behind using properties to measure similarity between
entity types. Additionally, we introduce a method of knowledge formalization
based on formal concept analysis lattices. Following this, we present
property-based similarity metrics and theoretically contrast them with
existing similarity metrics. Finally, we validate the proposed metrics through
qualitative experiments.

Chapter 5 presents the machine learning-based entity type recognition method.
Initially, we introduce the overall pipeline and its components, which include
knowledge pre-processing, the computation of property-based similarity
metrics, and machine learning-based recognizers. We distinguish between
schema-level and instance-level recognition, as well as the alignment of
properties from various sources. A thorough discussion on the selection of
learning algorithms is provided, incorporating both classic machine learning
models and novel neural networks explored in several studies. Subsequently, we
delve into model training, detailing the training procedures and parameter
configurations. Lastly, we conduct a series of experiments to evaluate the
performance of entity type recognition and the proposed property-based
similarity metrics, both quantitatively and qualitatively.

Chapter 6 is primarily focused on introducing a set of metrics for assessing
the quality of extended knowledge graphs. We begin by discussing several
existing metrics used in knowledge graph assessment. Following this, we
propose our assessing metrics, namely Focus, which are designed to measure the
categorization purpose of given entity types based on their associated
properties. The experimental results affirm the validity of the Focus metrics.

In Chapter 7, we develop an online knowledge management platform, i.e.,
Liveschema, with the goal of integrating all the introduced functionalities
into a unified system. We detail the data and service layers of the platform,
visually depicting Liveschema’s development. Importantly, we introduce a
knowledge graph extension algorithm that utilizes the functionalities provided
by Liveschema. Finally, we present multiple case studies to demonstrate the
effectiveness of the extended knowledge graph.

###  8.2 Prospects for Future Work

In this concluding section of our final chapter, we wish to deliberate on
potential avenues for future research. Firstly, there is scope for the entity
type recognition method to extend its applicability. Although its current
purpose is to identify entity types and entities within candidate knowledge
graphs, heterogeneous structured data in real-world research are not confined
solely to knowledge graphs but also encompass knowledge bases and tabular
data. However, the challenge posed by knowledge conflicts is likely to become
more difficult with the increase in data sources. In this paper, we have
discussed methods specific to the tasks of entity alignment and knowledge
extension in the context of knowledge graph extension, as well as
corresponding strategies for conflict resolution. We envisage that the
application of the method to a wider range of data types could lead to
enhanced depth and accuracy in knowledge extraction. This necessitates further
development of the refinement and completion functions of knowledge graphs,
which is an insufficient part of our work. Concurrently, we hope for the
recognition method to increase processing speeds to facilitate real-time
knowledge extraction. Such advancements would significantly boost the
effectiveness of the proposed services to knowledge engineers.

With the ongoing progression in the field of artificial intelligence, the
demand for structured data represented by knowledge graphs is also increasing.
Knowledge graphs frequently serve as a trusted data source for the validation
of content generated by artificial intelligence. Consequently, a second
potential area for enhancement involves customizing knowledge management
services for users based on their unique needs, especially for those requiring
more precise domain knowledge. This would entail selectively extracting and
integrating target data through a detailed analysis of user requirements.
Given the strong foundational information generation capability of large pre-
trained language models and their anticipated widespread application scenarios
in the future, mining the trustworthiness attributes of knowledge graphs will
contribute to addressing the hallucination issues of large language models.
The exploration of how to represent the vast amount of domain-specific,
trustworthy knowledge within knowledge graphs appropriately and interact with
large language models will become an important area of research.

The last direction represents a promising research topic, knowledge graph
evolution, namely the continuous enhancement of the reference knowledge graph
through the use of streaming data. This procedure can be modeled as a
continuous process of knowledge graph extension utilizing multiple data
sources. A key challenge is maintaining the integrity and quality of the
reference knowledge graph while integrating a multitude of data. One potential
approach to address this is by leveraging the concept of continuous learning.
In this context, the knowledge graphs are possible to have further refinements
after the extension operation, and consequently, to update machine learning
models for the ensuing cycle of knowledge extraction and extension.

## Bibliography

  * [1] Harith Alani and Christopher Brewster.  Metrics for ranking ontologies. 4th int.  In EON Workshop, 15th Int. World Wide Web Conference, 2006.
  * [2] Alsayed Algergawy, Samira Babalou, Mohammad J Kargar, and S Hashem Davarpanah.  Seecont: A new seeding-based clustering approach for ontology matching.  In Advances in Databases and Information Systems: 19th East European Conference, ADBIS 2015, Poitiers, France, September 8-11, 2015, Proceedings 19, pages 245–258. Springer, 2015.
  * [3] Alsayed Algergawy, Michelle Cheatham, Daniel Faria, Alfio Ferrara, Irini Fundulaki, Ian Harrow, Sven Hertling, Ernesto Jiménez-Ruiz, Naouel Karam, Abderrahmane Khiat, et al.  Results of the ontology alignment evaluation initiative 2018.  In 13th International Workshop on Ontology Matching co-located with the 17th ISWC (OM 2018), volume 2288, pages 76–116, 2018.
  * [4] Alsayed Algergawy, Sabine Massmann, and Erhard Rahm.  A clustering-based approach for large-scale ontology matching.  In Advances in Databases and Information Systems: 15th International Conference, ADBIS 2011, Vienna, Austria, September 20-23, 2011. Proceedings 15, pages 415–428. Springer, 2011.
  * [5] Mehdi Ali, Max Berrendorf, Charles Tapley Hoyt, Laurent Vermue, Sahand Sharifzadeh, Volker Tresp, and Jens Lehmann.  Pykeen 1.0: a python library for training and evaluating knowledge graph embeddings.  The Journal of Machine Learning Research, 22(1):3723–3728, 2021.
  * [6] Peshawa Jamal Muhammad Ali, Rezhna Hassan Faraj, Erbil Koya, Peshawa J Muhammad Ali, and Rezhna H Faraj.  Data normalization and standardization: a technical report.  Mach Learn Tech Rep, 1(1):1–6, 2014.
  * [7] Siham Amrouch, Sihem Mostefai, and Muhammad Fahad.  Decision trees in automatic ontology matching.  International Journal of Metadata, Semantics and Ontologies, 11(3):180–190, 2016.
  * [8] Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives.  Dbpedia: A nucleus for a web of open data.  In The semantic web, pages 722–735. Springer, 2007.
  * [9] Hannah Bast, Florian Bäurle, Björn Buchhold, and Elmar Haußmann.  Easy access to the freebase dataset.  In Proceedings of the 23rd International Conference on World Wide Web, pages 95–98, 2014.
  * [10] Gabor Bella, Fausto Giunchiglia, and Fiona McNeill.  Language and domain aware lightweight ontology matching.  Journal of Web Semantics, 43:1–17, 2017.
  * [11] Gábor Bella, Alessio Zamboni, and Fausto Giunchiglia.  Domain-based sense disambiguation in multilingual structured data.  In The Diversity Workshop at the European Conference on Artificial Intelligence (ECAI), 2016.
  * [12] Alexandre Bento, Amal Zouaq, and Michel Gagnon.  Ontology matching using convolutional neural networks.  In Proceedings of the 12th language resources and evaluation conference, pages 5648–5653, 2020.
  * [13] Russa Biswas, Jan Portisch, Heiko Paulheim, Harald Sack, and Mehwish Alam.  Entity type prediction leveraging graph walks and entity descriptions.  In International Semantic Web Conference, pages 392–410. Springer, 2022.
  * [14] Russa Biswas, Radina Sofronova, Harald Sack, and Mehwish Alam.  Cat2type: Wikipedia category embeddings for entity typing in knowledge graphs.  In Proceedings of the 11th on Knowledge Capture Conference, pages 81–88, 2021.
  * [15] Christian Bizer, Tom Heath, and Tim Berners-Lee.  Linked data: The story so far.  In Semantic services, interoperability and web applications: emerging concepts, pages 205–227. IGI Global, 2011.
  * [16] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.  Freebase: a collaboratively created graph database for structuring human knowledge.  In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250, 2008.
  * [17] Antoine Bordes and Evgeniy Gabrilovich.  Constructing and mining web-scale knowledge graphs: Kdd 2014 tutorial.  In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1967–1967, 2014.
  * [18] E Bornemann, JH Doveton, et al.  Log normalization by trend surface analysis.  The log analyst, 22(04), 1981.
  * [19] Jos De Bruijn, Thomas Eiter, Axel Polleres, and Hans Tompits.  Embedding nonground logic programs into autoepistemic logic for knowledge-base combination.  ACM Transactions on Computational Logic (TOCL), 12(3):1–39, 2011.
  * [20] Lev Bulygin and Sergey A Stupnikov.  Applying of machine learning techniques to combine string-based, language-based and structure-based similarity measures for ontology matching.  In DAMDID/RCDL, pages 129–147, 2019.
  * [21] Anila Sahar Butt, Armin Haller, and Lexing Xie.  Ontology search: An empirical evaluation.  In The Semantic Web–ISWC 2014: 13th International Semantic Web Conference, Riva del Garda, Italy, October 19-23, 2014. Proceedings, Part II 13, pages 130–147. Springer, 2014.
  * [22] Anila Sahar Butt, Armin Haller, and Lexing Xie.  Dwrank: Learning concept ranking for ontology search.  Semantic Web, 7(4):447–461, 2016.
  * [23] Nicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, et al.  Proceedings of the 12th language resources and evaluation conference.  In Proceedings of The 12th Language Resources and Evaluation Conference, 2020.
  * [24] Andrew Carlson, Justin Betteridge, Richard C Wang, Estevam R Hruschka Jr, and Tom M Mitchell.  Coupled semi-supervised learning for information extraction.  In Proceedings of the third ACM international conference on Web search and data mining, pages 101–110, 2010.
  * [25] Michelle Cheatham and Pascal Hitzler.  String similarity metrics for ontology alignment.  In International semantic web conference, pages 294–309. Springer, 2013.
  * [26] Guowei Chen and Songmao Zhang.  Identifying mappings among knowledge graphs by formal concept analysis.  In OM@ ISWC, pages 25–35, 2019.
  * [27] Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.
  * [28] Yang Chi, Fausto Giunchiglia, Daqian Shi, Xiaolei Diao, Chuntao Li, and Hao Xu.  Zinet: Linking chinese characters spanning three thousand years.  In Findings of the Association for Computational Linguistics: ACL 2022, pages 3061–3070, 2022.
  * [29] Bonggeun Choi and Youngjoong Ko.  Knowledge graph extension with a pre-trained language model via unified learning method.  Knowledge-Based Systems, 262:110245, 2023.
  * [30] Kenneth Ward Church.  Word2vec.  Natural Language Engineering, 23(1):155–162, 2017.
  * [31] William Cohen, Pradeep Ravikumar, and Stephen Fienberg.  A comparison of string metrics for matching names and records.  In Kdd workshop on data cleaning and object consolidation, volume 3, pages 73–78, 2003.
  * [32] Olivier C Curé, Henri Maurer, Nigam H Shah, and Paea Le Pendu.  A formal concept analysis and semantic query expansion cooperation to refine health outcomes of interest.  BMC medical informatics and decision making, 15:1–6, 2015.
  * [33] Belur V Dasarathy.  Nearest neighbor (nn) norms: Nn pattern classification techniques.  IEEE Computer Society Tutorial, 1991.
  * [34] Auriol Degbelo.  A snapshot of ontology evaluation criteria and strategies.  In Proceedings of the 13th International Conference on Semantic Systems, pages 1–8, 2017.
  * [35] Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Xiaobin Wang, Pengjun Xie, Haitao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Gee Kim.  Prompt-learning for fine-grained entity typing.  In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6888–6901, 2022.
  * [36] Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang.  Knowledge vault: A web-scale approach to probabilistic knowledge fusion.  In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 601–610, 2014.
  * [37] Alishiba Dsouza, Nicolas Tempelmeier, and Elena Demidova.  Towards neural schema alignment for openstreetmap and knowledge graphs.  In International Semantic Web Conference, pages 56–73. Springer, 2021.
  * [38] Islam Akef Ebeid, Majdi Hassan, Tingyi Wanyan, Jack Roper, Abhik Seal, and Ying Ding.  Biomedical knowledge graph refinement and completion using graph representation learning and top-k similarity measure.  In Diversity, Divergence, Dialogue: 16th International Conference, iConference 2021, Beijing, China, March 17–31, 2021, Proceedings, Part I 16, pages 112–123. Springer, 2021.
  * [39] Jérôme Euzenat, Alfio Ferrara, Laura Hollink, Antoine Isaac, Cliff Joslyn, Véronique Malaisé, Christian Meilicke, Andriy Nikolov, Juan Pane, Marta Sabou, et al.  Results of the ontology alignment evaluation initiative 2009.  In Proc. 4th ISWC workshop on ontology matching (OM), pages 73–126. No commercial editor., 2009.
  * [40] Jérôme Euzenat, Pavel Shvaiko, et al.  Ontology matching, volume 18.  Springer, 2007.
  * [41] Muhammad Fahad.  Initial results for ontology matching workshop 2015 dkp-aom: results for oaei 2015.  In CEUR Workshop Proceedings, 2015.
  * [42] RA Falbo, MP Barcellos, FB Ruy, G Guizzardi, and RSS Guizzardi.  Ontology pattern languages.  In Ontology Engineering with Ontology Design Patterns: Foundations and Applications. IOS Press, 2016.
  * [43] Michael Färber, Basil Ell, Carsten Menne, and Achim Rettinger.  A comparative survey of dbpedia, freebase, opencyc, wikidata, and yago.  Semantic Web Journal, 1(1):1–5, 2015.
  * [44] Daniel Faria, Catia Pesquita, Emanuel Santos, Isabel F Cruz, and Francisco M Couto.  Agreementmakerlight 2.0: Towards efficient large-scale ontology matching.  In ISWC (Posters & Demos), pages 457–460, 2014.
  * [45] Daniel Faria, Catia Pesquita, Emanuel Santos, Matteo Palmonari, Isabel F Cruz, and Francisco M Couto.  The agreementmakerlight ontology matching system.  In On the Move to Meaningful Internet Systems: OTM 2013 Conferences: Confederated International Conferences: CoopIS, DOA-Trusted Cloud, and ODBASE 2013, Graz, Austria, September 9-13, 2013. Proceedings, pages 527–541. Springer, 2013.
  * [46] Christiane Fellbaum.  WordNet: An electronic lexical database.  MIT press, 1998.
  * [47] Steven R Finch.  Mathematical constants.  Cambridge university press, 2003.
  * [48] Mattia Fumagalli, Gábor Bella, Samuele Conti, and Fausto Giunchiglia.  Ontology-driven cross-domain transfer learning.  In Formal Ontology in Information Systems, pages 249–263. IOS Press, 2020.
  * [49] Mattia Fumagalli, Marco Boffo, Daqian Shi, Mayukh Bagchi, and Fausto Giunchiglia.  Liveschema: A gateway towards learning on knowledge graph schemas.  Proceeding of the 13th international conference on formal ontology in information systems., 2023.
  * [50] Mattia Fumagalli, Daqian Shi, and Fausto Giunchiglia.  Ranking schemas by focus: A cognitively-inspired approach.  In International Conference on Conceptual Structures, pages 73–88. Springer, 2021.
  * [51] Aldo Gangemi, Carola Catenacci, Massimiliano Ciaramita, and Jos Lehmann.  A theoretical framework for ontology evaluation and validation.  In SWAP, volume 166, page 16. Citeseer, 2005.
  * [52] B. Ganter and R. Wille.  Formal concept analysis: mathematical foundations.  Springer, 2012.
  * [53] Fausto Giunchiglia, Aliaksandr Autayeu, and Juan Pane.  S-match: an open source framework for matching lightweight ontologies.  Semantic Web, 3(3):307–317, 2012.
  * [54] Fausto Giunchiglia and Mattia Fumagalli.  Teleologies: Objects, actions and functions.  In International conference on conceptual modeling, pages 520–534. Springer, 2017.
  * [55] Fausto Giunchiglia and Mattia Fumagalli.  On knowledge diversity.  In JOWO, 2019.
  * [56] Fausto Giunchiglia and Mattia Fumagalli.  On knowledge diversity.  Proceedings of the 2019 Joint Ontology Workshops, WOMoCoE, 2019.
  * [57] Fausto Giunchiglia and Mattia Fumagalli.  Entity type recognition–dealing with the diversity of knowledge.  In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, volume 17, pages 414–423, 2020.
  * [58] Fausto Giunchiglia, Daqian Shi, et al.  Property-based entity type graph matching.  In CEUR Workshop Proceedings, volume 3063, pages 1–12. OM, ISWC 2021, 2021.
  * [59] Fausto Giunchiglia, Pavel Shvaiko, and Mikalai Yatskevich.  S-match: an algorithm and an implementation of semantic matching.  In The Semantic Web: Research and Applications: First European Semantic Web Symposium, ESWS 2004 Heraklion, Crete, Greece, May 10-12, 2004. Proceedings 1, pages 61–75. Springer, 2004.
  * [60] Fausto Giunchiglia, Ilya Zaihrayeu, and Uladzimir Kharkevich.  Formalizing the get-specific document classification algorithm.  In International Conference on Theory and Practice of Digital Libraries, pages 26–37. Springer, 2007.
  * [61] Jorge Gracia, Vanessa López, Mathieu D’Aquin, Marta Sabou, Enrico Motta, and Eduardo Mena.  Solving semantic ambiguity to improve semantic web based ontology matching.  In Proceedings of the 2nd International Conference on Ontology Matching-Volume 304, pages 1–12, 2007.
  * [62] Ramanathan V Guha, Dan Brickley, and Steve Macbeth.  Schema. org: evolution of structured data on the web.  Communications of the ACM, 59(2):44–51, 2016.
  * [63] Stevan Harnad.  To cognize is to categorize: Cognition is categorization.  In Handbook of categorization in cognitive science, pages 21–54. Elsevier, 2017.
  * [64] Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al.  Knowledge graphs.  ACM Computing Surveys (Csur), 54(4):1–37, 2021.
  * [65] Wei Hu, Qingheng Zhang, Zequn Sun, and Jiacheng Huang.  Multike: a multi-view knowledge graph embedding framework for entity alignment.  In OM@ ISWC, pages 189–190, 2019.
  * [66] Y Kumar Jain and Santosh Kumar Bhandare.  Min max normalization based data perturbation method for privacy protection.  International Journal of Computer & Communication Technology, 2(8):45–50, 2011.
  * [67] Xiaotian Jiang, Quan Wang, Baoyuan Qi, Yongqin Qiu, Peng Li, and Bin Wang.  Attentive path combination for knowledge graph completion.  In Asian conference on machine learning, pages 590–605. PMLR, 2017.
  * [68] Ernesto Jiménez-Ruiz.  Logmap family participation in the oaei 2019.  In CEUR Workshop Proceedings. CEUR-WS. org, 2019.
  * [69] Ernesto Jiménez-Ruiz and Bernardo Cuenca Grau.  Logmap: Logic-based and scalable ontology matching.  In The Semantic Web–ISWC 2011: 10th International Semantic Web Conference, Bonn, Germany, October 23-27, 2011, Proceedings, Part I 10, pages 273–288. Springer, 2011.
  * [70] Fasihul Kabir, Sabbir Siddique, Mohammed Rokibul Alam Kotwal, and Mohammad Nurul Huda.  Bangla text document categorization using stochastic gradient descent (sgd) classifier.  In 2015 International Conference on Cognitive Computing and Information Processing (CCIP), pages 1–4. IEEE, 2015.
  * [71] Bogumił Kamiński, Michał Jakubczyk, and Przemysław Szufel.  A framework for sensitivity analysis of decision trees.  Central European journal of operations research, 26(1):135–159, 2018.
  * [72] Prodromos Kolyvakis, Alexandros Kalousis, and Dimitris Kiritsis.  Deepalignment: Unsupervised ontology matching with refined word vectors.  In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1-6 June 2018, 2018.
  * [73] Grzegorz Kondrak.  N-gram similarity and distance.  In International symposium on string processing and information retrieval, pages 115–126. Springer, 2005.
  * [74] Md Tahmid Rahman Laskar, Xiangji Huang, and Enamul Hoque.  Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task.  In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5505–5514, 2020.
  * [75] Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer.  End-to-end neural coreference resolution.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2017.
  * [76] Jens Lehmann and Lorenz Bühmann.  Ore-a tool for repairing and enriching knowledge bases.  In International semantic web conference, pages 177–193. Springer, 2010.
  * [77] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al.  Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia.  Semantic web, 6(2):167–195, 2015.
  * [78] Maurizio Lenzerini.  Data integration: A theoretical perspective.  In Proceedings of the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 233–246, 2002.
  * [79] Jian Li, Ziyao Meng, Daqian Shi, Rui Song, Xiaolei Diao, Jingwen Wang, and Hao Xu.  Fcc: Feature clusters compression for long-tailed visual recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24080–24089, 2023.
  * [80] John Li.  Lom: A lexicon-based ontology mapping tool.  Proceedings of the Performance Metrics for Intelligent Systems (PerMIS, page 2004, 2004.
  * [81] Juanzi Li, Jie Tang, Yi Li, and Qiong Luo.  Rimom: A dynamic multistrategy ontology alignment framework.  IEEE Transactions on Knowledge and data Engineering, 21(8):1218–1232, 2008.
  * [82] Feiyu Lin and Kurt Sandkuhl.  A survey of exploiting wordnet in ontology matching.  In IFIP International Conference on Artificial Intelligence in Theory and Practice, pages 341–350. Springer, 2008.
  * [83] Xiao Ling and Daniel Weld.  Fine-grained entity recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 94–100, 2012.
  * [84] Deryle Lonsdale, David W Embley, Yihong Ding, Li Xu, and Martin Hepp.  Reusing ontologies and language components for ontology generation.  Data & Knowledge Engineering, 69(4):318–330, 2010.
  * [85] Melinda McDaniel and Veda C Storey.  Evaluating domain ontologies: Clarification, classification, and challenges.  ACM Computing Surveys (CSUR), 52(4):1–44, 2019.
  * [86] Ruth Garrett Millikan.  Beyond concepts: Unicepts, language, and natural information.  Oxford University Press, 2017.
  * [87] Ishani Mondal, Yufang Hou, and Charles Jochim.  End-to-end construction of nlp knowledge graph.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1885–1895, 2021.
  * [88] Ndapandula Nakashole, Martin Theobald, and Gerhard Weikum.  Scalable knowledge harvesting with high precision and high recall.  In Proceedings of the fourth ACM international conference on Web search and data mining, pages 227–236, 2011.
  * [89] Souradip Nath and Ruchira Naskar.  Automated image splicing detection using deep cnn-learned features and ann-based classifier.  Signal, Image and Video Processing, 15(7):1601–1608, 2021.
  * [90] Azadeh Haratian Nezhadi, Bita Shadgar, and Alireza Osareh.  Ontology alignment using machine learning techniques.  International Journal of Computer Science & Information Technology, 3(2):139, 2011.
  * [91] Andrew Y Ng and Michael I Jordan.  On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes.  In Advances in neural information processing systems, pages 841–848, 2002.
  * [92] Ngonga Ngomo.  Applying edge-counting semantic similarities to link discovery: Scalability and accuracy.  Matching (OM 2020), page 36, 2020.
  * [93] Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich.  A review of relational machine learning for knowledge graphs.  Proceedings of the IEEE, 104(1):11–33, 2015.
  * [94] Mathias Niepert, Christian Meilicke, and Heiner Stuckenschmidt.  A probabilistic-logical framework for ontology matching.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1413–1418, 2010.
  * [95] Ian Niles and Adam Pease.  Mapping wordnet to the sumo ontology.  In Proceedings of the ieee international knowledge engineering conference, pages 23–26. Citeseer, 2003.
  * [96] Ikechukwu Nkisi-Orji, Nirmalie Wiratunga, Stewart Massie, Kit-Ying Hui, and Rachel Heaven.  Ontology alignment based on word embedding and random forest classification.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 557–572. Springer, 2018.
  * [97] Ikechukwu Nkisi-Orji, Nirmalie Wiratunga, Stewart Massie, Kit-Ying Hui, and Rachel Heaven.  Ontology alignment based on word embedding and random forest classification.  In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10–14, 2018, Proceedings, Part I 18, pages 557–572. Springer, 2019.
  * [98] Daniel Obraczka, Jonathan Schuchart, and Erhard Rahm.  Embedding-assisted entity resolution for knowledge graphs.  In Second International Workshop on Knowledge Graph Construction, 2021.
  * [99] Yasumasa Onoe and Greg Durrett.  Fine-grained entity typing for domain independent entity linking.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8576–8583, 2020.
  * [100] Mahesh Pal.  Random forest classifier for remote sensing classification.  International journal of remote sensing, 26(1):217–222, 2005.
  * [101] M Palmer and Z Wu.  Verb semantics and lexical zhibiao w u.  In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, Las Cruces, New Mexico, pages 133–138, 1994.
  * [102] Heiko Paulheim.  Knowledge graph refinement: A survey of approaches and evaluation methods.  Semantic web, 8(3):489–508, 2017.
  * [103] H Sofia Pinto, Asunción Gómez-Pérez, and João P Martins.  Some issues on ontology integration.  In Proceedings of the IJCAI, volume 99, pages 7–1. Citeseer, 1999.
  * [104] Jan Portisch, Michael Hladik, and Heiko Paulheim.  Background knowledge in schema matching: Strategy vs. data.  In International Semantic Web Conference, pages 287–303. Springer, 2021.
  * [105] Nikooie Pour, Alsayed Algergawy, Reihaneh Amini, Daniel Faria, Irini Fundulaki, Ian Harrow, Sven Hertling, Ernesto Jiménez-Ruiz, Clement Jonquet, Naouel Karam, et al.  Results of the ontology alignment evaluation initiative 2020.  In Proceedings of the 15th International Workshop on Ontology Matching (OM 2020), volume 2788, pages 92–138. CEUR-WS, 2020.
  * [106] Liu Qiao, Li Yang, Duan Hong, Liu Yao, and Qin Zhiguang.  Knowledge graph construction techniques.  Journal of computer research and development, 53(3):582–600, 2016.
  * [107] Philip Resnik.  Using information content to evaluate semantic similarity in a taxonomy.  In Proceedings of the 14th international joint conference on Artificial intelligence-Volume 1, pages 448–453, 1995.
  * [108] Ray Richardson, A Smeaton, and John Murphy.  Using wordnet as a knowledge base for measuring semantic similarity between words, 1994.
  * [109] Ana B Rios-Alvarado, Ivan Lopez-Arevalo, and Victor J Sosa-Sosa.  Learning concept hierarchies from textual resources for ontologies construction.  Expert Systems with Applications, 40(15):5907–5915, 2013.
  * [110] Eric Sven Ristad and Peter N Yianilos.  Learning string-edit distance.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(5):522–532, 1998.
  * [111] Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al.  Okapi at trec-3.  Nist Special Publication Sp, 109:109, 1995.
  * [112] E. Rosch.  Principles of categorization.  Concepts: core readings, 189, 1999.
  * [113] Eleanor Rosch and Carolyn B Mervis.  Family resemblances: Studies in the internal structure of categories.  Cognitive psychology, 7(4):573–605, 1975.
  * [114] Stuart J Russell and Peter Norvig.  Artificial intelligence a modern approach; pearsoneducation.  Artificial Intelligence: A Modern Approach: Pearson Education, 2003.
  * [115] S Rasoul Safavian and David Landgrebe.  A survey of decision tree classifier methodology.  IEEE transactions on systems, man, and cybernetics, 21(3):660–674, 1991.
  * [116] Tomohiro Saito and Yutaka Watanobe.  Learning path recommendation system for programming education based on neural networks.  International Journal of Distance Education Technologies (IJDET), 18(1):36–64, 2020.
  * [117] Gerard Salton and Christopher Buckley.  Term-weighting approaches in automatic text retrieval.  Information processing & management, 24(5):513–523, 1988.
  * [118] Jürgen Schmidhuber.  Deep learning in neural networks: An overview.  Neural networks, 61:85–117, 2015.
  * [119] Walid Shalaby, Khalifeh Al Jadda, Mohammed Korayem, and Trey Grainger.  Entity type recognition using an ensemble of distributional semantic models to enhance query understanding.  In 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC), volume 1, pages 631–636. IEEE, 2016.
  * [120] Chuan Shi, Jian Liu, Fuzhen Zhuang, Philip S Yu, and Bin Wu.  Integrating heterogeneous information via flexible regularization framework for recommendation.  Knowledge and Information Systems, 49:835–859, 2016.
  * [121] Daqian Shi and Fausto Giunchiglia.  Recognizing entity types via properties.  In Proceeding of the 13th international conference on formal ontology in information systems., 2023.
  * [122] Daqian Shi, Ting Wang, Hao Xing, and Hao Xu.  A learning path recommendation model based on a multidimensional knowledge graph framework for e-learning.  Knowledge-Based Systems, 195:105618, 2020.
  * [123] Lida Shi, Fausto Giunchiglia, Rui Song, Daqian Shi, Tongtong Liu, Xiaolei Diao, and Hao Xu.  A simple contrastive learning framework for interactive argument pair identification via argument-context extraction.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10027–10039, 2022.
  * [124] Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel.  Neural architectures for fine-grained entity type classification.  In 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, pages 1271–1280. Association for Computational Linguistics (ACL), 2017.
  * [125] Jennifer Sleeman and Tim Finin.  Type prediction for efficient coreference resolution in heterogeneous semantic graphs.  In 2013 IEEE Seventh International Conference on Semantic Computing, pages 78–85. IEEE, 2013.
  * [126] Jennifer Sleeman, Tim Finin, and Anupam Joshi.  Entity type recognition for heterogeneous semantic graphs.  AI Magazine, 36(1):75–86, 2015.
  * [127] Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng.  Reasoning with neural tensor networks for knowledge base completion.  Advances in neural information processing systems, 26, 2013.
  * [128] Geri Steve, Aldo Gangemi, and Domenico M Pisanelli.  Integrating medical terminologies with onions methodology. information modeling and knowledge bases viii, 1998.
  * [129] Gerd Stumme and Alexander Maedche.  Fca-merge: Bottom-up merging of ontologies.  In IJCAI, volume 1, pages 225–230, 2001.
  * [130] Yufei Sun, Liangli Ma, and Shuang Wang.  A comparative evaluation of string similarity metrics for ontology alignment.  Journal of Information &Computational Science, 12(3):957–964, 2015.
  * [131] Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu.  Bootstrapping entity alignment with knowledge graph embedding.  In IJCAI, volume 18, 2018.
  * [132] Bayu Distiawan Trisedya, Jianzhong Qi, and Rui Zhang.  Entity alignment between knowledge graphs using attribute embeddings.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 297–304, 2019.
  * [133] Tim Van Erven and Peter Harremos.  Rényi divergence and kullback-leibler divergence.  IEEE Transactions on Information Theory, 60(7):3797–3820, 2014.
  * [134] Pierre-Yves Vandenbussche, Ghislain Atemezing, María Poveda-Villalón, and Bernard Vatant.  Linked open vocabularies (lov): a gateway to reusable semantic vocabularies on the web.  Semantic Web Journal, 8:437–452, January 2017.
  * [135] Pierre-Yves Vandenbussche, Ghislain A Atemezing, María Poveda-Villalón, and Bernard Vatant.  Linked open vocabularies (lov): a gateway to reusable semantic vocabularies on the web.  Semantic Web, 8(3):437–452, 2017.
  * [136] Denny Vrandečić and Markus Krötzsch.  Wikidata: a free collaborative knowledgebase.  Communications of the ACM, 57(10):78–85, 2014.
  * [137] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo.  Knowledge graph embedding: A survey of approaches and applications.  IEEE Transactions on Knowledge and Data Engineering, 29(12):2724–2743, 2017.
  * [138] Ting Wang, Daqian Shi, Zhaodan Wang, Shuai Xu, and Hao Xu.  Mrp2rec: Exploring multiple-step relation path semantics for knowledge graph-based recommendations.  IEEE Access, 8:134817–134825, 2020.
  * [139] Xiang Wang, Tinglin Huang, Dingxian Wang, Yancheng Yuan, Zhenguang Liu, Xiangnan He, and Tat-Seng Chua.  Learning intents behind interactions with knowledge graph for recommendation.  In Proceedings of the web conference 2021, pages 878–887, 2021.
  * [140] Kemas Wiharja, Jeff Z Pan, Martin Kollingbaum, and Yu Deng.  More is better: Sequential combinations of knowledge graph embedding approaches.  In Semantic Technology: 8th Joint International Conference, JIST 2018, Awaji, Japan, November 26–28, 2018, Proceedings 8, pages 19–35. Springer, 2018.
  * [141] Mark D Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E Bourne, et al.  The fair guiding principles for scientific data management and stewardship.  Scientific data, 3(1):1–9, 2016.
  * [142] Jinge Wu, Daqian Shi, Abul Hasan, and Honghan Wu.  Knowlab at radsum23: comparing pre-trained language models in radiology report summarization.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 535–540. ACL, 2023.
  * [143] Ji Xin, Yankai Lin, Zhiyuan Liu, and Maosong Sun.  Improving neural fine-grained entity typing with knowledge attention.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [144] Yadollah Yaghoobzadeh and Hinrich Schütze.  Multi-level representations for fine-grained typing of knowledge base entities.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 578–589, 2017.
  * [145] Li Yujian and Liu Bo.  A normalized levenshtein distance metric.  IEEE transactions on pattern analysis and machine intelligence, 29(6):1091–1095, 2007.
  * [146] Ondřej Zamazal and Vojtěch Svátek.  The ten-year ontofarm and its fertilization within the onto-sphere.  Journal of Web Semantics, 43:46–53, 2017.
  * [147] Chuxu Zhang, Huaxiu Yao, Chao Huang, Meng Jiang, Zhenhui Li, and Nitesh V Chawla.  Few-shot knowledge graph completion.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3041–3048, 2020.
  * [148] Yu Zhao, Sheng Gao, Patrick Gallinari, and Jun Guo.  Knowledge base completion by learning pairwise-interaction differentiated embeddings.  Data mining and knowledge discovery, 29:1486–1504, 2015.
  * [149] Yu Zhao, Anxiang Zhang, Ruobing Xie, Kang Liu, and Xiaojie Wang.  Connecting embeddings for knowledge graph entity typing.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6419–6428, 2020.
  * [150] Hao Zhu, Ruobing Xie, Zhiyuan Liu, and Maosong Sun.  Iterative entity alignment via joint knowledge embeddings.  In IJCAI, volume 17, pages 4258–4264, 2017.
  * [151] Changlong Zou, Jingmin An, and Guanyu Li.  Knowledge graph entity type prediction with relational aggregation graph attention network.  In European Semantic Web Conference, pages 39–55. Springer, 2022.