# R2RML Mappings in OBDA Systems:
Enabling Comparison among OBDA Tools

Manuel Namici
DIAG, Sapienza, University of Rome
<EMAIL_ADDRESS>
###### Abstract

In today’s large enterprises there is a significant increasing trend in the
amount of data that has to be stored and processed. To complicate this
scenario the complexity of organizing and managing a large collection of data,
structured according to a single, unified schema, makes so that there is
almost never a single place where to look to satisfy an information need.

The Ontology-Based Data Access (OBDA) paradigm aims at mitigating this
phenomenon by providing to the users of the system a unified and shared
conceptual view of the domain of interest (ontology), while still enabling the
data to be stored in different data sources, which are managed by a relational
database. In an OBDA system the link between the data stored at the sources
and the ontology is provided through a declarative specification given in
terms of a set of mappings.

In this work we focus on comparing two of the available systems for OBDA,
namely, Mastro and Ontop, by adopting OBDA specifications based on W3C
recommendations. We first show how support for R2RML mappings has been
integrated in Mastro, which was the last feature missing in order to enable
the system to use specifications based solely on W3C recommendations relevant
to OBDA. We then proceed in performing a comparison between these systems over
two OBDA specifications, the NPD Benchmark and the ACI specification.

Keywords— Ontology-Based Data Access, R2RML, Mastro, Ontop

##  1 Introduction

In today’s large enterprises there is a significant increasing trend in the
amount of data that has to be stored and processed. To complicate this
scenario the complexity of organizing and managing a large collection of data,
structured according to a single, unified schema, makes so that there is
almost never a single place where to look to satisfy an information need.

The Ontology-Based Data Access (OBDA) paradigm aims at providing to the users
of the system a unified and shared conceptual view of the domain of interest
(ontology), while still enabling the data to be stored in different data
sources, which are managed by a relational database. In an OBDA system the
link between the data stored at the sources and the ontology is provided
through a declarative specification given in terms of a set of mappings.

The interest in the adoption of the OBDA paradigm has lead to the creation of
prototype tools, that have evolved into full-fledged systems. Although, while
these tools effectively enable to answer queries over the ontology, their use
in industrial applications still represents a challenge due to the performance
requirements that have to be met.

Several studies on the performance comparison among tools that enable semantic
data integration have been performed, but to the best of our knowledge none of
them focused on comparing fully-implemented systems designed to work in the
OBDA setting, in order to gain insight on the advantages and disadvantages of
each of the approach adopted, as this requires all the systems to be able to
use completely-standard specifications.

In this work we focus on comparing two systems for OBDA, namely,
Mastro111https://www.obdasystems.com/mastro and
Ontop222https://ontop.inf.unibz.it/.

In particular, we first enable the use of R2RML mappings in Mastro, a tool for
Ontology-Based Data Access developed at Sapienza, University of Rome. R2RML is
the W3C recommendation for expressing mappings in an OBDA specification, and
was the last feature missing in order to enable Mastro to use a completely-
standard specification. We then proceed in performing a comparison between
these systems over two OBDA specifications, the NPD Benchmark and the ACI
specification.

The rest of this work is organized as follows: In Section 2, we briefly
describe the basic notions of the R2RML mapping language, and then show how
such mappings can be interpreted for their use in Mastro. In Section 3, we
discuss the comparison over the NPD Benchmark, a specification developed by
the University of Oslo, and adapted for its use as a benchmark in the OBDA
setting333https://github.com/ontop/npd-benchmark. In Section 4, we discuss the
comparison over an application of the OBDA paradigm, developed in
collaboration between Sapienza University of Rome and ACI
Informatica444http://www.informatica.aci.it/, that is used in a real
industrial setting to evaluate the benefits of the OBDA approach. Finally, in
Section 5 we show some conclusions and present some possible future works.

##  2 Adding R2RML Support to Mastro

_R2RML_ (RDB to RDF mapping language) is the W3C recommendation for expressing
mappings from relational databases to RDF datasets[9]. Such mappings provide
the ability to view existing relational data as RDF graphs, expressed in a
structure and target vocabulary of the mapping author’s choice.

Compared to other RDB to RDF mapping languages R2RML can be classified as a
general purpose mapping language that allows to express customized, domain-
specific mappings. An R2RML mapping is itself represented as an RDF graph. A
system that makes use of an R2RML mapping to provide access to an RDF dataset
from a relational database is called an R2RML _processor_. An R2RML processor
could, for example, generate an RDF dump of the relational data, or it could
offer a SPARQL endpoint over the _virtual_ RDF dataset represented by the
database and the R2RML mapping, through an interface that queries the
underlying database without explicitly materializing the dataset. The latter
scenario is particularly suitable in the case of a system that wants to
realize the OBDA paradigm and is the reason why it as been considered as a
solution to represent the mapping between the ontology and the data sources.

An overview of the main R2RML classes in the form of a UML class diagram is
provided in Figure 1. Moreover, Listing 1 provides a complete example of an
R2RML mapping that maps the example database shown in Figure 2 to the RDF
dataset shown in Listing 2. For ease of presentation we denote with rr: the
prefix for R2RML vocabulary terms, with rdf: the prefix for RDF vocabulary
terms, and ex: the prefix for the specific target domain terms.

Figure 1: An overview of the R2RML classes

⬇

1@prefix rr: <http://www.w3.org/ns/r2rml#> .

2@prefix ex: <http://data.example.com/> .

3

4ex:CustomersMap rr:logicalTable [

5 rr:sqlQuery "SELECT C_ID, C_NAME FROM customers" ;

6 ] ;

7 rr:subjectMap [

8 rr:template "http://data.example.com/customer/{C_ID}" ;

9 rr:class ex:Customer ;

10 ] ;

11 rr:predicateObjectMap [

12 rr:predicate ex:name ;

13 rr:objectMap [ rr:column "C_NAME" ]

14 ] .

15

16ex:ProductsMap rr:logicalTable [ rr:tableName "products" ] ;

17 rr:subjectMap [

18 rr:template "http://data.example.com/product/{P_ID}" ;

19 rr:class ex:Product

20 ] ;

21 rr:predicateObjectMap [

22 rr:predicate ex:price;

23 rr:objectMap [ rr:column "P_PRICE" ]

24 ] .

25

26ex:OrdersMap rr:logicalTable [ rr:tableName "orders" ] ;

27 rr:subjectMap [

28 rr:template "http://data.example.com/order/{O_ID}" ;

29 rr:class ex:Order ;

30 ] ;

31 rr:predicateObjectMap [

32 rr:predicate ex:customer ;

33 rr:objectMap [

34 rr:parentTriplesMap ex:CustomersMap ;

35 rr:joinCondition [

36 rr:child "C_ID" ;

37 rr:parent "C_ID" ;

38 ]

39 ] ;

40 rr:predicateObjectMap [

41 rr:predicate ex:product ;

42 rr:objectMap [

43 rr:parentTriplesMap ex:ProductsMap ;

44 rr:joinCondition [

45 rr:child "P_ID" ;

46 rr:parent "P_ID" ;

47 ]

48 ]

49 ] ;

50 rr:predicateObjectMap [

51 rr:predicate ex:quantity ;

52 rr:objectMap [ rr:column "QUANTITY" ]

53 ].

Listing 1:  Example of a complete R2RML mapping

⬇

1@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

2@prefix ex: <http://data.example.com/>

3

4# RDF triples generated by ex:CustomersMap

5<http://data.example.com/customer/3211> rdf:type ex:Customer .

6<http://data.example.com/customer/3211> ex:name "Alice" .

7

8<http://data.example.com/customer/3253> rdf:type ex:Customer .

9<http://data.example.com/customer/3253> ex:name "Bob" .

10

11# RDF triples generated by ex:ProductsMap

12<http://data.example.com/product/2532> rdf:type ex:Product .

13<http://data.example.com/product/2532> ex:price "12.00" .

14

15<http://data.example.com/product/2533> rdf:type ex:Product .

16<http://data.example.com/product/2533> ex:price "41.00" .

17

18# RDF triples generated by ex:OrdersMap

19<http://data.example.com/order/4301> rdf:type ex:Order ;

20 ex:customer <http://data.example.com/customer/3211> ;

21 ex:product <http://data.example.com/product/2532> ;

22 ex:quantity "1" .

23

24<http://data.example.com/orders/4302> rdf:type ex:Order ;

25 ex:customer <http://data.example.com/customer/3211> ;

26 ex:product <http://data.example.com/product/2533> ;

27 ex:quantity "1" .

28

29<http://data.example.com/orders/4303> rdf:type ex:Order ;

30 ex:customer <http://data.example.com/customer/3253> ;

31 ex:product <http://data.example.com/product/2532> ;

32 ex:quantity "3" .

Listing 2:  Example of the generated RDF dataset

Customers

Products

C_ID | C_NAME
---|---
3211 | Alice
3253 | Bob

P_ID | P_PRICE
---|---
2532 | 12.00
2533 | 41.00

Orders

O_ID | C_ID | P_ID | QUANTITY
---|---|---|---
4301 | 3211 | 2532 | 1
4302 | 3211 | 2533 | 1
4303 | 3253 | 2532 | 3

Figure 2: Example relational database

###  2.1 R2RML mappings in OBDA

Using Semantic Web technologies, the standard format for representing
knowledge is through RDF triples. RDB to RDF languages such as R2RML allow to
transform relational data into a set of RDF triples.

In OBDA the role of the mapping is to generate the set of membership
assertions that form the extensional level of the ontology, and this can be
represented by a _virtual_ RDF graph when using these technologies.

We now show how an OBDA mapping can be expressed in R2RML, and then proceed in
illustrating how this can be integrated into the Mastro system by showing a
how a set of R2RML triples maps can be converted into a set of mappings
partitioned into view predicates and ontology predicate mappings, and a set of
mappings in the latter form can be converted to a set of equivalent R2RML
triples maps.

An OBDA mapping is represented by an expression of the form:

$\Phi(\vec{x})\rightsquigarrow\Psi(\vec{y},\vec{t})$

where $\Phi(\vec{x})$ is a conjunctive query over the source schema, with free
variables $\vec{x}$, $\vec{y}\subseteq\vec{x}$, $\Psi(\vec{y},\vec{t})$ is a
conjunctive query over the ontology alphabet.

When trying to map the component of an OBDA mapping to the components of an
R2RML triples map, we face two problems: The first issue is how to represent
the query over the data source, that constitutes the left-hand side of the
mapping. The second issue is how to represent the conjunction on the right-
hand side, and how are the objects build from the values returned by the
logical table rows.

For what regards the first issue the most appropriate solution is to use the
effective SQL query of the logical table. This corresponds exactly to the
meaning of the query in the mapping assertion.

For what regards the second issue, if we apply the procedure for generating
RDF triples from a triples map, as defined by the standard, we obtain a set of
triples, all having the same subject, corresponding to a conjunction of atoms
composed by a unary atom for each class specified in the subject map, and a
binary predicate for each of the predicate map-object map pair.

######  Example 2.1.

Let’s consider, for example, the triples map ex:CustomersMap of Listing 1:

⬇

ex:CustomersMap rr:logicalTable [

rr:sqlQuery "SELECT C_ID, C_NAME FROM customers" ;

] ;

rr:subjectMap [

rr:template "http://data.example.com/customer/{C_ID}" ;

rr:class ex:Customer ;

] ;

rr:predicateObjectMap [

rr:predicate ex:name ;

rr:objectMap [ rr:column "C_NAME" ]

] .

ex:ProductsMap rr:logicalTable [ rr:tableName "products" ] ;

This triples map specifies that, for each logical table row resulting from the
evaluation of the SQL query:

⬇

SELECT C_ID, C_NAME FROM customers

the system should produce the following set of assertions in the virtual RDF
graph:

⬇

ex:customer/{C_ID} rdf:type ex:Customer .

ex:customer/{C_ID} ex:name "{C_NAME}" .

where `{C_ID}` is replaced with the value of the column `C_ID`, and `{C_NAME}`
is replaced with the value of the column `C_NAME` in the logical table row.
This is equivalent to a mapping assertion of the form:

⬇

SELECT C_ID, C_NAME

FROM customers

$\rightsquigarrow$

⬇

ex:Customer(cust(C_ID)),

ex:name(cust(C_ID), C_NAME)

where we replaced the IRIs that represent the subjects with the function term
$cust$. In fact, in R2RML, the role of the constructors of the individuals in
the ontology from the values stored in the database is played by the _string
templates_. This means that we can see each different string template in the
mapping as a different function symbol, whose arity is the same as the number
of placeholders in the template.

Additionally, in this setting, a mapping assertion of the previous form has
been shown[16] to be equivalent to the following set of mappings:

⬇

SELECT C_ID

FROM customers

$\rightsquigarrow$

⬇

ex:Customer(cust(C_ID))

⬇

SELECT C_ID, C_NAME

FROM customers

$\rightsquigarrow$

⬇

ex:name(cust(C_ID), C_NAME)

######  Example 2.2.

Let’s now see the case of a triples map of the form corresponding to the one
of ex:OrdersMap in the example of Listing 1:

⬇

rr:template "http://data.example.com/order/{O_ID}" ;

rr:class ex:Order ;

] ;

rr:predicateObjectMap [

rr:predicate ex:customer ;

rr:objectMap [

rr:parentTriplesMap ex:CustomersMap ;

rr:joinCondition [

rr:child "C_ID" ;

rr:parent "C_ID" ;

]

] ;

rr:predicateObjectMap [

rr:predicate ex:product ;

rr:objectMap [

rr:parentTriplesMap ex:ProductsMap ;

rr:joinCondition [

rr:child "P_ID" ;

rr:parent "P_ID" ;

]

]

] ;

rr:predicateObjectMap [

rr:predicate ex:quantity ;

rr:objectMap [ rr:column "QUANTITY" ]

].

The logical table of this triples map specifies that the rows to consider are
directly those of the relation `orders` in the input database schema.
Moreover, the presence of the referencing object map imposes that, when
generating the instances of the roles ex:customer and ex:product, we need to
make sure that the objects we consider are only those that are subjects of the
assertions generated by the ex:CustomersMap and ex:ProductMap respectively. We
do this by using the joint SQL query of the referencing object map when
generating the mappings. Similarly to the previous case, the set of mappings
corresponding to this triples map are:

⬇

SELECT *

FROM orders

$\rightsquigarrow$

⬇

ex:Order(ord(O_ID))

⬇

SELECT *

FROM

(SELECT C_ID, C_NAME

FROM customers) AS parent,

(SELECT *

FROM orders) AS child

WHERE child.C_ID=parent.C.ID

$\rightsquigarrow$

⬇

ex:customer(ord(O_ID), cust(C_ID))

⬇

SELECT *

FROM

(SELECT *

FROM products) AS parent,

(SELECT *

FROM orders) AS child

WHERE child.P_ID=parent.P_ID

$\rightsquigarrow$

⬇

ex:product(ord(O_ID), prod(C_ID))

where we introduced two new function terms $ord$ and $prod$, corresponding to
the string templates of the customers and products respectively.

###  2.2 Importing R2RML Mappings in Mastro

Up to now we have seen how a set of R2RML mappings are interpreted as a form
of mappings that can be used in an OBDA system to generate the virtual ABox
assertions.

In Mastro, the set of mappings is composed by a pair
$\mathcal{M}=\langle\mathcal{M}_{v},\mathcal{M}_{o}\rangle$, where:

  * •

$\mathcal{M}_{v}$ is a set of assertions of the form:

$q_{DB}(\vec{x})\rightsquigarrow v(\vec{x})$

where $q_{DB}$ is an SQL query, and $v$ is a view predicate.

  * •

$\mathcal{M}_{o}$ is a set of assertions of the form:

$q_{v}(\vec{x})\rightsquigarrow P\left(\vec{x},\vec{t}\right)$

where $P$ is an atomic ontology predicate, built from function terms in
$\vec{t}$ applied over the variables in $\vec{x}$.

The view predicates act as an intermediate level of abstraction and have a
dual purpose: On one side they enable the database manager to focus on
improving the efficiency of the database management system, by having a
description of the relevant queries that will have to be performed. On the
other side they free the ontology designer from having to consider the
technical detail of the database schema.

In R2RML the closest notion to the views used in Mastro are the logical
tables. Unfortunately, the R2RML language does not provide the vocabulary for
expressing a logical table as an arbitrary conjunction of other logical tables
(except when using referencing object maps, which are restricted to express
foreign key relationships among logical tables), but relies completely on the
queries at the SQL level. This does not really pose a problem when importing
an R2RML mapping, as we can see each logical table as a view mapping, where
the associated query is the effective SQL query of the logical table, and
build the ontology mappings for each of the assertions generated by the
triples map using this newly generated view mapping. The challenge is how to
encode mappings that are expressed as arbitrary conjunction of view predicates
in R2RML. The solution we adopt in order to capture the full semantics of the
mappings in Mastro is to create a new logical table for each of this
conjunctions, where the corresponding SQL query is the unfolding of the query
over the views. In the following, we illustrate both processes by means of
some examples.

######  Example 2.3.

Starting from the example 2.1 from the previous section, in order to obtain a
set of mappings in the form used by Mastro, we need to introduce an auxiliary
view predicate, corresponding to the SQL query of the left-hand side of the
original mapping, and substitute it in the left-hand side of the previously
shown mappings:

  * •

$M_{v}$:
⬇ SELECT C_ID, C_NAME FROM customers $\rightsquigarrow$ ⬇ customers_view(C_ID,
C_NAME)

  * •

$M_{o}$:
⬇ customers_view(C_ID, C_NAME) $\rightsquigarrow$ ⬇ ex:Customer(cust(C_ID))

⬇

customers_view(C_ID, C_NAME)

$\rightsquigarrow$

⬇

ex:name(cust(C_ID), C_NAME)

######  Example 2.4 (Importing a mapping from a referencing object map).

When translating R2RML mappings of the type shown in example 2.2 we can
express the query of the referencing object map directly as queries over the
view predicates:

  * •

$M_{v}$:
⬇ SELECT C_ID, C_NAME FROM customers $\rightsquigarrow$ ⬇ customers_view(C_ID,
C_NAME)

⬇

SELECT P_ID, P_PRICE

FROM products

$\rightsquigarrow$

⬇

products_view(P_ID, P_PRICE)

⬇

SELECT O_ID, C_ID,

P_ID, QUANTITY

FROM orders

$\rightsquigarrow$

⬇

orders_view(O_ID, C_ID, P_ID, QUANTITY)

  * •

$M_{o}$:
⬇ orders_view(O_ID, C_ID, P_ID, QUANTITY) $\rightsquigarrow$ ⬇
ex:Order(ord(C_ID))

⬇

customers_view(C_ID,C_NAME)

orders_view(O_ID,C_ID,

P_ID,QUANTITY)

$\rightsquigarrow$

⬇

ex:customer(ord(O_ID), cust(C_ID))

⬇

products_view(P_ID,P_PRICE)

orders_view(O_ID,C_ID,

P_ID,QUANTITY)

$\rightsquigarrow$

⬇

ex:product(ord(O_ID), prod(P_ID))

where the join between the parent and child triples map is performed on the
left-hand side of the mappings in $M_{o}$ by performing unification over the
attributes stated in the join conditions.

###  2.3 Exporting Mastro Mappings in R2RML

The procedure for exporting a set of mappings expressed in the Mastro mapping
format is the dual of the one previously described, with an important
distinction: We need to introduce a new logical table for each of the
conjunctions on the left-hand side of the ontology predicate mappings in order
to be able to capture in R2RML the complete semantics that form the left-hand
side of the mapping, This is because R2RML does not allow to express logical
tables as arbitrary conjunctive queries of other logical tables. To obtain the
SQL query of the generated logical table we unfold the query over the view
predicates. During this translation all the constraints over the views, and
the other optimizations expressed in the internal mapping format have to be
discarded, as they cannot be expressed in R2RML.

######  Example 2.5 (Exporting a mapping with a single view atom).

Consider the case shown in example 2.3. In this case it is enough to generate
a logical table for the view `customers_view`, and a triples map for each of
the mappings corresponding to the right-hand side of the ontology predicate
mappings:

⬇

_:customer_view rr:sqlQuery " SELECT C_ID, C_NAME FROM customers "

ex:Customer rr:logicalTable _:customer_view

rr:subjectMap [

rr:template "http://data.example.com/customer/{C_ID}" ;

rr:class ex:Customer

]

ex:name rr:logicalTable _:customer_view

rr:subjectMap [

rr:template "http://data.example.com/customer/{C_ID}"

]

rr:predicateObjectMap [

rr:predicate ex:name ;

rr:objectMap [ rr:column "C_NAME" ]

]

######  Example 2.6 (Exporting a mapping with a conjunction of view atoms).

Now consider the mappings shown in example 2.4. The mappings for the roles
ex:customer and ex:products are expressed as a conjunction of view predicates.
Even though there may be cases where this type of assertion can be translated
with the use of a referencing object map, we cannot assume that this is always
the case. A general solution, that preserves the semantics of the mapping, is
to transform the conjunction on the left-hand side of both roles into a new
logical table, corresponding to the unfolding of the query over the views.
This means generating triples maps in the following form:

⬇

ex:customerMap rr:logicalTable [

rr:sqlQuery """

SELECT v1.C_ID AS C_ID, v2.O_ID AS O_ID

FROM

(SELECT C_ID, C_NAME

FROM customers) AS v1,

(SELECT O_ID, C_ID

FROM orders) AS v2

WHERE v1.C_ID=v2.C.ID

""".

] ;

rr:subjectMap [

rr:template "http://data.example.com/order/{O_ID}" ;

] ;

rr:predicateObjectMap [

rr:predicate ex:customer ;

rr:objectMap [

rr:template "http://data.example.com/customer/{C_ID}"

]

] .

##  3 Comparison on the NPD Benchmark

The increase in the rate of adoption of the OBDA paradigm has lead to the
creation of several prototype tools. Although, while these tools effectively
enable to answer queries over the ontology, their use in industrial
applications still represents a challenge due to the performance requirements
that have to be met.

Towards this direction, several benchmarks from the Semantic Web world have
been proposed as a way of evaluating the performance of an OBDA system [10,
3].

Unfortunately, all these benchmarks lack a fundamental property, that is the
presence of a complex and expressive ontology, which is required in order to
effectively evaluate the performance of an OBDA system in an industrial
application setting. For this purpose, recently a new benchmark has been
proposed, based on the _Norvegian Pretroleum Directorate (NPD) FactPages_
555http://factpages.npd.no/factpages/.

The Norwegian Petroleum Directorate is a governmental organization whose main
objective is to contribute to maximize the value that society can obtain from
the oil and gas activities [12]. The NPD FactPages contains information
regarding the petroleum activities on the Norwegian continental shelf. Such
information is actively used by oil companies like Statoil. The Factpages are
periodically synchronized with the NPD’s databases.

The NPD ontology has been mapped to the NPD FactPages and stored in a
relational database. Together with the ontology, the benchmark is provided
with a dump of the original database created from the NPD FactPages, the set
of mappings expressed in the R2RML mapping language, and a set of queries that
have been formulated by domain experts starting from an informal set of
questions provided to the users of the NPD FactPages.

In the following we will first describe the main characteristics of what
composes the NPD Benchmark666https://github.com/ontop/npd-benchmark
specification, with respect to version 1.9. We will then proceed in
illustrating the setup for our experimentation, with a presentation of the
results obtained.

#### Ontology

The NPD Ontology [20] describes activities on the Norwegian continental shelf
(NCS), e.g., about companies that own or operate petroleum fields, results of
tests taken during drilling, geographical data for physical installations and
the areas of fields and seismic surveys, transfers of shares of fields between
companies, and production results measured in volumes of petroleum [20]. The
ontology has been created by the University of Oslo, and presents rich
hierarchies of classes and properties, axioms that infer new objects, and
disjointness assertions.

The ontology is specified in OWL and for the purpose of the benchmark it has
been restricted to the fragment corresponding to the OWL 2 QL profile. Overall
is composed by about $350$ concepts, $142$ roles and $238$ attributes, with a
maximum hierarchy depth of $10$. This restriction is essential for its use in
the context of OBDA as guarantees first-order rewritability for the class of
union of conjunctive queries.

#### Mappings

The NPD specification provides a set of $1173$ mapping assertions,
characterized by an average of $1.7$ joins per query. The mappings have been
partially bootstrapped automatically from the database and the ontology, and
partially created by hand, and are specified in the R2RML mapping language.

The mappings have purposely not optimized, to measure the efficiency of the
optimization strategies employed by an OBDA system. This means that the number
of mappings that refer to the same ontology predicate is in general very
large, up to about $30$ in some cases.

#### Query Set

The latest revision of the NPD benchmark devises $30$ between real world and
technical queries of different complexity, expressed in SPARQL, and defined by
domain experts starting from an informal set of questions to the users of the
NPD FactPages. Among the set of queries, some have been specifically generated
to stress the efficiency of a system when reasoning with respect to
existential variables.

Some of the characteristics of the queries are the presence of concepts with a
rich hierarchy and the presence of aggregations. For the purpose of this
experimentation, we are only interested in the subset of these SPARQL queries
corresponding to the class of union of conjunctive queries, as this is the
semantics for SPARQL queries that is adopted by Mastro. The only exception is
that we make is for the use of duplicate elimination from the results. This
requires to changes part of the query set.

In the following we present each individual query, together with its _CQ_
restriction, the changes required, and a brief description of the query.

NPD query q1 ⬇ SELECT DISTINCT ?licenceURI ?interest ?date WHERE { ?licenceURI
a npdv:ProductionLicence . [ ] a npdv:ProductionLicenceLicensee ;
npdv:dateLicenseeValidFrom ?date ; npdv:licenseeInterest ?interest ;
npdv:licenseeForLicence ?licenceURI . FILTER(?date >
"1979-12-31T00:00:00"^^xsd:dateTime) } CQ query q1 ⬇ SELECT DISTINCT
?licenceURI ?interest ?date WHERE { ?licenceURI a npdv:ProductionLicence . [ ]
a npdv:ProductionLicenceLicensee ; npdv:dateLicenseeValidFrom ?date ;
npdv:licenseeInterest ?interest ; npdv:licenseeForLicence ?licenceURI .
FILTER(?date > "1979-12-31T00:00:00"^^xsd:dateTime) } Modifications No
modification is required for query $q1$. Description Query q1 asks for the
licenses, the interest applied to the respective licensees, and from when they
are valid. Listing 3:  NPD Query q1

NPD query q2 ⬇ SELECT ?licenceURI ?company ?date WHERE { ?licenceURI a
npdv:ProductionLicence . [ ] a npdv:ProductionLicenceOperator ;
npdv:dateOperatorValidFrom ?date ; npdv:licenceOperatorCompany [ npdv:name
?company ] ; npdv:operatorForLicence ?licenceURI . FILTER(?date >
"1979-12-31T00:00:00"^^xsd:dateTime) } ORDER BY ?licenceURI CQ query q2 ⬇
SELECT ?licenceURI ?company ?date WHERE { ?licenceURI a npdv:ProductionLicence
. [ ] a npdv:ProductionLicenceOperator ; npdv:dateOperatorValidFrom ?date ;
npdv:licenceOperatorCompany [ npdv:name ?company ] ; npdv:operatorForLicence
?licenceURI . FILTER(?date > "1979-12-31T00:00:00"^^xsd:dateTime) }
Modifications The CQ version of query q2 is obtained by removing the ORDER BY
clause. Description Query $q2$ asks for the operators for licences whose
contracts were started after $1980$. Listing 4:  NPD Query q2

NPD query q3 ⬇ SELECT ?licence ?dateGranted ?dateValidTo WHERE { [ ] a
npdv:ProductionLicence ; npdv:name ?licence ; npdv:dateLicenceGranted
?dateGranted ; npdv:dateLicenceValidTo ?dateValidTo . FILTER(?dateValidTo >
"1979-12-31T00:00:00"^^xsd:dateTime) } ORDER BY ?licence CQ query q3 ⬇ SELECT
?licence ?dateGranted ?dateValidTo WHERE { [ ] a npdv:ProductionLicence ;
npdv:name ?licence ; npdv:dateLicenceGranted ?dateGranted ;
npdv:dateLicenceValidTo ?dateValidTo . FILTER(?dateValidTo >
"1979-12-31T00:00:00"^^xsd:dateTime) } Modifications The CQ version of query
$q3$ is obtained by removing the ORDER BY clause with respect to the ?licence
variable. Description Query $q3$ asks for the licences whose expiration dates
were after $1980$. Listing 5:  NPD Query q3

NPD query q4 ⬇ SELECT ?licence ?company ?licenseeFrom WHERE { [ ]
npdv:licenseeForLicence [ a npdv:ProductionLicence ; npdv:name ?licence ] ;
npdv:licenceLicensee [ npdv:name ?company ] ; npdv:dateLicenseeValidFrom
?licenseeFrom . FILTER(?licenseeFrom > "1979-12-31T00:00:00"^^xsd:dateTime) }
ORDER BY ?licence ASC(?licenseeFrom) CQ query q4 ⬇ SELECT ?licence ?company
?licenseeFrom WHERE { [ ] npdv:licenseeForLicence [ a npdv:ProductionLicence ;
npdv:name ?licence ] ; npdv:licenceLicensee [ npdv:name ?company ] ;
npdv:dateLicenseeValidFrom ?licenseeFrom . FILTER(?licenseeFrom >
"1979-12-31T00:00:00"^^xsd:dateTime) } Modifications The CQ of query $q4$ is
obtained by removing the ORDER BY clause. Listing 6:  NPD Query q4

NPD query q5 ⬇ SELECT ?fr ?OE ?oil ?gas ?NGL ?con WHERE { ?fr a
npdv:FieldReserve ; npdv:remainingCondensate ?con ; npdv:remainingGas ?gas ;
npdv:remainingNGL ?NGL ; npdv:remainingOil ?oil ; npdv:remainingOilEquivalents
?OE . FILTER(?gas < 100) } ORDER BY DESC(?OE) CQ query q5 ⬇ SELECT ?fr ?OE
?oil ?gas ?NGL ?con WHERE { ?fr a npdv:FieldReserve ; npdv:remainingCondensate
?con ; npdv:remainingGas ?gas ; npdv:remainingNGL ?NGL ; npdv:remainingOil
?oil ; npdv:remainingOilEquivalents ?OE . FILTER(?gas < 100) } Modifications
The CQ of query $q5$ is obtained by removing the ORDER BY clause. Listing 7:
NPD Query q5

NPD query q6 ⬇ SELECT DISTINCT ?wellbore (?length AS ?lenghtM) ?company ?year
WHERE { ?wc npdv:coreForWellbore [ rdf:type npdv:Wellbore ; npdv:name
?wellbore ; npdv:wellboreCompletionYear ?year ; npdv:drillingOperatorCompany [
npdv:name ?company ] ] . { ?wc npdv:coresTotalLength ?length } FILTER(?year >=
"2008"^^xsd:integer && ?length > 50) } ORDER BY ?wellbore Description This is
a query that asks for the wellbores, their length, and the companies that
completed the drilling of the wellbore after 2008, and sampled more than 50m
of cores. The use of graph patterns of this form is not supported in Mastro,
and so it has not been considered. Listing 8:  NPD Query q6

NPD query q7 ⬇ SELECT * WHERE { [ ] a npdv:FieldMonthlyProduction ;
npdv:productionYear ?year; npdv:productionMonth ?month;
npdv:producedCondensate ?con ; npdv:producedGas ?gas ; npdv:producedNGL ?NGL ;
npdv:producedOil ?oil ; npdv:producedOilEquivalents ?maxOE . FILTER(?gas <
100) } CQ query q7 ⬇ SELECT * WHERE { [ ] a npdv:FieldMonthlyProduction ;
npdv:productionYear ?year; npdv:productionMonth ?month;
npdv:producedCondensate ?con ; npdv:producedGas ?gas ; npdv:producedNGL ?NGL ;
npdv:producedOil ?oil ; npdv:producedOilEquivalents ?maxOE . FILTER(?gas <
100) } Modifications No modification is required for query $q7$. Listing 9:
NPD Query q7

NPD query q8 ⬇ SELECT * WHERE { [ npdv:productionYear ?year ;
npdv:productionMonth ?m ; npdv:producedGas ?g ; npdv:producedOil ?o ] FILTER
(?year > 1999) FILTER(?m >= 1 && ?m <= 6 ) } CQ query q8 ⬇ SELECT * WHERE { [
npdv:productionYear ?year ; npdv:productionMonth ?m ; npdv:producedGas ?g ;
npdv:producedOil ?o ] FILTER (?year > 1999) FILTER(?m >= 1 && ?m <= 6 ) }
Modifications No modification is required for query $q8$. Listing 10:  NPD
Query q8

NPD query q9 ⬇ SELECT * WHERE { [ ] a npdv:Facility ; npdv:name ?facility ;
npdv:registeredInCountry ?country; npdv:idNPD ?id . FILTER (?id >
"400000"^^xsd:integer) } ORDER BY ?facility CQ query q9 ⬇ SELECT * WHERE { [ ]
a npdv:Facility ; npdv:name ?facility ; npdv:registeredInCountry ?country;
npdv:idNPD ?id . FILTER (?id > "400000"^^xsd:integer) } Modifications The CQ
version of $q9$ is obtained by removing the ORDER BY clause. Listing 11:  NPD
Query q9

NPD query q10 ⬇ SELECT DISTINCT * WHERE { [] a npdv:DiscoveryWellbore ;
npdv:name ?wellbore; npdv:dateUpdated ?date . FILTER (?date >
"2013-01-01T00:00:00.0"^^xsd:dateTime) } ORDER BY ?wellbore CQ query q10 ⬇
SELECT DISTINCT * WHERE { [] a npdv:DiscoveryWellbore ; npdv:name ?wellbore;
npdv:dateUpdated ?date . FILTER (?date >
"2013-01-01T00:00:00.0"^^xsd:dateTime) } Modifications The CQ version of $q10$
is obtained by removing the ORDER BY clause. Listing 12:  NPD Query q10

NPD query q11 ⬇ SELECT DISTINCT ?wellbore (?length AS ?lenghtM) ?company ?year
WHERE { ?wc npdv:coreForWellbore [ rdf:type npdv:Wellbore ; npdv:name
?wellbore ; npdv:wellboreCompletionYear ?year ; npdv:drillingOperatorCompany [
npdv:name ?company ] ] . { ?wc npdv:coresTotalLength ?length ;
npdv:coreIntervalUOM "[m ]"^^xsd:string . } FILTER(?year >= 2008 && ?length >
50) } ORDER BY ?wellbore Description Query $q11$ is a variation of $q6$ where
the wellbore core length has a value expressed in meters. Listing 13:  NPD
Query q11

NPD query q12 ⬇ SELECT DISTINCT ?wellbore (?length AS ?lenghtM) ?company ?year
WHERE { ?wc npdv:coreForWellbore [ rdf:type npdv:Wellbore ; npdv:name
?wellbore ; npdv:wellboreCompletionYear ?year ; npdv:drillingOperatorCompany [
npdv:name ?company ] ] . { ?wc npdv:coresTotalLength ?l ; npdv:coreIntervalUOM
"[m ]"^^xsd:string . BIND(?l AS ?length) } UNION { ?wc npdv:coresTotalLength
?l ; npdv:coreIntervalUOM "[ft ]"^^xsd:string . BIND((?l * 0.3048) AS ?length)
} FILTER(?year >= "2008"^^xsd:integer && ?length > 50) } ORDER BY ?wellbore
Description Query $q12$ is an extension of $q6$, and makes use of the SPARQL
operator BIND and arithmetical operations on the results, which are not
supported on Mastro. Listing 14:  NPD Query q12

NPD query q13 ⬇ SELECT DISTINCT * WHERE { ?x a npdv:SeismicSurvey . OPTIONAL
{?x npdv:lengthCdpTotalKm ?cdpKM .} OPTIONAL {?x npdv:lengthBoatTotalKm
?boatKM .} FILTER (?cdpKM > 3660) } NPD query q14 ⬇ SELECT DISTINCT * WHERE {
?x a npdv:WellboreDrillingMudSample ; npdv:dateMudMeasured ?date . OPTIONAL {
?x npdv:mudType ?type . OPTIONAL { ?x npdv:mudWeight ?w ;
npdv:mudMeasuredDepth ?d . } } FILTER (?date >
"1986-08-25T00:00:00"^^xsd:dateTime) } Description Queries $q_{13}$ and
$q_{14}$ make use of the the SPARQL operators OPTIONAL and BIND which are not
supported in the current version of Mastro. Listing 15:  NPD Query q13-14

NPD query q15 ⬇ SELECT ?licenceURI (AVG(?interest) AS ?vavg) WHERE {
?licenceURI a npdv:ProductionLicence . [ ] a npdv:ProductionLicenceLicensee ;
npdv:dateLicenseeValidFrom ?date ; npdv:licenseeInterest ?interest ;
npdv:licenseeForLicence ?licenceURI . FILTER(?date >= "1979-12-31"^^xsd:date)
} GROUP BY ?licenceURI CQ query q15 ⬇ SELECT ?licenceURI ?interest WHERE {
?licenceURI a npdv:ProductionLicence . [ ] a npdv:ProductionLicenceLicensee ;
npdv:dateLicenseeValidFrom ?date ; npdv:licenseeInterest ?interest ;
npdv:licenseeForLicence ?licenceURI . FILTER(?date >= "1979-12-31"^^xsd:date)
} Modifications The CQ of query $q15$ is obtained by removing the aggregation
over the `?licenceURI` variable. Description Query $q15$ asks for the
licenses, the interest applied to the respective licensees. Listing 16:  NPD
Query q15

NPD query q16 ⬇ SELECT (COUNT(?licence ) AS ?licnumber) WHERE { [ ] a
npdv:ProductionLicence ; npdv:name ?licence ; npdv:dateLicenceGranted
?dateGranted ; FILTER(?dateGranted >= "1999-12-31"^^xsd:date) } CQ query q16 ⬇
SELECT ?licence WHERE { [ ] a npdv:ProductionLicence ; npdv:name ?licence ;
npdv:dateLicenceGranted ?dateGranted ; FILTER(?dateGranted >=
"1999-12-31"^^xsd:date) } Modifications The CQ of query $q16$ is obtained by
removing the COUNT operator. Description This query asks for the name of the
licenses granted starting from year $2000$. Listing 17:  NPD Query q16

NPD query q17 ⬇ SELECT ?field (SUM(?g) AS ?gas) WHERE { [ npdv:productionYear
2008 ; npdv:productionMonth ?m ; npdv:producedGas ?g ; npdv:productionForField
[ rdf:type npdv:Field ; npdv:name ?field ; npdv:currentFieldOperator [
npdv:shortName "STATOIL PETROLEUM AS"^^xsd:string ]]] } GROUP BY ?field ORDER
BY ?field CQ query q17 ⬇ SELECT ?field ?g WHERE { [ npdv:productionYear 2008 ;
npdv:productionMonth ?m ; npdv:producedGas ?g ; npdv:productionForField [
rdf:type npdv:Field ; npdv:name ?field ; npdv:currentFieldOperator [
npdv:shortName "STATOIL PETROLEUM AS"^^xsd:string ]]] } Modifications Query
$q17$ is obtained by removing aggregation and the ordering over the `?field`
variable. Listing 18:  NPD Query q17

NPD query q18 ⬇ SELECT ?field (AVG(?oil) AS ?avgOil) WHERE { [ ] a
npdv:FieldYearlyProduction ; npdv:productionForField [ npdv:name ?field ] ;
npdv:producedOil ?oil ; npdv:productionYear ?year . FILTER (?year < 2013) }
GROUP BY ?field ORDER BY DESC(?avgOil) CQ query q18 ⬇ SELECT ?field ?oil WHERE
{ [ ] a npdv:FieldYearlyProduction ; npdv:productionForField [ npdv:name
?field ] ; npdv:producedOil ?oil ; npdv:productionYear ?year . FILTER (?year <
2013) } Modifications The CQ of query $q18$ is obtained by removing the
aggregation over the `?field` variable and the ordering over the averaged
`?oil` variable. Listing 19:  NPD Query q18

NPD query q19 ⬇ SELECT ?field (SUM(?o) AS ?oil) WHERE { [ npdv:productionYear
1993 ; npdv:productionMonth ?m ; npdv:producedOil ?o ; npdv:productionForField
[ rdf:type npdv:Field ; npdv:name ?field ; npdv:currentFieldOperator [
npdv:shortName "STATOIL PETROLEUM AS"^^xsd:string ]]] FILTER(?m >= 1 && ?m <=
6) } GROUP BY ?field ORDER BY ?field CQ query q19 ⬇ SELECT ?field ?o WHERE { [
npdv:productionYear 1993 ; npdv:productionMonth ?m ; npdv:producedOil ?o ;
npdv:productionForField [ rdf:type npdv:Field ; npdv:name ?field ;
npdv:currentFieldOperator [ npdv:shortName "STATOIL PETROLEUM AS"^^xsd:string
]]] FILTER(?m >= 1 && ?m <= 6) } Modifications The CQ of query $q19$ is
obtained by removing the aggregation and ordering over the `?field` variable.
Listing 20:  NPD Query q19

NPD query q20 ⬇ SELECT ?fr (Max(?g) AS ?max) WHERE { ?fr npdv:productionYear
?year ; npdv:productionMonth ?m ; npdv:producedGas ?g . FILTER (?year > 2000)
} GROUP BY ?fr NPD query q21 ⬇ SELECT ?fr (Min(?g) AS ?min) WHERE { ?fr
npdv:productionYear ?year ; npdv:productionMonth ?m ; npdv:producedGas ?g .
FILTER (?year > 2000) } GROUP BY ?fr CQ query q20 ⬇ SELECT ?fr ?g WHERE { ?fr
npdv:productionYear ?year ; npdv:productionMonth ?m ; npdv:producedGas ?g .
FILTER (?year > 2000) } Modifications Query $q20$ and $q21$ differ only by the
type of aggregation performed. For this reason after removing the aggregation
only one of them has been considered. Listing 21:  NPD Query q20

NPD query q22 ⬇ SELECT DISTINCT ?wc WHERE { ?wc npdv:coreForWellbore [
rdf:type npdv:Wellbore ]. } NPD query q23 ⬇ SELECT DISTINCT ?member ?awc WHERE
{ ?member npdv:member ?awc. ?awc npdv:coreForWellbore [ rdf:type npdv:Wellbore
]. } NPD query q24 ⬇ SELECT DISTINCT ?member WHERE { ?member npdv:member ?awc.
?awc npdv:coreForWellbore [ rdf:type npdv:Wellbore ]. } NPD query q25 ⬇ SELECT
DISTINCT ?licensee WHERE { ?licensee npdv:licenseeForLicence ?licence. }
Description Queries $q22-25$ are already in the fragment of conjuntive
queries, so no modification is required. Listing 22:  NPD Query q22-25

NPD query q26 ⬇ SELECT DISTINCT ?licensee WHERE { ?licensee
npdv:licenseeForLicence [ npdv:licenceOperatorCompany ?company ] } NPD query
q27 ⬇ SELECT DISTINCT ?company WHERE { ?licensee npdv:licenseeForLicence [
npdv:licenceOperatorCompany ?company ]. } Description Queries $q26-27$ are
already in the fragment of conjuntive queries, so no modification is required.
Listing 23:  NPD Query q26-27

NPD query q28 ⬇ SELECT DISTINCT ?wellbore ?wc ?well WHERE { ?wellbore
npdv:wellboreForDiscovery ?discovery; npdv:belongsToWell ?well. ?wc
npdv:coreForWellbore ?wellbore. } NPD query q29 ⬇ SELECT DISTINCT ?wellbore
?wc ?well ?length WHERE { ?wellbore npdv:wellboreForDiscovery ?discovery;
npdv:belongsToWell ?well. ?wc npdv:coreForWellbore ?wellbore;
npdv:coresTotalLength ?length; npdv:coreIntervalUOM "[ft ]"^^xsd:string . #
feets FILTER (?length < 56796) } Description Queries $q28-29$ are already in
the fragment of conjuntive queries, so no modification is required. Listing
24:  NPD Query q28-29

NPD query q30 ⬇ SELECT DISTINCT ?wellbore ?wc ?well ?length WHERE { ?wellbore
npdv:wellboreForDiscovery ?discovery; npdv:belongsToWell ?well. ?wc
npdv:coreForWellbore ?wellbore. { ?wc npdv:coresTotalLength ?lmeters ;
npdv:coreIntervalUOM "[m ]"^^xsd:string . BIND(?lmeters AS ?length) } UNION {
?wc npdv:coresTotalLength ?lfeets ; npdv:coreIntervalUOM "[ft ]"^^xsd:string .
BIND((?lfeets * 0.3048) AS ?length) } FILTER (?length < 22337) } Description
Query $q30$ is an extension of $q12$, and makes use of the SPARQL operator
BIND and arithmetical operations on the results, which are not supported on
Mastro. Listing 25:  NPD Query q30

#### Data generation

The original NPD databases is derived from the data published on the Norvegian
Petroleum Directorate FactPages777http://factpages.npd.no/factpages/.

The data from FactPages has been translated from CSV files into a structured
database. The generated schema consists of $70$ tables with $276$ distinct
columns (about $1000$ columns in total), and $94$ foreign keys.

The schemas of the tables overlap in the sense that several attributes are
replicated in several tables. In fact, there are tables with more than $100$
columns. The total size of the initial database is about $60$ MB.

Since OBDA are expected to work in the context of Big Data, the authors of the
benchmark have provided a tool that enables the initial database instance to
be scaled in order to obtain larger instances. The scaling process,
implemented by the Virtual Instances Generator
(VIG)[13]888https://github.com/ontop/vig, is performed by taking into account
the axioms in the ontology, the structure of the mappings, and the database
constraints in order to preserve a set of similarity measures in the original
database.

Compared to a random generation approach, the algorithm adopted by VIG
preserves important factors, such as the ratio of column-based duplicates,
null values ratios, and the ratio for join result sets. This is a requirement
when the generation has to be performed in order to evaluate the performance
of an OBDA system. Adopting a completely random approach would simply make the
scaled database non-suitable for the evaluation of an OBDA system, since the
number of joining columns in the mappings would be completely random and non-
representative of the original instance.

Starting from this initial database, instances of different size have been
created with the use of the VIG generator, and have been loaded into separate
databases. Table 1 shows the scaling factor and the size for each of the
generated databases that have been used in our experiment. The number of the
database represents its scale with respect to the original instance.

Name | Scale Factor | Size
---|---|---
NPD1 | 1 | 60 MB
NPD10 | 10 | 710 MB
NPD50 | 50 | 2570 MB
NPD100 | 100 | 5300 MB
Table 1: Generated databases

#### Experimental Setup

We ran the NPD Benchmark on both the latest version of Mastro $1.0.2$ and
Ontop $3.0$, on the same physical system using the same set of generated
databases. The underlying DBMS is MySQL version $5.7.21$, running locally on
the testing machine.

The specifications of the platform used for the experiments are the following:

CPU

Intel Xeon E5-2670 running at 2.60GHz

RAM

16 GB DDR3 1600 MHz

OS

Ubuntu 17.10 running in a virtualized environment (4 cores)

The experimentation is performed in the following mode:

  * •

We iterate over the set of queries. At each iteration we pick a random query
from the set and evaluate it over the system. This is done in order to reduce
the effect of the caching in the DBMS.

For each execution we store the results and the time it took to complete. We
consider the combined time needed for evaluating the query and processing the
set of results. Queries are executed sequentially through the use of a testing
platform designed specifically for this task, which accesses directly the
internal APIs of the systems.

  * •

We repeat the process until all queries have been executed a fixed amount of
time, in this case $5$ executions were performed.

  * •

Finally, we take the average of the execution times for each of the queries in
the set.

In order to compare the systems under the same setting we enabled for both
reasoning with respect to existential variables. Our metric of comparison is
the total time taken to complete the execution of the query and to process the
results. A comparison of the execution times for both systems for the database
instances NPD1, NPD10, NPD50, and NPD100 are shown respectively in Figure 3,
4, 5, and 6. Table 2 reports a summary of the average execution time for each
query and database size.

Table 2: Query Answering Times over the generated databases (in seconds) Query | NPD1 | NPD10 | NPD50 | NPD100
---|---|---|---|---
| Mastro | Ontop | Mastro | Ontop | Mastro | Ontop | Mastro | Ontop
$q1$ | $0.364$ | $0.692$ | $3.320$ | $3.627$ | $14.277$ | $36.657$ | $37.107$ | $159.733$
$q2$ | $0.316$ | $0.098$ | $0.201$ | $0.078$ | $1.302$ | $2.944$ | $2.939$ | $6.762$
$q3$ | $0.117$ | $0.064$ | $0.279$ | $0.485$ | $0.900$ | $2.304$ | $1.939$ | $9.450$
$q4$ | $0.620$ | $0.996$ | $2.815$ | $3.779$ | $69.200$ | $34.961$ | $32.037$ | $76.069$
$q5$ | $0.082$ | $0.024$ | $0.115$ | $0.043$ | $0.108$ | $0.090$ | $0.134$ | $0.074$
$q7$ | $0.788$ | $0.258$ | $0.694$ | $0.425$ | $0.464$ | $5.793$ | $0.656$ | $19.252$
$q8$ | $0.141$ | $0.076$ | $0.250$ | $0.197$ | $0.395$ | $0.394$ | $0.612$ | $0.651$
$q9$ | $0.937$ | $0.125$ | $1.176$ | $2.096$ | $7.284$ | $5.788$ | $7.012$ | $18.753$
$q10$ | $0.161$ | $0.250$ | $0.600$ | $2.523$ | $3.401$ | $17.849$ | $6.207$ | $33.731$
$q15$ | $0.288$ | $0.111$ | $1.161$ | $0.586$ | $19.688$ | $5.454$ | $22.683$ | $43.994$
$q16$ | $0.093$ | $0.011$ | $0.093$ | $0.011$ | $0.116$ | $0.023$ | $0.172$ | $0.085$
$q17$ | $2.121$ | $0.004$ | $2.287$ | $0.006$ | $2.167$ | $0.006$ | $2.213$ | $0.004$
$q18$ | $0.071$ | $0.009$ | $0.074$ | $0.004$ | $0.082$ | $0.005$ | $0.074$ | $0.004$
$q19$ | $2.024$ | $0.005$ | $2.043$ | $0.005$ | $2.138$ | $0.004$ | $3.051$ | $0.004$
$q20$ | $0.249$ | $0.113$ | $2.901$ | $1.579$ | $15.146$ | $11.898$ | $44.538$ | $118.329$
$q22$ | $0.457$ | $0.565$ | $3.421$ | $5.408$ | $14.808$ | $28.372$ | $26.425$ | $55.841$
$q23$ | $0.752$ | $0.243$ | $6.065$ | $2.363$ | $42.884$ | $22.180$ | $65.265$ | $46.621$
$q24$ | $0.562$ | $0.114$ | $3.546$ | $0.715$ | $18.224$ | $3.640$ | $39.278$ | $7.772$
$q25$ | $0.745$ | $0.284$ | $6.551$ | $2.985$ | $61.744$ | $19.048$ | $94.802$ | $35.893$
$q26$ | $0.088$ | $0.004$ | $0.086$ | $0.006$ | $0.094$ | $0.003$ | $0.109$ | $0.007$
$q27$ | $0.089$ | $0.003$ | $0.082$ | $0.002$ | $0.107$ | $0.003$ | $0.114$ | $0.004$
$q28$ | $13.496$ | $1.648$ | $49.303$ | $20.641$ | $220.932$ | $133.396$ | $438.149$ | $319.339$
$q29$ | $6.001$ | $0.155$ | $8.211$ | $0.440$ | $27.770$ | $9.396$ | $58.841$ | $18.525$

Figure 3: Query Answering over NPD1 (Execution Time) Figure 4: Query Answering
over NPD10 (Execution Time) Figure 5: Query Answering over NPD50 (Execution
Time) Figure 6: Query Answering over NPD100 (Execution Time)

#### Experimental results

We start our analysis by looking at some of the most interesting results. In
particular, the queries where Ontop performs worse are those requiring
reasoning with respect to the existential variables. Examples of these queries
are $q9$ and $q10$, which cause the system to produce rewritings consisting of
unions of tens of sub-queries.

Queries $q1$,$q2$,$q3$,$q4$,$q7$,$q15$,$q16$,$q25$ instead produce a simple
SPJ rewriting, although the difference here is given by the fact that Ontop
applies strictly the rules specified by the OWL 2 standard, performing the
datatype casts and the generation of the IRIs directly at the SQL level, which
causes a slowdown of the execution. Instead, Mastro adopts a less strict
approach for dealing with datatypes, avoiding to perform casts and IRI-
construction at the SQL level.

As for queries $q24$-$q25$, and $q28$-$q29$, it can be noted that in this case
Mastro performs considerably slower than Ontop. This is due to the high number
of mappings for the ontology predicates involved in these queries, that have
to be unfolded by the system, and by the high number of sub-classes and sub-
properties for the concepts and properties in the query, which cause the
rewriting to grow exponentially in size. In Mastro, this phenomenon is
mitigated with the addition of data constraints during the design of the
views, in such a way that this redundancy is avoided. In this case, since view
are generated automatically from the R2RML mapping, these constraints are not
available to the system, effectively disabling all the optimizations that the
system is capable of performing. Instead, in Ontop this problem is mitigated
during the offline stage, when the mappings and the ontology are compiled to
form the so-called $\mathcal{T}$-mappings, using the database constraints to
optimize the mappings.

Finally, queries $q17$-$q19$, and $q26$-$q27$ are less interesting, as they
produce empty unfoldings (due to mismatching function names in the mappings),
and their execution time does not depend on the database size. For these
queries, it can be noted that moving part of the computation to an offline
stage gives Ontop a big advantage, as the system has to check a small amount
of mappings at query execution time.

##  4 Comparison on a full-fledged OBDA solution

While the use synthetic benchmarks can be a starting point in order to
understand the type of performance that we can expect from a system, and
allows to draw some conclusions about the effectiveness of the adopted
approach, there is also the need to verify that such conclusions hold also
when we move to real world applications.

The main problem with the specification that we presented in Section 3 is that
while the ontology has been effectively designed to model accurately the
conceptual reality of the domain of interest, the data sources did not exist
before and have been created specifically for the purpose of evaluating the
specification. Unfortunately, this means that they do not reflect what is the
real structure that can be found when attempting to experiment the ontology-
based data access approach in real world scenarios.

For this purpose we decided to use one of the specifications that is currently
in development between Sapienza University of Rome and the Automobile Club
d’Italia (ACI).

In this chapter we first give an overview of the specification, by describing
the ontology, the mappings and the data sources. Then, we proceed in showing
the experimentation that we performed, aimed at understanding the usability of
both Mastro and Ontop in such scenario. Finally, we present the results and
draw the conclusions.

#### Ontology

The ontology comprises about $500$ concepts, $200$ roles and $200$ attributes,
and it is divided into $11$ logically interconnected modules. Among these
modules some of the most important are represented by:

  * •

The ontological module describing the concept of _Vehicle_ (Veicolo),
characterized as a central object in the ontology, modeling also its relevant
state (_Stato_), that models the evolution of vehicles’ property over time.

  * •

The ontological module describing the concept of _Subject_ (Soggetto),
modeling the possible roles played by the subjects (physical people or
organizations) with respect to the taxation concerning vehicles.

  * •

The ontological module describing the concept of _Formality_ (Formalità), and
_payment_ (Pagamento).

The ontology has been defined following rigorously the best practice for
ontology design, based on the analysis and definition of the domain of
interest through a series of interviews with the domain experts, so that the
ontology would reflect exactly the reality of the domain of interest, and not
the structure of the data sources. This characteristics of the ontology makes
so that the specification of the mappings is extremely complex, due to the
large _semantic gap_ between the reality of the domain of interest and the
semantics of the data sources, which are structured in such a way that enables
the applications that make use of them meet the efficiency requirements that
they need.

Moreover, the modeling of the ontology also describes the evolution in time of
its elements. For example, during its lifespan a vehicle can change owner,
license plate, and so on. This is captured by the notion of state of a vehicle
(Stato). This is a recurring aspect in the queries as we will show in the
following sections.

#### Data source

For this experiment the database is managed by an instance of Oracle 12c. The
database is accessed remotely through the use of a VPN. The relevant portion
of the data for our experiment is distributed across $6$ schemas. These
schemas are composed of hundreds of tables, but for what regards the portion
of the domain that is of interest in this experiment we concentrate on about
$90$ relational tables, ranging from information regarding the domain of PRA
(Pubblico Registro Automobilistico), to those regarding the taxation
concerning the vehicles. Some of these tables count from $200$ million tuples
up to above $1$ billion tuples in some cases, with a number of attributes
ranging from $30$ to $100$. The overall size of the portion of interest of the
data source is several gigabytes of data (an accurate estimate was not
possible).

#### Mappings

The specification comprises $976$ ontology mappings, composed from a set of
about $110$ views over the data source. About $300$ of these ontology mappings
are built from single view atoms, while the remaining are specified as
conjunctive queries over the view predicates. For the purpose of our
experimentation the mappings have been previously translated in R2RML through
the use of the approach described in Section 2, and then imported back into
both systems. During the translation process the constraints over the view
predicates are discarded, since they cannot be expressed in R2RML.

#### Query Set

In this section we describe the set of queries that have been defined to
evaluate the usability of both systems. Query $q1$ to $q5$ are basic
navigational queries, that span some of the relevant part of the ontology. The
remaining queries $q6$ to $q10$ are derived the real queries used in the
original experimentation of the project. These queries are built from a set of
_competency questions_ , defined by interviewing the domain experts over the
practical questions that have to be posed to the system, and are used to
validate the quality and the coherency of the specification.

Before showing the set of queries that have been used for our experimentation
we have to point out that in order to comply with the time restrictions we
were granted, these queries have been restricted to extracting information for
single vehicles. In some cases this was already enough to fill the time slot
that was allowed. In their real application, queries are used to build reports
for several hundred of thousands of vehicles, that require several hours to
complete.

ACI query q1 ⬇ PREFIX aci: <http://www.aci.it/ontology#> PREFIX xsd:
<http://www.w3.org/2001/XMLSchema#> SELECT ?veicolo ?stato ?proprieta WHERE {
?veicolo aci:ID_veicolo "<id>"^^xsd:string. ?veicolo aci:ha_stato_di_veicolo
?stato. ?proprieta aci:proprieta_di ?veicolo. } Description Query $q_{1}$ asks
for vehicles that have a state and their ownership. Listing 26:  ACI query q1
ACI query q2 ⬇ PREFIX aci: <http://www.aci.it/ontology#> PREFIX xsd:
<http://www.w3.org/2001/XMLSchema#> SELECT ?veicolo ?stato ?proprieta
?formalita WHERE { ?veicolo aci:ID_veicolo "<id>"^^xsd:string. ?veicolo
aci:ha_stato_di_veicolo ?stato. ?proprieta aci:proprieta_di ?veicolo.
?formalita aci:formalita_presentata_per_veicolo ?veicolo. } Description Query
$q2$ asks for vehicles that have a state, their ownership, and the
formalities. Listing 27:  ACI query q2 ACI query q3 ⬇ PREFIX aci:
<http://www.aci.it/ontology#> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
SELECT ?veicolo ?stato ?proprieta ?accettazione ?anno ?descrizione WHERE {
?veicolo aci:ID_veicolo "<id>"^^xsd:string. ?veicolo aci:ha_stato_di_veicolo
?stato. ?proprieta aci:proprieta_di ?veicolo. ?formalita
aci:formalita_presentata_per_veicolo ?veicolo. ?formalita
aci:data_accettazione_formalita ?accettazione. ?formalita aci:anno_formalita
?anno. ?formalita aci:descrizione_tipo ?descrizione. } Description Query $q2$
asks for vehicles that have a state, their ownership, and the formalities,
with their acceptance dates, year of acceptance, and a description of the type
of formality. Listing 28:  ACI query q3 ACI query q4 ⬇ PREFIX aci:
<http://www.aci.it/ontology#> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
SELECT ?veicolo ?stato ?proprieta ?accettazione ?descrizione ?anno
?certificato WHERE { ?veicolo aci:ID_veicolo "<id>"^^xsd:string. ?veicolo
aci:ha_stato_di_veicolo ?stato. ?proprieta aci:proprieta_di ?veicolo.
?formalita aci:formalita_presentata_per_veicolo ?veicolo. ?formalita
aci:data_accettazione_formalita ?accettazione. ?formalita aci:anno_formalita
?anno. ?formalita aci:descrizione_tipo ?descrizione. ?formalita
aci:emette_certificato ?certificato. } Description Query $q4$ extends $q3$
with the certificate released for the formalities. Listing 29:  ACI query q4
ACI query q5 ⬇ PREFIX aci: <http://www.aci.it/ontology#> PREFIX xsd:
<http://www.w3.org/2001/XMLSchema#> SELECT ?veicolo ?stato ?proprieta
?accettazione ?descrizione ?anno ?certificato ?codiceCDP WHERE { ?veicolo
aci:ID_veicolo "<id>"^^xsd:string. ?veicolo aci:ha_stato_di_veicolo ?stato.
?proprieta aci:proprieta_di ?veicolo. ?formalita
aci:formalita_presentata_per_veicolo ?veicolo. ?formalita
aci:data_accettazione_formalita ?accettazione. ?formalita aci:anno_formalita
?anno. ?formalita aci:descrizione_tipo ?descrizione. ?formalita
aci:emette_certificato ?certificato. ?certificato aci:codice_cdp ?codiceCDP. }
Description Query $q5$ extends $q4$ with the code of the certificate. Listing
30:  ACI query q5 ACI query q6 ⬇ PREFIX aci: <http://www.aci.it/ontology#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> SELECT ?veicolo ?stato WHERE {
?veicolo aci:ID_veicolo "<id>"^^xsd:string. ?veicolo aci:ha_stato_di_veicolo
?stato. ?stato a aci:Stato_rappresentato_valido. } Description Query $q6$ asks
for vehicles that have a state which is valid. Listing 31:  ACI query q6 ACI
query q7 ⬇ PREFIX aci: <http://www.aci.it/ontology#> PREFIX xsd:
<http://www.w3.org/2001/XMLSchema#> SELECT ?veicolo ?stato ?numeroTarga
?serieTarga WHERE { ?veicolo aci:ID_veicolo "<id>"^^xsd:string. ?veicolo
aci:ha_stato_di_veicolo ?stato. ?stato a aci:Stato_rappresentato_valido.
?stato aci:ha_targa ?targa. ?targa aci:numero_targa ?numeroTarga. ?targa
aci:serie_targa ?serieTarga. } Description Query $q7$ extends $q6$ with
information about the license plate of the vehicles. Listing 32:  ACI query q7
ACI query q8 ⬇ PREFIX aci: <http://www.aci.it/ontology#> PREFIX xsd:
<http://www.w3.org/2001/XMLSchema#> SELECT ?veicolo ?stato ?numeroTarga
?serieTarga ?formalita ?codiceFormalita WHERE { ?veicolo aci:ID_veicolo
"<id>"^^xsd:string. ?veicolo aci:ha_stato_di_veicolo ?stato. ?stato a
aci:Stato_rappresentato_valido. ?stato aci:ha_targa ?targa. ?targa
aci:numero_targa ?numeroTarga. ?targa aci:serie_targa ?serieTarga. ?evento
aci:determina_stato ?stato. ?formalita aci:formalita_genera_evento ?evento.
?formalita aci:codice_tipo ?codiceFormalita. } Description Query $q8$ extends
$q7$ with information about the events that determinate the state. Listing 33:
ACI query q8 ACI query q9 ⬇ PREFIX aci: <http://www.aci.it/ontology#> PREFIX
xsd: <http://www.w3.org/2001/XMLSchema#> SELECT ?veicolo ?stato ?numeroTarga
?serieTarga ?inizioStato ?fineStato ?formalita ?codiceFormalita WHERE {
?veicolo aci:ID_veicolo "<id>"^^xsd:string. ?veicolo aci:ha_stato_di_veicolo
?stato. ?stato a aci:Stato_rappresentato_valido. ?stato aci:ha_targa ?targa.
?targa aci:numero_targa ?numeroTarga. ?targa aci:serie_targa ?serieTarga.
?evento aci:determina_stato ?stato. ?formalita aci:formalita_genera_evento
?evento. ?formalita aci:codice_tipo ?codiceFormalita. ?stato
aci:inizio_stato_del_mondo ?inizioStato. ?stato aci:fine_stato_del_mondo
?fineStato. } Description Query $q9$ extends $q8$ with more informations about
the state. Listing 34:  ACI query q9 ACI query q10 ⬇ PREFIX aci:
<http://www.aci.it/ontology#> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
SELECT ?veicolo ?stato ?numeroTarga ?serieTarga ?formalita ?codiceFormalita
?KW ?cilindrata ?classe WHERE { ?veicolo aci:ID_veicolo "<id>"^^xsd:string.
?veicolo aci:ha_stato_di_veicolo ?stato. ?stato a
aci:Stato_rappresentato_valido. ?stato aci:ha_targa ?targa. ?targa
aci:numero_targa ?numeroTarga. ?targa aci:serie_targa ?serieTarga. ?evento
aci:determina_stato ?stato. ?formalita aci:formalita_genera_evento ?evento.
?formalita aci:codice_tipo ?codiceFormalita. ?stato aci:kw ?KW. ?stato
aci:cilindrata ?cilindrata. ?stato aci:classe_veicolo ?classe. } Description
Query $q10$ extends $q9$ with more informations about the state. This
corresponds to one of the competency questions that are asked to the system,
about technical data of the vehicles. Listing 35:  ACI query q10

#### Experimental Setup

For the purpose of the evaluation we ran the queries over both the latest
version of Mastro $1.0.2$ and Ontop version $3.0$, using the same
specification composed by the ontology and the mappings exported in R2RML. The
database is managed by an instance of Oracle 12c, that is accessed remotely
over a VPN connection.

The specifications of the platform used for the experiments are the following:

CPU

Intel Xeon E5-2670 running at 2.60GHz

RAM

16 GB DDR3 1600 MHz

OS

Ubuntu 17.10 running in a virtualized environment (4 cores)

The experimentation is performed in the following mode:

  * •

We iterate over the set of queries. At each iteration we pick a random query
from the set and evaluate it over the system. This is done in order to
minimize the caching in the DBMS.

For each execution we store the results and the time it took to complete. We
consider the combined time needed for evaluating the query and processing the
set of results. Queries are executed sequentially through the use of a testing
platform designed specifically for this task, which accesses directly the
internal APIs of the systems.

  * •

We repeat the process until all queries have been executed a fixed amount of
time, in this case $5$ executions were performed.

  * •

Finally, we take the average of the execution times for each of the queries in
the set.

In order to compare the systems under the same setting we enabled for both
reasoning with respect to existential variables. Our metric of comparison is
the total time taken to complete the execution of the query and to the process
the results. Due to restrictions in the amount of time we were allowed for
experiments, the queries were executed with a timeout of $3$ hours. As a
reference, we report also the time needed by the Mastro system using the
original mapping specification to evaluate the same set of queries. Figure 7
shows the overall execution times for both systems. Table 3 reports the
execution times.

Table 3: Query Answering Times on the ACI Specification (in seconds) Query | Mastro | Mastro R2RML | Ontop
---|---|---|---
$q1$ | $3.808$ | $4.866$ | $15.356$
$q2$ | $5.081$ | $8.362$ | $20.874$
$q3$ | $9.958$ | $19.139$ | $32.607$
$q4$ | $55.310$ | $87.586$ | $153.308$
$q5$ | $56.452$ | $85.284$ | $398.358$
$q6$ | $0.639$ | $1.011$ | – (> 3h)
$q7$ | $2.692$ | $19.113$ | – (> 3h)
$q8$ | $6.353$ | $85.915$ | – (> 3h)
$q9$ | $16.934$ | $5894.621$ | – (> 3h)
$q10$ | $16.104$ | $9123.935$ | – (> 3h)
Figure 7: Query Answering over ACI (Execution Time)

#### Experimental results

There are several considerations that can be made by looking at the results of
our experimentation, but the most important one is that in none of the queries
based on the real competency questions that this solution is designed to
answer, Ontop was able to complete in the time allowed, even though both
systems are using the same specification. We identified three main reasons for
this:

  * •

The first reason is that, in the case of Ontop, the entire rewriting is
executed as a single, complex SQL query, and the database management system is
not able to compute an efficient query plan for such large queries.

  * •

The second reason is that when the optimizations performed to reduce the size
of the $\mathcal{T}$-mappings fail, because there may be missing database
constraints, or simply because the queries in the mappings contain complex
conditions that don’t allow to apply such optimizations, the system produces
rewriting containing complex sub-queries, composed of unions of several SPJ
queries and these types of queries are not evaluated efficiently.

  * •

The third reason is that, even though representing a very elegant solution,
having the objects to be constructed directly at the level of the data
sources, through the use inherently poorly performing operations such as
string concatenation and type casts, is not feasible in real industrial
applications.

This is due to the fact that the assumption at the base of the Ontop
optimizations, that the axioms in the ontology duplicates the constraints on
the database does not hold in this particular case. In this case the structure
of the data sources reflects the application needs and not the conceptual
reality of the domain of discourse. When this condition arises, the system is
not able to optimize the queries in the mappings and causes it to produce
rewritings that are too difficult to be dealt with efficiently by the DBMS,
where the intermediate views are composed by complex unions of _select-
project-join_ queries.

On the other hand, the approach adopted in Mastro, of splitting the queries in
several, simpler conjunctive queries, even if it can result in an exponential
blowup in the size of the rewriting still enables the system to complete the
task in the time allowed, even if in this case there are no optimizations
performed, as the constraints over the views are not expressible in R2RML,
which cause the time taken by the system to increase up to almost three orders
of magnitude for the largest query.

##  5 Conclusions and Future Works

In this article we have worked on OBDA systems, and in particular, on the
Mastro OBDA system, developed at Sapienza. For such system we have studied the
support of the standard R2RML for realizing OBDA mappings. We have implemented
software components to support such kind of mappings and our implementation is
now part of the current release of Mastro. Interestingly, we have seen that we
can translated any R2RML mapping into Mastro mappings, while the contrary is
true only if it is acceptable to lose some efficiency.

Using R2RML mappings we have been able to compare various OBDA systems against
fully standard specifications, where the ontology is specified in OWL 2, the
queries are specified in SPARQL, and the mappings (to standard SQL DBs) are
expressed in R2RML.

In fact, we have seen that only two systems have support fully the reasoning
techniques needed for OBDA solutions: Mastro and Ontop. Hence, we have
concentrated on such two systems.

Our comparison has focused on two benchmarks. The first one is NPD, a
benchmark defined by the Ontop team. Specifically we have modified NPD queries
focusing on the features that require reasoning support. The most important
result in this case is that it shows how the idea of moving part of the
optimization complexity during an offline stage can indeed provide an
advantage, as it reduces the complexity of applying optimization algorithms
entirely at query execution.

The second benchmark has been based on a full-fledged OBDA solution, developed
within a contract between the department of DIAG at Sapienza, University of
Rome, and ACI Informatica. In particular, we have used the real sources from
ACI as well as the OWL 2 ontology developed by the department for them. With
respect to the mappings, we have translate (losing some efficiency) the real
Mastro mappings into R2RML, so as to allow the comparison with Ontop. Finally,
as queries we used simplified versions of the actual queries used in the
application. The results in this case are quite different. Most often, Ontop
could not handle the queries, while Mastro could still complete the task in
the allowed time slot. Instead, in the case of Mastro, the approach of
separating the rewritings into single SPJ queries evaluated individually,
still enables the system to complete the task within the allowed time slot,
even without applying any optimization, as these are not expressible in R2RML.

This comparison lead to some interesting results, that can drive research for
future works. In this section we highlight some of them:

Moving computation to an offline stage:

We have seen how moving part of the computation to an offline stage can
provide large benefits in some cases, not only because it could allow the
system to produce smaller rewritings by applying more extensive optimization
techniques that would not be possible to execute during query evaluation due
to their expensive cost in terms of complexity, but also because this enables
these optimizations to be computed only once, without having to recompute them
for every execution, effectively reducing the time spent during query
evaluation. This has been already considered in Mastro, and a new optimization
based on this idea is already under investigation.

Improving R2RML support with database constraints:

Database constraints can alone represent a useful source of information that
can be exploited for optimizations, as can be seen in the case of the NPD
specification. A possible extension to the technique presented in this article
for translating R2RML mappings into view and ontology mappings can take
advantage of these constraints to automatically generate data constraints
between view mappings.

Combining both approaches:

In the case of Ontop, the idea compiling the ontology into the mappings,
combined with the use of database constraints to simplify the resulting
mappings, can be indeed a very effective technique when the database
constraints reflect the knowledge expressed by the axioms of the ontology, as
they can be combined with techniques that avoid the exponential blowup in the
size of the produced CQ rewritings that can be generated by algorithms such as
PerfectRef, and can be computed only once and used for all the unfoldings.
Although, when such optimization fails to simplify the generated mapping, a
better approach would be to treat each of the unions as a different mapping,
and combine it into several, simpler SQL queries that can be treated
separately, even if it can result in an exponential blowup in the number of
rewritings, as it is done in Mastro. This would at least enable the queries in
such case to be efficiently evaluated by the database management system.

## References

  * [1] A. Acciarri, D. Calvanese, G. De Giacomo, D. Lembo, M. Lenzerini, M. Palmieri, and R. Rosati.  Quonto: Querying ontologies.  In Proceedings, The Twentieth National Conference on Artificial Intelligence and the Seventeenth Innovative Applications of Artificial Intelligence Conference, July 9-13, 2005, Pittsburgh, Pennsylvania, USA, pages 1670–1671, 2005.
  * [2] F. Baader, D. Calvanese, D. L. McGuinness, D. Nardi, and P. F. Patel-Schneider, editors.  The Description Logic Handbook: Theory, Implementation, and Applications.  Cambridge University Press, 2003.
  * [3] C. Bizer and A. Schultz.  The berlin SPARQL benchmark.  Int. J. Semantic Web Inf. Syst., 5(2):1–24, 2009.
  * [4] D. Calvanese, B. Cogrel, E. G. Kalayci, S. Komla-Ebri, R. Kontchakov, D. Lanti, M. Rezk, M. Rodriguez-Muro, and G. Xiao.  OBDA with the Ontop Framework.  In 23rd Italian Symposium on Advanced Database Systems, SEBD 2015, Gaeta, Italy, June 14-17, 2015., pages 296–303, 2015.
  * [5] D. Calvanese, G. De Giacomo, D. Lembo, M. Lenzerini, A. Poggi, M. Rodriguez-Muro, and R. Rosati.  Ontologies and Databases: The DL-Lite Approach.  In Reasoning Web. Semantic Technologies for Information Systems, 5th International Summer School 2009, Brixen-Bressanone, Italy, August 30 - September 4, 2009, Tutorial Lectures, pages 255–356, 2009.
  * [6] D. Calvanese, G. De Giacomo, D. Lembo, M. Lenzerini, and R. Rosati.  DL-Lite: Tractable Description Logics for Ontologies.  In Proceedings, The Twentieth National Conference on Artificial Intelligence and the Seventeenth Innovative Applications of Artificial Intelligence Conference, July 9-13, 2005, Pittsburgh, Pennsylvania, USA, pages 602–607, 2005.
  * [7] D. Calvanese, G. De Giacomo, D. Lembo, M. Lenzerini, and R. Rosati.  Tractable Reasoning and Efficient Query Answering in Description Logics: The _DL-Lite_ Family.  J. Autom. Reasoning, 39(3):385–429, 2007.
  * [8] D. Calvanese, G. De Giacomo, D. Lembo, M. Lenzerini, and R. Rosati.  Data complexity of query answering in description logics.  Artif. Intell., 195:335–360, 2013.
  * [9] S. Das, S. Sundara, and R. Cyganiak.  R2RML: RDB to RDF Mapping Language, 2012.
  * [10] Y. Guo, Z. Pan, and J. Heflin.  LUBM: A benchmark for OWL knowledge base systems.  J. Web Sem., 3(2-3):158–182, 2005.
  * [11] J. Heflin and J. A. Hendler.  A Portrait of the Semantic Web in Action.  IEEE Intelligent Systems, 16(2):54–59, 2001.
  * [12] D. Lanti, M. Rezk, G. Xiao, and D. Calvanese.  The NPD Benchmark: Reality Check for OBDA Systems.  In Proceedings of the 18th International Conference on Extending Database Technology, EDBT 2015, Brussels, Belgium, March 23-27, 2015., pages 617–628, 2015.
  * [13] D. Lanti, G. Xiao, and D. Calvanese.  Data scaling in OBDA benchmarks: The VIG approach.  CoRR, abs/1607.06343, 2016.
  * [14] M. Lenzerini.  Data Integration: A Theoretical Perspective.  In Proceedings of the Twenty-first ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, June 3-5, Madison, Wisconsin, USA, pages 233–246, 2002.
  * [15] F. D. Pinto, D. Lembo, M. Lenzerini, R. Mancini, A. Poggi, R. Rosati, M. Ruzzi, and D. F. Savo.  Optimizing query rewriting in ontology-based data access.  In Joint 2013 EDBT/ICDT Conferences, EDBT ’13 Proceedings, Genoa, Italy, March 18-22, 2013, pages 561–572, 2013.
  * [16] A. Poggi, D. Lembo, D. Calvanese, G. De Giacomo, M. Lenzerini, and R. Rosati.  Linking Data to Ontologies.  J. Data Semantics, 10:133–173, 2008.
  * [17] M. Rodriguez-Muro, R. Kontchakov, and M. Zakharyaschev.  Ontology-Based Data Access: Ontop of Databases.  In The Semantic Web - ISWC 2013 - 12th International Semantic Web Conference, Sydney, NSW, Australia, October 21-25, 2013, Proceedings, Part I, pages 558–573, 2013.
  * [18] M. Rodriguez-Muro and M. Rezk.  Efficient SPARQL-to-SQL with R2RML mappings.  J. Web Sem., 33:141–169, 2015.
  * [19] N. Shadbolt, T. Berners-Lee, and W. Hall.  The semantic web revisited.  IEEE Intelligent Systems, 21(3):96–101, 2006.
  * [20] M. G. Skjæveland, E. H. Lian, and I. Horrocks.  Publishing the norwegian petroleum directorate’s factpages as semantic web data.  In The Semantic Web - ISWC 2013 - 12th International Semantic Web Conference, Sydney, NSW, Australia, October 21-25, 2013, Proceedings, Part II, pages 162–177, 2013.
  * [21] W3C.  OWL 2 Web Ontology Language Profiles (Second Edition), 11 2012.  http://www.w3.org/TR/owl2-profiles/.