# QoS-based Trust Evaluation for Data Services as a Black Box
††thanks: This work has been done in the context of the project SUMMIT
(http://summit.imag.fr) funded by the Auvergne Rhône Alpes region.

Senda Romdhani  Univ. of Lyon, CNRS
Univ. of Lyon 3, LIRIS
Lyon, France
<EMAIL_ADDRESS>Genoveva Vargas-Solar  French Council of
Scientific Research (CNRS)
LIRIS
Lyon, France
<EMAIL_ADDRESS>Nadia Bennani  Univ. of Lyon, CNRS
INSA-Lyon, LIRIS
Villeurbanne, France
<EMAIL_ADDRESS>Chirine Ghedira-Guegan  Univ. of Lyon, CNRS
iaelyon - Univ. of Lyon 3, LIRIS
Lyon, France
<EMAIL_ADDRESS>
###### Abstract

This paper proposes a QoS-based trust evaluation model for black box data
services. Under the black-box model, data services neither export (meta)-data
about conditions in which they are deployed and collect and process data nor
the quality of data they deliver. Therefore, the black-box model creates blind
spots about the extent to which data providers can be trusted to be used to
build target applications. The trust evaluation model for black box data
services introduced in this paper originally combines QoS indicators, like
service performance and data quality, to determine services trustworthiness.
The paper also introduces DETECT: a Data sErvice as a black box Trust
Evaluation arChitecTure, that validates our model. The trust model and its
associated monitoring strategies have been assessed in experiments with
representative case studies. The results demonstrate the feasibility and
effectiveness of our solution.

###### Index Terms:

Trust, Data Service, QoS, Data Quality, Performance

##  I Introduction

The explosion of data collected, managed, and provided using online services
leads to the logic of ”Everything as a Service” (XaaS: X as a Service).
Currently, the use of data as a service for accessing large volumes of data
from different providers concerns mainly data consumers who see services as
practical and easy-to-use components for accessing data.

In this context, searching data implies looking up for data services that can
potentially provide them according to specific conditions and quality
requirements (freshness, availability etc.). However, reliance on services for
data access requires them to be trustworthy and of good quality. This need is
even more crucial when we are in the case of data integration[3] as data is
being provided and composed using multiple data services presenting different
quality of services and distinct data quality. By trust, we mean the belief
that a service meets the terms and quality of service as promised by the
service provider before use.

In the case of data as a service (i.e. data services), this trust is related
not only to the technical behaviour of the service in terms of the way it
deals with requests including its QoS (e.g., availability, response time,
etc.) but also in terms of the conditions in which it provides data
(freshness, update frequency, delivery delay).

Service level agreements (SLAs) contain clauses about the promised QoS for a
given service along with penalties that detail acts to be made in case the
service provider violates these QoS clauses. Service providers often deploy
solution to monitor the functioning of their services (i.e. service
performance) by computing QoS measures[21]. The monitoring is performed to
make internal technical decisions related to resource allocation and so on.
However, for privacy and security reasons, service providers rarely share
their service monitoring measures. Thus, they do not provide guarantees to
service users that service performance adhere to the promised QoS in SLA.
Furthermore, SLA does not yet contain clauses about the promised data quality
factor and thus, users do not know what to expect regarding this factor. In
this sense, data services adopt a black-box model to export their API
(Application Programming Interface) with methods and Input/Output parameters
specifying their data types.

Under the black-box model, services do not provide details on their backend,
including how data are collected, updated and to which extent data are fresh
(i.e. up-to-date). Still, clients are looking for alternatives to evaluate the
trust level of both services and data to use it to choose services for
building information systems and retrieving data.

To this end, we deem necessary providing a trust measure for black box data
services to applications using them. This trust measure is computed using
service performance and data quality factors. Therefore, the challenge is to
propose (1) a trust model that captures and combines service performance and
data quality aspects in order to meet tightly applications needs and (2) a
mechanism to collect information required to compute services trust level, in
case there is no or little access to performance and data quality meta-data.

In order to illustrate the aspects to consider behind this challenge, let us
consider an e-health scenario 111This scenario was proposed in the context of
the project xx to be precised later that highlights the importance of
considering trust when looking up for data services that can provide the right
data for a target application.

Fever in chemotherapy-induced neutropenia (FN) is the most frequent,
potentially lethal complication of chemotherapy in pediatric and adult
patients with cancer [1, 2].

Fever is particularly concerning if it occurs when the white blood count level
is known to be low. During this time, the body’s normal defences against
infections are expected to be down. To this end, fever is considered as a
medical emergency since it indicates a possible infection during chemotherapy.
Therefore, cancer patients’ body temperature should be monitored continuously.

Alice is a 50-year-old woman with breast cancer treated with chemotherapy. To
avoid infection complications when she is at home, she uses medical devices
provided by the hospital’s emergency service to keep track of her temperature
and other physiological measures.

Devices are programmed to capture data according to some unknown rate.
According to a specific update frequency (i.e. insertion frequency), measures
are sent to the hospital that maintains a database with Alice’s sensed data.
To complete this sensing procedure, Alice uses a connected thermometer to
measure her temperature frequently. The device sends the information directly
to an e-health service installed in her smartphone. She also records her
temperature using a regular thermometer every x days or weeks. Alice keeps
track of her manually measured temperature in a specific service installed in
her smartphone too. The manual temperature measurements frequency is variable
as it is determined by Alice’s mood, memory to measure the temperature,
whether she is alone and whether someone else comes by and encourages her to
do a check-up.

Alice gives access to all her collected data through the services that are
deployed on her smartphone. Doctors should be able to access her data at any
moment. This data must be reliable to be used effectively to make accurate
decisions, and it must be delivered promptly in case of an emergency. Deciding
which service to use to retrieve Alice’s physiologic data in a given situation
is challenging as services have different trust levels. Indeed, each service
ensures different QoS and data quality guarantees since data are produced and
updated under different conditions (production rates, update frequencies that
affect data freshness).

In case of an emergency, choosing the right service to provide the best
quality data can be critical for doctors. Therefore, in ideal conditions,
services must be tagged with a trust measure (updated regularly) that could be
used as data services’ selection criterion.

Our work addresses two research problems (RPs) for measuring data services
trust level to support data lookup and selection where trust is critical.
(RP1) What is the appropriate model for describing data services QoS-based
trust? How to define both service performance metrics and data quality metrics
to be used in the trust evaluation process?
(RP2) How to collect the necessary information for this trust evaluation
model?

The main contribution of our work is a data service trust model combining
performance (availability, task success ratio and response time) and data
quality metrics (data and services’ database timeliness).

Our proposal for estimating data services trust has been implemented by DETECT
a Data sErvice as a black box Trust Evaluation arChitecTure and validated
using an e-health scenario. The scenario consists of data services giving
access to HAPI-FHIR222HAPI FHIR is a complete implementation of the HL7 FHIR
(Fast Healthcare Interoperability Resources) standard for healthcare
interoperability and data exchange developed in Java, published by
HL7https://hapifhir.io servers on top of which trust metrics are observed and
measured. DETECT’s general principle is to continuously collect and monitor
technical and data quality meta-data so that data services are tagged with an
up-to-date trust measure. The experimental results show how the trust of
services with different behaviour can be computed and used to rank them given
services lookup requests.

The remainder of this paper is organized as follows. Section II enumerates
related work addressing data and service trust solutions. Section III
describes our proposed QoS-based trust model for data services while defining
the necessary performance and data quality metrics. Section IV presents the
description of the used system DETECT for implementing our proposed trust
evaluation model. The experiments to prove our solution’s feasibility is
presented in section V. The last section concludes the paper and discusses
future work.

##  II Related Work

Service trust evaluation is the subject of many proposals in the literature
that treat trust estimation issues in different service-based environments,
including SOA, web services, cloud services, etc. Reading through the related
work, we summarize the existing proposals focusing on trust: at (i) the
service level, (ii) and the data level.

###  II-A Trustworthy Services

Existing service trust solutions differ in the type of the adopted trust
assessment model (fuzzy models [7, 8, 9], probabilistic models [10], machine
learning models [11, 12, 13], multi-criteria decision-making models (MCDM)
[14, 15] and classical mathematical models such as weighted-addition [16]).
The service trust evaluation can be subjective [17, 18, 40], objective[16, 19]
or hybrid [20, 30] (subjective + objective). Subjective evaluation relies on
user preferences and feedback about the service usage for evaluating trust.
Users’ feedback may be delivered as a quantitative opinion or as a qualitative
opinion in textual form. Objective evaluation, on the other hand, involves
measurable information such as the service capacities, service capabilities,
and service performance [5]. Note that users’ subjective feedback may be
influenced by several factors and may exhibit significant variations in the
evaluation of the same service from one user to another. Besides, there is no
guarantee of users’ honesty.

As for performance, it can be measured using several metrics: service
availability, service response time, service reliability, and service
security. Thus, objective service trust evaluation mainly uses service
performance [5] as a reference measure which is also considered a significant
criterion for selecting services. Accordingly, services performance is
computed through SLA monitoring, where information is collected from previous
experiences and is used to measure the degree of SLA fulfillement [41]. If the
SLA is satisfied, then the service is considered trustworthy.

###  II-B Trustworthy Data

Data similarity and data provenance are the two most commonly used
characteristics for assessing data trust. Similarity-based solutions consider
that the more similar the data values from the same event are, the more likely
the data is correct and therefore trustworthy. Provenance-based solutions
considers that data with good provenance are more likely to be trustworthy.

In addition, several studies focus on the definition of metrics for data
quality evaluation[28, 31, 32, 33]. Authors assume that data providers already
export this information as evidence of honesty or that it is accessible.

Authors in [23] evaluate data quality and data trustworthiness using the web
provenance information. Their approach can collect provenance information,
evaluate it, and use it for data quality evaluation.

Works proposed in [24, 25] evaluate location-based data trustworthiness using
data provenance. Their approach employs GPS data to trace people’s location to
generate meta-data about provenance available and accessible. Thus, they apply
data similarity from different provenances to verify their trustworthiness.

In [26, 27], authors propose a systematic method for assessing the
trustworthiness of data items in sensor networks. Accordingly, a trust score
is computed using data provenance and their values based on their similarity
”the more similar values for the same event, the higher the trust scores”. In
this approach, the trust score of the data affects the trust score of the
network nodes that created and manipulated the data and vice-versa.

The work in [37] introduces a novel data provenance trusted model in cloud
computing. Authors believe that to establish trust with users, cloud service
providers should consider providing trust information such as provenance
record. Based on this hypothesis, meta-data about provenance is provided by a
cloud provider.

The work in [38] proposes trustworthy and secure management of provenance in
cloud computing. It captures the complete provenance of any entities in the
cloud without trusting cloud providers. In this proposal, the authors do not
take into consideration the fact that provenance is meta-data hidden behind
the service’s API.

In [33, 36], authors propose a framework for data freshness evaluation in a
data integration system (DIS). This framework assesses data freshness value
changes in DIS for data moving from data sources to users. Besides, it
evaluates the evolution of data freshness using graphs representing the
integration workflows in a controlled environment. Thus, the proposed
framework does not require meta-data about data provenance.

###  II-C Trustworthy Data Services: Discussion

Existing work proposes models to define data quality metrics, including data
freshness and performance metrics, service response time, task success ratio
and availability.

Certain solutions propose services trust evaluation models taking into
consideration the performance factor. Those models are often not proposed for
a specific type of service. In the cloud environment, solutions were proposed
for services deployed at the ”software a service” (SaaS) and ”infrastructure
as a service” (IaaS) layers. These trust solutions generally differ in the
used trust factors and trust metrics and are not defined with the same
formulas. To the best of our knowledge, no trust evaluation solutions
targeting precisely data services combining QoS and data quality were
proposed. There is also no solution considering data freshness in their data
trust evaluation models. Nevertheless, they do define data quality metrics,
including data timeliness. Data quality metrics are evaluated in these works,
assuming that the necessary information and meta-data are available for this
evaluation. The general assumption is that data providers may export data
quality observations as proof of goodwill to cooperate and show that they
provide ”good” data. In the absence of information, those measures need to be
defined differently to evaluate data freshness, especially database
timeliness.

##  III Data Service Trust Evaluation Model

Service providers propose services under heterogenous quality of service (QoS)
conditions to support various users needs related to scalability, fast
response time, availability etc. QoS varies from one service provider to
another, from the same service provider at different moments (e.g., from day
to day), or even from different applications according to the application
domain. As a result, services have different associated QoS levels. However,
there are no guarantees that the QoS is continuously ensured, and thus, it is
necessary to provide measures that can serve to determine to which extent a
certain level of QoS can be guaranteed and for how long. Besides, when
services are data providers, it is also necessary to determine data
reliability. As mentioned in the e-health scenario, data reliability is
determined by different factors, including data freshness.

A trust measure can be a representative indicator about services QoS and data
quality used as a quality reference by services consumers. Therefore, a high
trustworthiness level of a given service can be a representative indicator of
the performance and data quality standards followed by a service and satisfies
the quality requirements of service consumers.

To this end, when we seek a data service from a provider, it is relevant to
consider two factors. First, we consider past experiences with the data
service, resulting in an overview of its performance level [13]. Second, we
look at the quality of the data delivered by the data service. Thus, the trust
level of a data service is based on two factors: performance and data quality,
and it is defined in [0,1] as follows:

$T_{DS}=\alpha*Performance+\beta*DataQuality$ (1)

Where $\alpha,\beta$ weights pondering the performance and data quality
factors are described in the following subsection. These ponder values vary
according to the importance given to each of these factors by an application.

###  III-A Performance Factor

Performance describes the computing capacity of a data service defined by
three metrics: availability, time efficiency, and task success ratio. The
general idea behind the performance factor is that the data service is
expected to be available when it is requested, it should adhere to the
response time described in its SLA, and it must systematically deliver data to
consumers successfully.

$Performance=\sum W_{j}*Q_{j}$ (2)

Where $Q_{j}$ = {availability, time efficiency, task success ratio}. $W_{j}$:
weight of the metric j, which varies according to the importance of a metric
in a given context to the data service’s consumer.

  * •

Availability (Av): A data service is unavailable when a request is denied [16,
29]. For instance, doctors in the e-health scenario must be sure that data
services are available whenever they need to access Alice’s data.

Availability can be defined as the degree to which a data service is
operational and accessible when requested. Therefore, we define availability
in [0,1] as follows:

$Av=\frac{A_{k}}{N_{k}}$ (3)

Where $A_{k}$ is the number of accepted requests by the data service k and
$N_{k}$ is the total number of requests submitted to the data service k.

  * •

Task Success Ratio (TSR): Sometimes, data services may be available and
accessible but are unable to deliver data to data consumers due to network
failures, time outs set by the consumer, etc. The task success ratio of a
service measures successful data delivery in response to accepted requests.
Back to our scenario, a low task success ratio associated with a given data
service indicates to doctors that there is a considerable risk of not getting
the needed information.

$TSR=\frac{S_{k}}{A_{k}}$ (4)

Where $S_{k}$ is the number of successful requests where data arrived to
destination by the data service _k_. TSR is in [0,1].

  * •

Time Efficiency (TE): quantifies how well a data service meets the expected
response time (ERt) promised by the service provider. Indeed, having a data
service with a low response time violation on average w.r.t ERt indicates that
one can rely on this service for urgent decision making more than a service
with a higher response time violation. Time efficiency metric is defined in
[0,1] as follows:

$\displaystyle TE=$ $\displaystyle\quad 1-\frac{Rt}{ERt}$
$\displaystyle\qquad\textrm{if}\qquad Rt<ERt$ (5) $\displaystyle TE=$
$\displaystyle\quad 0$ $\displaystyle\qquad\textrm{if}\qquad Rt>ERt$ (6)

where $Rt$ is the response time of the service on average.

###  III-B Data Quality Factor

Knowing the quality of data delivered by data services helps to determine the
reliability of data and its fitness-for-use. According to the literature [5],
data quality can be evaluated using multiple data quality dimensions including
completeness, timeliness, accessibility, interpretability etc.

In our work, we estimate data data freshness because we apply our results to
applications for which data must be up to date, for example e-health
applications.

Data freshness indicates the extent to which data is fresh in the sense that
it is meaningful for a target application [22, 36]. The principle behind this
is that fresh data is valuable and trustworthy compared to outdated data that
loses value and can negatively affect decisions made using it.

Data freshness is measured using the notion of timeliness covered by two
dimensions: _data timeliness_ measuring the extent to which data is up-to-
date; and _database timeliness_ measuring how often data is inserted into a
database. Intuitively, the idea is that ”data is made available as quickly as
necessary to preserve its value” [34, 35]. Thus, data is said fresh when it is
up-to-date and inserted frequently into a database. Let $T_{D}$ and $T_{DB}$
denote respectively data timeliness and database timeliness for a given data
set.

We assume that data are produced under different production rates $P_{R}$ that
indicate how often data is collected (i.e. produced), for example, from the
”real world” and timestamped when produced. Data production rate may be static
(i.e., it does not change over time) or programmed to be triggered upon the
production of specific events. For instance, in the e-health scenario, the
smartphone or smartwatch can be programmed to collect temperature when Alice
is active or far from home.

We suppose that data remains fresh within a specific time interval $T$ called
the validity interval. $T$ is predefined according to the application domain.

Assuming that data producers continuously send data to a data service with a
specific update frequency $U_{f}$.

  * •

Data Timeliness captures the gap between the production of data and the time
when it is needed and makes sure it is produced within the data validity
interval $T=$[$t_{min},t_{max}$].

We define timeliness for data D in [0,1] as follows:

$\displaystyle T_{D}=$ $\displaystyle\quad 1-\frac{t_{R}-t_{P}}{T}$
$\displaystyle\qquad\textrm{if}\qquad t_{R}<t_{max}$ (7) $\displaystyle
T_{D}=$ $\displaystyle\quad 0$ $\displaystyle\qquad\textrm{if}\qquad
t_{R}>t_{max}$ (8)

Where $t_{R}$ represents the request time, $t_{P}$ is the data production
time, $t_{max}$ is the maximum time for data to be fresh and $t_{min}$ =
$t_{P}$. The closer $T_{D}$ is to $t_{max}$, the less the data is considered
timely, that is, the less fresh it is. Beyond $t_{max}$, data in no longer
fresh.

For example, using our scenario, data timeliness indicates to the doctors that
the data service managing home thermometer measures might provide outdated or
stale temperature readings that might not be accurate concerning the actual
patient’s health state.

Figure 1: Data Timeliness.

  * •

Database timeliness measures how often a database is updated given by the
database update frequency. We say that a database is updated when new data are
inserted. The intuition is that frequent updates can contribute to the
preservation of data freshness. The frequency of insertions gives the database
timelines. For black-box services, this frequency must be ”guessed” by
comparing data obtained by successive requests. Thus, the database timeliness
is not stated by a formula but through a protocol based on statistical and
heuristic analytics strategies. The protocol concerns our ongoing work, and it
is out of the scope of this paper.

From the above definitions of data quality metrics, it is possible to have an
intuition concerning a possible correlation among the data freshness metrics.
For example, if a database is updated frequently, it is more likely to be
timely, but not necessarily the contrary. If the database has not been updated
within a data validity interval (stated by the application domain of data
consumers), data is more likely to be outdated from the data consumer
perspective. Thus, we define data quality in [0,1] as follows:

$DataQuality=T_{D}*T_{DB}$ (9)

##  IV DETECT: General Description

Figure 2 shows the general structure of DETECT that implements our trust
evaluation model for black-box data services. The main objective of DETECT is
enabling data requesters to select the most reliable data service according to
their needs providing a trust-sorted list of data services tagged with their
up-to-date trust index. DETECT is composed of three main modules: (i) the
performance measuring module (PMM); (ii) the data quality measuring module
(DQMM); and (iii) the trust measuring module (TMM). PMM monitors data
service’s performance, DQMM observes and collects relevant metrics for
evaluating the data quality provided by a data service, and TMM collects both
data quality and performance measurements and computes data services trust
scores.

Figure 2: Data Service Trust Evaluation Architecture.

###  IV-A Performance Measurement Module

The PMM performs its operations in two phases: monitoring and evaluation:

  * •

During the monitoring phase, the Performance Monitor scrapes continuously the
performance metrics using the service’s API including data service response
time, task success ratio and availability and record for a given instant (i.e.
timestamp) their values.

Several interesting tools for scraping and observing such performance metrics
have been proposed in the literature [39] and are available on the internet
(i.e. JMETER, Prometheus etc.).

The Performance Monitor stores performance measurements in a time-series
database (TSDB) which indicates the recorded response time of services in
milliseconds, whether the service was available and whether it successfully
finished the job for each scraping timestamp.

  * •

During the evaluation phase, the Performance Evaluator creates a performance
level base by collecting evidence from the TSDB.

First, it collects performance measurements made within a specific time
interval including response times for the different timestamps, the number of
requests made, the number of accepted requests and the number of successful
requests within this time interval.

Second, it computes performance metrics as defined in III by computing (1) the
reponse time on average for the corresponding time interval, (2) the
availability and (3) the task success ratio for each service.

Third, the Performance Evaluator evaluates the performance level of the
corresponding service by applying equation 2. The PerfDB database stores the
list of services tagged with their evaluated performance level along with the
evaluation timestamp. This list is updated periodically each time the
performance of the service is evaluated.

an API is made available to enable the trust measuring module to access the
latest recorded performance for the available services.

###  IV-B Data Quality Measurement Module

DQMM observes and evaluates the quality of the data accessed by the
corresponding data service. This evaluation is based on both data timeliness
and database timeliness defined in section III.

The assessment of data timeliness and database timeliness can be achieved from
the knowledge about data sources and meta-data about data quality, including
data production time and database update frequency. As aforementioned, we
suppose that data is timestamped. However, the database update frequency is
unknown to the data service’s user, and the black box character of these data
services creates abstraction about how data are captured and processed.

A resource-consuming data quality evaluation process must be set up to provide
timeliness measures. Thus, we propose defining and deploying a meta-data
observability protocol addressing issues related to the absence of the
necessary meta-data for data quality evaluation, especially the database
timeliness evaluation.

The protocol follows a series of steps. First, the knowledge constructor
observes the database state changes of a given data service as a black box
(DSaaB) using sampling techniques. Changes in which we are interested are the
INSERTs statement performed of the service’s related database.

By pulling data samples continuously with a specific sampling frequency,
whether it is random or static, the knowledge constructor observes and
measures some metrics that help build knowledge about the data service’s
change history. The sampling frequency must be chosen in such a way as to find
a trade-off between retrieving enough representative samples for the
measurement of those metrics and reducing the overhead produced by sending
requests and processing data.

For data sample timeliness, we apply the equation 8 on every data item in the
sample and then compute the average for all the data set. For the database
change of state, one would compare the different database states such as that
we have a $\delta\neq\\{\\}$. $\delta$ represents the intersection between two
data sets belonging to two consecutive samplings. Thus, the approximate update
frequency of the database managed by a data service can be measured. The
database KDB stores those metrics along with the observation’s timestamp.

Using KDB and per observation period, the quality evaluator (1) measures
database update frequency, (2) computes data timeliness on average and (3)
uses those last two metrics to evaluate the data timeliness and thus the data
freshness of the data source in question. The result of this module is a list
of data services tagged with their measured data quality level stored in the
database EDQ. Data quality is measured periodically in order to keep this list
up-to-date.

An API is made available to enable the trust measurement module to access the
latest recorded data quality for the available services.

###  IV-C Data Service Trust Measuring Module

The Trust Measuring Module (TMM) evaluates a data service’s trust level using
the PMM and DQMM results. Thus, two collectors are used in DETECT: the
performance collector which collects performance levels of the available data
services; and the data quality collector which collects data quality levels of
the available data services. These collectors gather trust factors and feed
the trust engine on-demand. The trust engine evaluates the trust level of each
available data service using equation 1. It further enables the data requester
to specify the degree of importance of trust factors, including performance
and data quality. A history of the evaluated trust levels of each service for
the different requests is stored in the trust history which contains for every
measuring timestamp the service ID, its data quality level, its performance
level and the corresponding data service trust level.

##  V Implementation

We implemented a prototype that realizes the QoS-based trustworthy data
services selection. In the following, we present the implementation details
using our e-health scenario; we describe two case studies and the experimental
setting to show the feasibility of our solution.

We used DETECT to set up Alice’s homecare scenario for temperature monitoring.
We used Python to develop the different modules of our architecture (i.e.,
PMM, DQMM and TMM). We use SQLite to implement the database integrated in the
architecture. 333http://sqlite.org.

We used Prometheus444https://prometheus.io, a monitoring tool, and its related
JMETER exporters555https://jmeter.apache.org to implement the Performance
Monitor and to continuously observe availability, response time and task
success ratio metrics.

We implemented data services using HAPI FHIR solutions which are built from a
set of components called Resources used to exchange and$/$or store data and
can be easily assembled to solve a wide range of healthcare-related problems.
A resource is an entity that (1) has a known URL by which it can be accessed,
(2) contains a set of structured data items and (3) has an identified version
that changes if its contents change.

We mainly measured the quality of data accessed through the resource
_Observation_ 666https://www.hl7.org/fhir/observation.html that is a central
element in healthcare, used to support diagnosis and monitoring progress. Uses
of the Observation resource include (1) vital signs such as blood pressure and
temperature, (2) laboratory data like blood cells count, (3) device
measurements such as EKG data or Pulse Oximetry data, and so on. Therefore, we
can develop our e-health scenario using this resource for monitoring
temperature. HAPI FHIR resources are queried using RESTful APIs (GET, POST,
PUT, DELETE, UPDATE).

The scenario runs on a 64GB macintosh Macbook Pro where data services are
deployed on self-contained environments using DOCKER777https://www.docker.com.
Containers were configured with different performance levels managed through
the number of CPU cores and the memory allocated to each container.

###  V-A Trust case studies

This section presents two case studies based on the eHealth scenario to
validate the trust model and DETECT’s applicability. Each case study is
designed to target one or both trust factors, including performance and data
quality. We assume that we have two kind of services: on-demand services and
pushed services. The chosen case studies illustrate various applications
requirements for which performance and data quality will be weighted
accordingly.

  * •

Case Study 1: Alerting

This case study addresses continuous monitoring of Alice’s temperature. In the
case of a significant variation in Alice’s temperature, the devices’ push
service must immediately alert Alice’s doctor. For instance, sending a message
with her current temperature and location. Both performance and data freshness
are essential since up-to-date data must be transmitted immediately when a
sudden variation in temperature occurs. Since Alice’s safety is at risk, data
must be sent timely, and thus the performance criterion for selecting services
seems more critical than data freshness. Therefore, more weight is attributed
to the performance factor when evaluating the trust levels of the available
and pre-selected services using the equation 1.

  * •

Case Study 2: Instant checkup by Alice’s doctor

In this case study, we assume that Alice’s doctor wants to perform from time
to time an instant checkup on her health latest indicators using on-demand
services. For example, she wants to have the latest recorded temperature, and
thus data must be fresh. As described in our scenario, temperature readings
are done manually and digitally using different devices (e.g., hospital’s
devices, her connected thermometer and her manual thermometer), and measures
are taken at different frequencies and rates. This use case emphasises the
data quality factor (i.e., data freshness produced by services managing the
data produced by the different devices). Thus, one expects the system gives
higher weight to data quality factor, when evaluating the services trust using
the equation 1.

###  V-B Experimental Setting

In the case studies described above, three HAPI FHIR servers with different
FHIR standards have been deployed on docker containers: one simulating the
hospital’s server, one simulating Alice’s smartphone’ server and the last one
simulating the SOS server of the hospital. Each server has its independent
database on the corresponding server and is reachable through its own URL.

In order to give access to these servers, we deployed 7 data services each
with its own access point: three data services giving access to the three
devices used by Alice are deployed on the first two HAPI FHIR servers. Only
one service giving access to the third HAPI FHIR server.

Figure 3: Experimental setting: Docker containers.

Note that these servers are configured the same way in our controlled
environment. However, to simulate the variation in the performance of the
different data services, we allocated different number of CPU cores and
different number of services to the three HAPI FHIR servers in a way that: (1)
The bigger the number of the allocated CPU resources, the better the
performance of the service since we have more resources for the server and (2)
The smaller the number of the deployed services on the same container, the
better the performance since the available resources on the server are shared
between a smaller number of services.

1 CPU core is allocated to Alice’s smartphone’ server, 2 CPU cores are
allocated to the hospital’s server, and 2 CPU cores are allocated to the SOS
server (see figure3). By allocating CPU, we limit the performance of the
different services. Following this logic, service S31 has the highest
performance since it is deployed solely on the container with 2 CPU cores
followed by the three services (S21, S22, S23), which are deployed on the
second container and also have 2 CPU cores and finally the three services
(S11, S12, S13) on the first container have the worst performance rate since
they have only 1 CPU core compared to the second container.

The performance metrics are observed and measured using a configurable thread
group with the help of JMETER exporters. A thread group is a set of processes
executing the same job. JMETER enables us to set the number of processes of a
thread group to access a given service’s URI simultaneously. Each thread
simulates a user request to a data service. JMETER also enables us to set the
duration for all users in the thread group to execute all processes. For
instance, if a thread group has 20 processes and the period to access the
service is set to 40 sec, JMETER will take precisely 40 sec to start all
processes (i.e. users), and thus a process is started every 2 sec. The
observed performance metrics are scraped by Prometheus every 15 sec and stored
in its TSDB. The Performance Monitor has access to this TSDB every 300 sec to
compute the performance of the related data services.

In the absence of real-time real-world data services, we have simulated the
process of data production and data insertion for Alice’s three devices. To
reduce the experimental duration and for the sake of simplicity, we have
configured the data validity interval to 60 sec, which means that data about
temperature remains valid for only 60 sec, and beyond this time, data is no
longer considered fresh.

Also, we have varied data production rates and data insertion rates for all
devices in a way that they have different data quality levels w.r.t our
experimental setting: (1) all data services have the same data production rate
(5 sec), (2) Data services S11 and S21 have the highest insertion rate equal
to 20 sec (2 to 3 insertions per data validity interval), (3) followed by data
services S12 and S22 with an insertion rate of 40 sec (1 insertion per data
validity interval), (4) data services S13 and S23 with an insertion rate of
100 sec (1/2 insertion per validity interval) and (5) last data service S31
has the lowest insertion rate equal to 150 sec (1/3 insertion per validity
interval). According to this configuration, it is expected that services S11
and S21 have higher data quality as they have the highest database timeliness,
followed by S12 and S22, followed by S13 and S23 and finally S31.

Data insertion is performed using APIs of the deployed data services, and data
about Alice’s body temperature are made accessible through the resource
observation. Table I ranks services according to the expected performance and
data quality levels.

TABLE I: Services Ranked According to Trust Factors Performance | Data Quality
---|---
S31 | S11, S21
S21, S22, S23 | S12, S22
S11, S12, S13 | S13, S23
| S31
Figure 4: Experimental setting: Data Service Trust Evaluation.

Once all the experimental setting is ready and to test our solution, we
performed some trust requests to the Trust Engine. As a result, the trust
engine outputs a ranked list of the available services according to their
trustworthiness level.

Our experiment consists of testing the effect of $\alpha$ and $\beta$ over the
trust level of data services and their ranking: For each request, at a given
time _t_ and for a fixed performance and data quality levels, we varied
$\alpha$ and $\beta$ in order to show the effect of the evaluated data quality
level and performance level on the trust level evaluation of data services. To
do so, we decrease the weight $\alpha$ from 1 to 0 while increasing $\beta$
from 0 to 1 (see table II).

Concerning performance evaluation at the Performance Evaluator level, we
currently consider availability, task success ratio and response time equally
important, and thus they have equal weights when computing performance level
as in equation 2.

###  V-C Results and Discussion

We performed tests using the configuration described above in order to verify
our QoS-based trust model and architecture. As described for the test, we sent
a trust request and observed the results. The resulting ranked lists of
services are presented in table II.

TABLE II: trust Request Output $\alpha$=1,$\beta$=0 |  $\alpha$=0,7,$\beta$=0,3 |  $\alpha$=0,5,$\beta$=0,5 |  $\alpha$=0,3,$\beta$=0,7 |  $\alpha$=0,$\beta$=1
---|---|---|---|---
S31 | S22 | S21 | S21 | S21
S23 | S21 | S22 | S22 | S11
S22 | S31 | S31 | S31 | S22
S21 | S23 | S23 | S23 | S12
S13 | S11 | S11 | S11 | S31
S12 | S22 | S12 | S12 | S13
S11 | S13 | S13 | S13 | S23

According to table II, we notice that:

  * •

The service S31 has the highest trust level when we only give importance to
service performance ($\alpha$=1 and $\beta$=0); the less we give importance to
performance, the more S31 is ranked lower in the list (5th place when
$\alpha$=0).

  * •

The hospital’s services (S21, S22, S23) are ranked below S31 when $\alpha$=1
and $\beta$=0. The more we give importance to the data quality factor, the
better they are ranked and overpass the trust level of S31 excluding S23.

  * •

Services deployed on the first container (S11, S12, S13) are ranked last in
the list when $\alpha$=1 and $\beta$=0. The more we give importance to the
data quality factor, the better they are ranked and overpass the trust level
of S31 excluding S13. No matter the weight we attribute to trust factors,
services deployed on the smartphone HAPI FHIR server are consistently ranked
below the services deployed on the hospital’s HAPI FHIR server.

Hereafter, we discuss the results for each case study.

  * •

Case study 1: as aforementioned, in this case study, we are only interested in
selecting data services with the highest performance level: $\alpha$=1 and
$\beta$=0.

According to the above observations and as shown in table I, experimentations
demonstrate the feasibility of our trust model and architecture since results
provide the suitable trust-based ranking of data services as expected. Note
that in our configurations, we control the performance of HAPI FHIR servers
but not the performance of services deployed on the same server. These
services run independently, and the allocation of resources among them depends
on the docker containers’ load balancing and scheduling method. However, for
services deployed on the same container, we can perform some actions to
diminish the performance of some of them, like sending more user requests.

  * •

Case study 2: in this case study, we are interested in selecting data services
with the highest data quality: $\alpha$=0 and $\beta$=1.

According to the above observations: (1) Services S21 and S11 have the best
data quality. This result is not surprising because the configured data
insertion rate for these two services is the highest. However, S21 is better
than S11. (2) As expected, S12 and S22 are ranked next since they have a lower
insertion rate. Still, S22 is better than S21. Services S13 and S23 followed
them. (3) S31 was ranked before S13 and S23.

These results can be explained by the correlation that exists between data
quality and services’ performance. The value of the data timeliness highly
depends on the performance level of the corresponding service. The
characteristics of the infrastructure negatively influence data timeliness. An
infrastructure configuration providing limited resources punishes the
service’s response time, and in consequence, the data timeliness suffers too.
Note that this correlation affects the data timeliness value because data in
our scenario changes frequently and has a very small validity interval (in
matter of seconds). Thus, data looses rapidly its freshness: the longer the
service takes to respond to the query, the worst the data freshness. For
instance, services on the hospital’s server are always better ranked than
those deployed on the smartphones server even though their data production and
insertion rates configuration is similar.

The ”bad” metrics of the smartphones servers are because the performance
configuration of the hospital’s server gives access to more resources.

This dependence should be represented through the data service trust
evaluation model defined in equation 1. Nevertheless, we did not consider this
dependence yet between the trust factors in our proposed trust model.

##  VI Conclusion and Future Work

This paper proposes a QoS-based trust evaluation model and architecture for
black-box data services. In the model, trust evaluation is based on service
performance and data quality. Service performance is evaluated considering a
weighted addition of the response time, the task success ratio, and the
service’s availability.

Data quality is calculated as a correlation of data timeliness and database
timeliness. The first result based on a data quality observability protocol
shows the advantage to take both data quality and service performance in the
service trust calculation. This paper introduces DETECT, which implements the
trust evaluation model. We used an e-health scenario implemented on top of our
system DETECT to validate the feasibility of the proposed model.

In future work, we intend to enrich DETECT with a data quality evaluation
proposal that allows to guess data timeliness and database timeliness without
having access to data quality features with full observability. We also intend
to study the effect of the variation of the trust factors over the ranking of
services by performing multiple requests over time to show the effect of the
variation of the performance level and the data quality level of services on
their trust ranking. This situation can be tested by injecting faults in
services at different moments to decrease their data quality or performance.
Finally, as discussed in the experimental results section, we need first to
study the correlation/dependence that exists between the performance of given
data service and its data quality and to enhance our trust model in a way to
reduce/eliminate/delete the bias of the performance on the data quality
factor.

## Acknowledgment

Thanks to yy888to be precised later for funding this project xx999to be
precised later.

## References

  * [1] L. Lavieri, C. Koenig, O. Teuffel, P. Agyeman, R.A. Ammann. “Temperatures and blood counts in pediatric patients treated with chemotherapy for cancer”, NCT01683370. 2019, Sci Data 6, 108. https://doi.org/10.1038/s41597-019-0112-8
  * [2] M. van Vliet , JP. Donnelly, CM. Potting, NM. Blijlevens. “Continuous non-invasive monitoring of the skin temperature of HSCT recipients”. Support Care Cancer. 2010, vol. 18(1), pp. 37–42. doi:10.1007/s00520-009-0627-1
  * [3] L. Berti-Equille, F. Moussouni. “Quality-aware integration and warehousing of genomic data”. In ICIQ’05-10th International Conference on Information Quality, pp. 1–15, 2005.
  * [4] D. Ardagna, C. Cappiello, W. Samá, M. Vitali. “Context-aware data quality assessment for big data”. in Future Generation Computer Systems, vol. 89, pp. 548-562, 2018.
  * [5] authors . “Title.” International Conference, year.
  * [6] authors . “Title.” International Conference, year.
  * [7] J. Mitchell, S. Rizvi, J. Ryoo, “A Fuzzy-Logic Approach for Evaluating a Cloud Service Provider”. in 1st International Conference on Software Security and Assurance, ICSSA, 2015.
  * [8] Ritu, S. Jain, ”A trust model in cloud computing based on fuzzy logic”. IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), 2016, pp. 47-52, doi: 10.1109/RTEICT.2016.7807780.
  * [9] X. Zhao, L. Shen, X. Peng, W. Zhao. “Toward SLA-constrained service composition: An approach based on a fuzzy linguistic preference model and an evolutionary algorithm”. in Information Sciences, vol. 316, pp. 370-396, 2015.
  * [10] O. Jules, A. Hafid, M. A. Serhani. “Bayesian network, and probabilistic ontology driven trust model for SLA management of Cloud services”. in IEEE 3rd International Conference on Cloud Networking, CloudNet, 2014.
  * [11] Z. Wu, Y. Zhou. “Service trustworthiness evaluation using neural network and fuzzy logic”. in IEEE International Conference on Services Computing, SCC, 2016.
  * [12] S. Chattopadhyay, A. Banerjee. “QoS Value Prediction Using a Combination of Filtering Method and Neural Network Regression”. in International Conference on Service-Oriented Computing, 2019.
  * [13] C. Mao, R. Lin, C. Xu, Q. He. “Towards a Trust Prediction Framework for Cloud Services Based on PSO-Driven Neural Network”. in IEEE Access, vol. 5, pp. 2187-2199, 2017.
  * [14] M. Alhanahnah, P. Bertok and Z. Tari, ”Trusting Cloud Service Providers: Trust Phases and a Taxonomy of Trust Factors,” in IEEE Cloud Computing, vol. 4, no. 1, pp. 44-54, Jan.-Feb. 2017, doi: 10.1109/MCC.2017.20.
  * [15] J. Sidhu et S. Singh. “Design and Comparative Analysis of MCDM-based Multi-dimensional Trust Evaluation Schemes for Determining Trustworthiness of Cloud Service Providers”. in Grid Computing, vol. 15, n. 12, pp. 197-218, 2017.
  * [16] P. Manuel. “A trust model of cloud computing based on Quality of Service”. in Annals of Operations Research, vol. 233, n. 11, pp. 281–292, 2015.
  * [17] L. Qu, Y. Wang, M. A. Orgun. “Cloud service selection based on the aggregation of user feedback and quantitative performance assessment”. in 10th International Conference on Services Computing, 2013.
  * [18] S. M. Habib, S. Ries, M. Muhlhauser. “Towards a Trust Management System for Cloud Computing”. in 10th International Conference on Trust, Security and Privacy in Computing and Communications. IEEE, 2011.
  * [19] L. Bao. “QoS-based trust computing scheme for SLA guarantee in cloud cvomputing system”. in International Conference on Computing Intelligence and Information System (CIIS). IEEE, 2017.
  * [20] Y. Yuyu. “A novel TOPSIS evaluation scheme for cloud service trustworthiness combining objective and subjective aspects”. in Journal of Systems and Software, pp. 143, 2018.
  * [21] Ghosh, A., Arce, I. “Guest editors’ introduction: In cloud computing we trust-But should we?”. in Journal of IEEE security & privacy, 8(6), 14-16. (2010).
  * [22] O. Chayka, T. Palpanas, P. Bouquet. “Defining and measuring data-driven quality dimension of staleness”. Università degli Studi di Trento. 2012.
  * [23] O. Hartig. “Provenance Information in the Web of Data”. in Proc. of the Linked Data on the Web Workshop at WWW, 2009.
  * [24] C. Dai, D. Lin, E. Bertino, M. Kantarcioglu. “An approach to evaluate data trustworthiness based on data provenance”. In Workshop on Secure Data Management, pp. 82–98. Springer, Berlin, Heidelberg. 2008
  * [25] C. Dai, H. S. Lim, E. Bertino, Y. S. Moon. “Assessing the trustworthiness of location data based on provenance”. In Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pp. 276–285, 2009.
  * [26] E. Bertino, H. S. Lim. “Assuring data trustworthiness-concepts and research challenges”. In Workshop on Secure Data Management, pp. 1-12. Springer, Berlin, Heidelberg, 2010.
  * [27] Lim, H. S., Moon, Y. S., Bertino, E.: “Provenance-based trustworthiness assessment in sensor networks”. In Proceedings of the Seventh International Workshop on Data Management for Sensor Networks (pp. 2-7). (2010).
  * [28] T. Aamir, H. Dong, A. Bouguettaya. “Trust in social-sensor cloud service”. In 2018 IEEE International Conference on Web Services (ICWS), pp. 359-362. IEEE. 2018.
  * [29] Geraci, A., Katki, F., McMonegal, L., … Springsteel, F.:IEEE standard computer dictionary: Compilation of IEEE standard computer glossaries. IEEE Press. (1991)
  * [30] M. Mrabet, Y. ben Saied, L. A. Saidane. “A new trust evaluation approach for cloud computing environments”. In 2016 International Conference on Performance Evaluation and Modeling in Wired and Wireless Networks (PEMWN), pp. 1-6. IEEE.
  * [31] H. Jin, K. Zhou, H. Jiang, D. Lei, R. Wei, C. Li. “Full integrity and freshness for cloud data”. Future Generation Computer Systems, vol. 80, pp. 640–652, 2018.
  * [32] A. Labrinidis, N. Roussopoulos. “Exploring the tradeoff between performance and data freshness in database-driven web servers”. The VLDB Journal, vol. 13(3), pp. 240–255.
  * [33] S. Neumaier, J. Umbrich. “Measures for assessing the data freshness in Open Data portals”. In 2016 2nd International Conference on Open and Big Data (OBD), pp. 17–24. IEEE. 2016
  * [34] S. Kubler, J. Robert, S. Neumaier, J. Umbrich,Y. Le Traon. “Comparison of metadata quality in open data portals using the Analytic Hierarchy Process”. Government Information Quarterly, vol. 35(1), pp. 13–29. 2018
  * [35] M. Bouzeghoub. “A framework for analysis of data freshness”. In Proceedings of the 2004 international workshop on Information quality in information systems, pp. 59-67. 2004
  * [36] V. Peralta, R. Ruggia, Z. Kedad, M. Bouzeghoub. “A Framework for Data Quality Evaluation in a Data Integration System”. In SBBD, pp. 134–147, 2004.
  * [37] M. I. M. Saad, K. A. Jalil, M. Manaf. “Data provenance trusted model in cloud computing”. In 2013 International Conference on Research and Innovation in Information Systems (ICRIIS), pp. 257–262, IEEE, 2013.
  * [38] S. Zawoad, R. Hasan, K. Islam: “Secprov: Trustworthy and efficient provenance management in the cloud”. In IEEE INFOCOM Conference on Computer Communications pp. 1241–1249, IEEE, 2018.
  * [39] X. Li, J. Yuan, H. Ma, W. Yao. “Fast and parallel trust computing scheme based on big data analysis for collaboration cloud service”. IEEE Transactions on Information Forensics and Security, 13(8), 1917–1931, (2018)
  * [40] M. Chiregi, N. J. Navimipour. “A new method for trust and reputation evaluation in the cloud environments using the recommendations of opinion leaders’ entities and removing the effect of troll entities”. Computers in Human Behavior, 60, (2016)
  * [41] M. Alhamad, T. Dillon, E. Chang. Conceptual SLA framework for cloud computing. In: 4th International Conference on Digital Ecosystems and Technologies, pp. 606–610. IEEE, China, 2010.