‘_ Prolog-pretty language = Prolog-pretty, upquote = true, basicstyle=,
breakatwhitespace=false, stringstyle = , commentstyle = , literate = :-:-2 ,,1
..1

11institutetext: fortiss GmbH, Munich, Germany
11email<EMAIL_ADDRESS>
# Monitoring Auditable Claims in the Cloud

Lev Sorokin  Ulrich Schoepp

###### Abstract

When deploying mission-critical systems in the cloud, where deviations may
have severe consequences, the assurance of critical decisions becomes
essential. Typical cloud systems are operated by third parties and are built
on complex software stacks consisting of e.g., Kubernetes, Istio, or Kafka,
which due to their size are difficult to be verified. Nevertheless, one needs
to make sure that mission-critical choices are made correctly. We propose a
flexible runtime monitoring approach that is independent of the implementation
of the observed system that allows to monitor safety and data-related
properties. Our approach is based on combining distributed Datalog-based
programs with tamper-proof storage based on Trillian to verify the premises of
safety-critical actions. The approach can be seen as a generalization of the
Certificate Transparency project. We apply our approach to an industrial use
case that uses a cloud infrastructure for orchestrating unmanned air vehicles.

##  1 Introduction

Cloud systems are used in various domains, e.g., in healthcare, entertainment,
in the financial or the automotive sector. Following the HashiCorp cloud
report111https://www.hashicorp.com/state-of-the-cloud over 94% of all
enterprises world-wide use cloud services. Deployment of systems in the cloud
promises cost savings, scalability – by automatic provisioning of resources
depending on the request load – and high availability.

While cloud technologies have many advantages, using them for mission-critical
applications presents many challenges. We need to address several risks, such
as hardware, software faults or attacks that can have an impact on the
integrity of processed information or the availability of a service. However,
cloud software stacks based on components like Kubernetes or Kafka are
complex222https://www.influxdata.com/blog/will-kubernetes-collapse-under-the-
weight-of-its-complexity/ and are updated frequently, which makes assurance
tasks difficult.

If it is not feasible to verify the implementation, then one approach to
guaranteeing the safety of critical decisions is to track the events and
actions of the implementation in an independent runtime monitoring layer and
to verify the safety of decisions using the collected data.

Runtime monitoring is already standard in cloud systems in the form of
Security Information and Event Management (SIEM) [15, 4]. One continuously
aggregates log data from all cloud system components and correlates and
analyses it to identify potential security issues. However, information flows
only from the components to the SIEM and this allows one to identify problems
only after the fact. To make sure that mission-critical decisions are made
safely, we want to perform the analysis during system operation and provide
the results to the components so that they can make their decisions safely. An
example instance of this idea is the Certificate
Transparency333https://certificate.transparency.dev/ project, where HTTPS
certificates are stored in a public ledger that allows their verification, and
browsers can make the decision on whether to accept a certificate based on
information in such a ledger.

The problem of analysing the behavior with runtime monitoring has been the
topic of extensive work, see, e.g., [7]. For example, monitoring approaches
exist based on generating monitors from automata or using event stream
processing [12, 10, 6, 11]. To assure the safety of mission-critical
decisions, we are interested in a particular monitoring problem that does not
seem to be accounted for very well by such approaches. We want to record the
relevant history of data leading up to a mission-critical decision and then
verify that this data supports the decision before executing it. For example,
in a cloud service that optimizes the operation of a world-wide fleet of
unmanned air vehicles, we want to verify all preparation steps before starting
any drone. To do this, we typically need data from the event history, e.g.,
the JSON body of API requests, that finite state automata abstract.

In addition, documentation/auditability of process steps is becoming more and
more important, as seen in regulatory requirements for the privacy protection
(GDPR) or the protection of health data (HIPAA), among others. Traceability of
events is in the interest of a service provider to have a convincing argument
in case of disputes. Existing approaches that consider auditability, such as
[1, 21, 19], are based on distributed ledger technology and are only
applicable to low-volume processes, such as logistics or manufacturing,
because of high transaction latencies and low transaction rates (in the order
of 1000 transactions per second).

We present an approach, that builds on ideas of the Certificate Transparency
project. We generalized the approach to monitor safety properties in a cloud
system, since it is not directly applicable to monitor events in an arbitrary
cloud system. Our approach is characterized by the following contributions: 1)
We introduce Cyberlog, a Datalog-based language for specifying the system
state and properties in a distributed system. The language allows us to
specify temporal properties, parallel process instances and invariants about
data in messages. 2) A property is monitored by the deployment of distributed
security monitors which capture all messages sent between components during
runtime of the cloud system and analyze these events continuously. This
approach allows to generate a view on the systems state during execution
continuously without having access to the application code, i.e., it is not
intrusive. System components have access to the Cyberlog-claims, in particular
the result of the analysis of all available information, and can use this
knowledge in their decisions. 3) We integrate Trillian/Rekor into the
monitoring architecture for the tamper-proof logging of claims to allow an
indisputable/auditable documentation of events. 4) We provide a revision model
to make reasoning with claims feasible. A knowledge revision is necessary
because the size of the generated log data at each distributed monitor
otherwise does only increase. We demonstrate the applicability of our approach
on a prototypical cloud system from the avionics domain for booking
transportation services with unmanned air vehicles.

##  2 Motivating Example

Figure 1: RTAPHM digital platform system architecture

We motivate our approach with a use case from the RTAPHM
project444https://www.fortiss.org/forschung/projekte/detail/rtaphm. Unmanned
air vehicles (UAVs) have gained high popularity in the last few years, their
applications range from cargo transportation, to search-and-rescue operations
or environmental monitoring555https://droneii.com/237-ways-drone-applications-
revolutionize-business. The RTAPHM project develops a cloud platform for
orchestrating UAVs for such uses, in particular for the transportation of
transplant organs666https://www.cbc.ca/news/canada/toronto/first-lung-
transplant-drone-1.6208057. Such applications are subject to legal
auditability requirements. For example, §486.346 of the US Code of Federal
Regulations requires organ transport operators to “develop and follow a
written protocol for packaging, labeling, handling, and shipping organs in a
manner that ensures their arrival without compromise to the quality of the
organ.”

The platform consists of several micro-services as shown in Fig. 1: The
Service Broker (SB) for checking the correctness and authorization of
requests; the Multi-Resource-Manager (MRM) for managing available resources;
the Cognitive Assistant (CA) for optimizing the allocation of resources; the
Fleet Manager (FM) for predictive maintenance of UAVs; and the Operation
Manager (OM) for triggering the flight preparation tasks and launches of UAVs.
Each service has a local database. The services communicate both via REST APIs
and also event-based using the Kafka message broker.

The platform has an interface to Drone Operators (DOs), which are in charge of
executing planned UAV missions. DOs deploy a Drone Operation Manager (DOM)
component, which automates this task, using their own infrastructure. The DOM
is responsible for executing UAV missions and for regularly sending status
information (e.g., position, velocity) about UAVs on missions to the cloud
platform. In addition, the cloud platform is connected to a public web
application (App) to receive service requests from users. The connection to
both the DOM and the App is realized using an ingress gateway, what is common
in cloud infrastructures. We describe in the following the typical workflow in
the cloud platform for booking a UAV.

Organ transport example. A user requests an organ transport from destination
$A$ to destination $B$, the SB checks the request and forwards it to the MRM
and CA to select a UAV as well as assets as transport boxes, and personnel to
meet the request constraints (e.g., delivery-time, organ weight). The user
receives as a response a set of booking options together with prices. After he
has selected one option the request turns into a mission, the personnel is
triggered by the OM to prepare the drone and calculate waypoints for the
flight. If the preparation is completed, the DO receives a ready-to-fly (RTF)
request with the waypoints to launch the drone.

We next formulate some example properties that shall not be violated when
scheduling UAVs for organ transportation. We assume that these requirements
are derived from a careful safety analysis of the system, e.g., using an STPA
[18].

  1. 1.

Process requirements: A DO wants to start a mission only if all required steps
for a mission start have been completed and documented, e.g, a service request
is documented, the booking steps have been completed by all components and the
drone has been prepared correctly by ground staff.

  2. 2.

Temporal requirements: The time between receiving and sending a message in the
system should not exceed a threshold. If an RTF-message arrives delayed at the
DO, as a consequence the organ will be damaged.

  3. 3.

Data-related requirements: The waypoints in the RTF message have to match the
waypoints of one registered UAV base. Otherwise, an attacker could modify
waypoints in the RTF message that are sent to DO.

Our goal is to verify such properties independently of the implementation and
to provide the results to the system component. The DOM should be able to rely
on all process requirements being checked and documented before starting a
UAV. In the following sections we explain our approach to achieving this.

##  3 Monitoring Approach

Our proposed monitoring system adds a verification layer to the system.
Security monitors are added to the components of the systems, as Fig. 2(a)
shows. These monitors collect events during system operation, they analyse the
data to verify requirements during operation, and they provide the results of
their analysis to the components. To verify process requirements, the monitors
must be able to observe the interactions between multiple components. To this
end, they communicate claims about the process execution with each other. Each
monitor has an assigned identity and claims are exchanged in the form of
cryptographically signed attestations. For accountability and documentation of
processes, the monitors communicate with a claim database that stores the
exchanged claims in a tamper-proof manner.

(a) Monitoring with security monitors

(b) Security monitor architecture

Figure 2: Overview Security Monitoring

The three main aspects of the monitoring system are a specification language
for defining requirements, a security monitor and tamper-proof claim storage.
In the following section we outline them in more detail.

###  3.1 Specification Language

In order to monitor the dynamic behaviour of a system, we need to specify the
required behaviour/properties in a specification language. For this, we
propose an extension of Datalog [8], called _Cyberlog_. Our language is based
on Datalog, because Datalog allows to specify an abstract view on the overall
systems state by using rules that are applied over a dataset. Datalog has a
well-defined semantics and is often used as a query language for declarative
databases, and has been already applied to specify the behaviour of
distributed systems [2, 9]. Datalog requires an interpreter that operates on a
knowledge base (KB) and uses the predefined rules to incrementally update the
KB with new reasoned facts by applying the rules on the given data in the KB.
When new data, such as an API request in the web service is registered and
logged in the KB, the reasoning automatically takes place. This allows us
continuously to update the view on the systems state when new system events
are stored in the KB.

We describe briefly the syntax of Cyberlog: A Cyberlog rule has the form  a :-
b_1, ..., b_n, where a, b_i are atoms. The atom a is called the head of the
rule, while the b_i form the body of the rule. In Cyberlog, the atoms are
attestations of the form  principal attests symbol(t_1,...,t_n), where
principal is a term denoting the identity of a component or person associated
with the system, symbol is a predicate and the t_i are terms. A term can be a
variable (starts with an uppercase character), or a constant, such as string
literals ’text’ or numbers.

A Cyberlog program, called rulesheet, consists of a set of Cyberlog rules.
Such a program is intended to be executed by a single principal. We therefore
allow the user to write short symbol(...) for self attests symbol(...), where
self is the identity of the principal executing the program. We require that
the head of rules may only contain such self-attestations, so that principals
cannot make claims for others.

An example rulesheet for the principal ’SB’ is given in Listing 1 below. It
begins with a declaration of the identities of components, where we show the
details only for SB. Identities are represented through a X509 digital
certificate (RFC 5280) and the rulesheet specifies the subject and issuer
information of the certificate.

⬇

1 ’SB’: Subject: ’C=DE, ST=Hamburg, L=HH, O=ZAL, CN=SB’

2 Issuer: ’C=DE, O=Lets Encrypt, CN=R3’

3 ’MRM’: ...

4 ’OM’: ...

5 ’CA’: ...

6

7 // interpretation of events

8 request(RequestId, Data, Time) :-

9 postRequest(’/servicerequest’, Time, Data),

10 get_param_int(Data, ’request_id’, RequestId).

11

12 // workflow

13 good_rtf_exists(RequestId, AircraftId) :-

14 ’SB’ attests request(RequestId, Data, TimeRequest),

15 ’MRM’ attests feasible_config(RequestId, AircraftId),

16 ’OM’ attests tasks_done(RequestId, AircraftId),

17 ’OM’ attests ready_to_fly(RequestId, AircraftId, Data, TimeRTF).

18

19 // time

20 delayed_rtf(RequestId, DelayTime, SentTime) :-

21 ’OM’ attests ready_to_fly(RequestId, AircraftId, DataRTF, TimeRTF),

22 ’CA’ attests mission_confirmed(RequestId, Data, SentTime),

23 DelayTime == TimeRTF - SentTime,

24 DelayTime > 1000.

Listing 1: Example Cyberlog program

The rule in line 13 formalizes property 1 from Section 2. It specifies that an
RTF message is considered as good (modelled by good_rtf_exist) if SB has
registered a Service Request (request), MRM has provided an appropriate
booking option (feasible_config), and if all necessary UAV preparation tasks
(line 16) have been done by OM (tasks_done). The claim good_rtf_exist is
generated when the claims in the body of the rule are all available in the
monitor’s KB.

The rule in line 20 is an example of a time-related property. It verifies
whether an RTF message issued by the DO is delayed (threshold is 1000ms). The
rule generates a delayed_rtf fact whenever a ready_to_fly fact exists, CA has
confirmed the booking request, and the time between the sent RTF and
confirmation exceeds the threshold.

Lines 7-10 demonstrate the definition of views on the available data using
Cyberlog, which is in our example the modelling of a Service Request. The
parameter RequestId is a unique identifier for a service request to
distinguish between different requests, Time is the time when the request was
made, and Data is the content of the message in the request (line 8). The body
of the rule in lines 8-10 declares that a postRequest claim must exist, which
represents an API request events. Cyberlog also provides predicates for
working with JSON data, which is ubiquitous in cloud systems. In line 10,
where a request identifier is extracted from the request data Data (which
holds JSON data) and bound to the RequestId variable.

###  3.2 Security Monitor

As Fig. 2(b) shows, each monitor is configured with an identity and a
rulesheet. The monitor has an integrated Cyberlog-interpreter for the
evaluation of rules and a local knowledge base (KB), where Cyberlog facts are
stored.

The monitor continuously executes the rule from the rulesheet. Initially the
KB is empty. There are three ways in which facts can be added to the KB:

  * •

A fact is derived by applying a Cyberlog rule.

  * •

A claim is passed directly to the monitor, e.g., a claim representing an
event.

  * •

The monitor’s interaction agent loads new facts from the claim database, a
tamper-proof storage of claims. This is e.g., done when other monitors add
facts to the claim database, which the monitor has not yet received.

The monitor also provides a query interface where the services may request
information from the knowledge base. For instance, in our use-case, the DOM
may query good_rtf_exists before starting a drone.

###  3.3 Tamper-proof Claim Database and Auditability

The monitoring system guarantees that for each claim in the knowledge base of
a security monitor, the complete chain of how it was derived can be
reconstructed later. To achieve this, all Cyberlog rules and generated facts
are stored in the tamper-proof claim database which shall provide the
following guarantees: 1) Immutability: Once data has been stored in the
database, it cannot be changed afterwards. 2) Protection against manipulation:
Manipulation of the database can be detected. 3) Inclusion proofs: The
database provides cryptographic means to verify that a given claim is really
included in it777https://transparency.dev/verifiable-data-structures/. The
security monitors use inclusion proofs to make sure that only recorded claims
are used in reasoning. Inclusion proofs are necessary, e.g., to cope with the
situation where an attacker attacks the interface of the claim database so
that submitted claims are not stored or stored in a modified form, but are
reported to the security monitors being stored in the original form.

Auditability of claims is then enabled by storing evidence for claims in the
database: (1) For claims derived using rules, the evidence is the instance of
the rule used to derive the claim. E.g., for a claim request(id,data,time) the
evidence would be the rule in line 9 with the claims
postRequest(’/servicerequest’, time, data) and get_param_int(data,
’request_id’, id) in Listing 1. (2) For claims loaded from the claim database,
the evidence is an inclusion proof of the claim in the claim database. (3)
Claims which have been submitted directly to the KB, such as postRequest(...)
(which represents an API request), are atomic assertions that cannot be
verified further.

##  4 Claim Revision Control

We have outlined how Cyberlog allows us to track events and to automatically
derive auditable claims about the execution of a distributed application.
While flexible, the approach is naive with respect to the resource consumption
of monitors: The KB of the individual monitors will only grow larger during
the operation of the system. For the continuous operation of the security
monitors, it is essential to be bound the amount of data in the local KBs.

What we need is a mechanism for retracting claims from the local KBs that are
no longer needed. There seems to be no general criterion to decide when a
claim will not be needed anymore for reasoning, so it does not appear that
claims can be removed fully automatically. Instead, we propose an approach to
give the user the power to control the local KB explicitly. We provide the
user with mechanisms to retract claims as part of the rulesheet
specifications.

Our approach can be seen as a generalization of Dedalus [3], an extension of
Datalog with a notion of time. Dedalus models logic programming with a
discrete notion of times. All claims are annotated with a time and there are
rules such as next a :- b, c to express the transitions of time. The rule
expresses that if b and c are true at some time $t$, then a will be true at
time $t+1$. For monitoring applications, one would be interested in storing
only the claims of the current time.

One of the main conceptual contributions of this paper is to generalize
Dedalus to a workflow language for distributed claim revision control. We
replace the time indices by revision in a revision system inspired by version
control systems such as git or mercurial.

###  4.1 Revision Model

We propose a (knowledge) revision model to account for the change of claims
over time. The idea is to manage claims much like git manages files. The new
knowledge generated by the monitors in their local KBs is considered as a
staging revision of claims. This revision is completed by committing it to the
claim database. Once stored in the claim database, revisions cannot be
modified. However, it is possible to revise a revision, e.g., to revoke
claims, by committing a new revision that supersedes it.

        Revision 431b1:
        - owner: SB
        - supersedes: 551a1
        - includes: 145a1
        - rules: <reference to rules>
        - claims:
            - a (evidence: signed by SB)
            - d (evidence: derived by
                  rule d :- b, c)

Figure 3: Revision model with superseded and included revisions of claims

Fig. 3 illustrates the main concepts in the revision model. A revision (black
dot) stores a set of Cyberlog claims. It is identified in the claim database
by a unique ID, typically a hash like 431b1. Our revision model accounts for
two possible relations between revisions: a revision can supersede another
revision (represented by a solid arrow, e.g., 551a1 supersedes 8f711) or it
can include another revision (represented by a dashed arrow, e.g., between
145a1 and 431b1). That a revision supersedes another means that the claims in
the superseded revision are replaced by the superseding revision. Only the
owner of a revision is allowed to commit a superseding revision. Inclusion of
one revision in another means that the included claims can be used as facts
when applying rules.

For auditability, each claim in a revision is stored with evidence to make it
possible to reconstruct how it was made. The inclusion relation between
revisions allows us to include claims by other principals when applying rules.
All claims from an included revision may be used as facts when applying rules.

###  4.2 Monitoring with Revisions

To integrate the revision model into our monitoring framework, we regard the
local KB of each monitor as an incomplete revision, called staging. When the
monitors start, staging is empty. As before, a monitor can add claims to
staging or facts can be derived by applying rules.

A staging revision is transient, and needs to be committed to be stored
persistently in the claim database. When committed, the claims from staging
become a new revision $R$ in the claim database. After a commit, the monitor
starts with a fresh staging revision that will supersede $R$ when it is
committed.

We allow the user to define how the fresh staging revision should be
initialized from $R$ by means of _next-rules_ in Cyberlog rulesheets. A next-
rule has the form next a :- b1, ..., bn and is applied after the commit of a
staging revision. The claims b1,...,bn are claims of the committed revision.
This rule has the effect of adding the claim a to the new staging revision
after a commit.

As an example, recall the example from Listing 1 above. The following example
for a Cyberlog next-rule specifies that a request should stay in the staging
revision after a commit when it is still in process.

⬇

1 next request(Id, Data, TimeRequest) :-

2 request(Id, Data, TimeRequest),

3 in_process(Id).

The monitors are configured to commit their revisions periodically, so that
each monitor produces a linear chain of revisions, the last one of which
contains the latest set of claims of the monitor.

To access the claims of other monitors, we use the feature that facts of other
revisions can be included for reasoning by specifying the id of the revision
to be included. The facts of the corresponding revision are then retrieved
from the claim database and loaded in the corresponding KB. If the included
revision is, at any time, superseded by another revision in claim database,
then its claims and all consequences will be removed and be replaced with the
claims of the _new_ revision.

##  5 Implementation and Experimental Results

To evaluate the monitoring approach in practice, we have implemented it with
Kubernetes888https://www.kubernetes.io and the Istio service
mesh999https://www.istio.io.

Claim database. Our implementation of the claim database is based on the
tamper-proof log Trillian. Trillian implements an append-only storage and uses
a Merkle-tree data structure for preventing tampering and for providing
inclusion proofs, as discussed in Section 3.3. It is used in large-scale
applications, such as Certificate Transparency or
sigStore101010https://www.sigstore.dev.

Security Monitors. The security monitors implement a Cyberlog interpreter and
the revision model. They are implemented in OCaml and use an efficient in-
memory implementation of Datalog111111https://github.com/c-cube/datalog for
performing reasoning. In the current prototype implementation the monitors are
compiled to JavaScript using the ReScript compiler121212https://rescript-
lang.org/ and then executed using the NodeJS runtime. This makes it possible
to use libraries not available in OCaml, e.g., for verifying Trillian
inclusion proofs.

Figure 4: Security monitor integration using Istio’s Envoy proxy

Cloud Integration. To apply the monitoring approach to a cloud system, the
security monitors must be placed into the system to be able to observe events
and communicate with the components under observation. Our implementation
integrates transparently with a standard infrastructure built from Kubernetes
and Istio. Kubernetes is a system for orchestrating containerized
applications. Istio extends Kubernetes with a programmable software-defined
network. Istio deploys with each application component a proxy container that
is responsible for routing traffic, monitoring performance, enforcing
authorization policies, and similar tasks. We deploy the security monitors
together with the proxy containers with each application component. We use the
_external authorization service interface_ of the proxy container to give the
monitors access to the network traffic. This allows us to make network traffic
available to the monitor and gives us the option to make authorization
decisions in the monitors. Fig. 4 shows the integration of the security
monitor inside a Kubernetes pod (a logical host machine).

HTTP requests to the application component go to the proxy, which forwards
them to the security monitor. The monitor adds information about the request
to its local KB. For example, a POST request generates the fact
postRequest(’/endpoint’, body, timestamp) in the KB, where body is the message
body. Recall that the monitor applies its Cyberlog rule continuously, so all
consequences are also added to the KB.

We have applied the monitoring approach to the RTAPHM system. The rulesheets
and configurations files are
available131313https://git.fortiss.org/sorokin/cyberlog-monitoring.

Prototypical Evaluation. To demonstrate the applicability of our approach, we
have deployed the RTAPHM cloud system described in Section 2 in an on premise
kubernetes cluster with k3s141414https://k3s.io . The service components
inside the cloud have been implemented by different partners with different
technologies. We initiated the scheduling process of a UAV through simulating
requests by directly sending API requests to the SB. Outside the cluster we
did deploy prototypically a DOM application, which has been connected to a UAV
simulation (Gazebo).

We have triggered several parallel booking processes and evaluated the
processing delay caused by forwarding API requests to security monitors.
Further, we measured the amount of facts stored in each KB of a monitor. The
minimal delay was 2ms (at SB), while the average delay was 29ms. The maximal
introduced delay among all monitors was 113ms at MRM which is relatively high.
This is not surprising, as in the rules of the MRM monitor mostly JSON arrays
have to be processed and computation time is required to generate facts for
each element of an array. Further, the minimal overall delay was 135ms, while
the maximal was 157ms. The execution delays can further be reduced by
parallelizing the fact reasoning in the security monitor. Considering the
number of facts, the minimal KB size was 163 (at SB), while the maximal was
259 (at CA). After one service request the KB size increased maximally by less
than 184 facts (maximal fact size is 15KB). The processing of the security
monitor further required 1mCPU which is negligible.

##  6 Related Work

We give a brief overview of related work for the runtime monitoring of
distributed and web-based applications and describe some work that considers
the auditability of system events.

###  6.1 Runtime monitoring

Runtime monitoring of distributed applications has been exhaustively studied.
Many approaches are based on generating automata from temporal logics
specifications or using stream processing [12, 10, 6, 11, 20, 6]. For
instance, [20] introduces a distributed specification language to generate
distributed monitors. Bauer et al. [5] address the distributed synthesis of
monitors from LTL specifications based on system execution traces. In [12],
the authors present an approach to generate non-deterministic finite state
automata for monitoring global liveness and safety properties of web services.
Automata-based approaches cannot be easily extended to support the monitoring
of parallel processes, as well monitoring properties that depend on the data
in the exchanged messages.

Cotroneo et al. have proposed an approach [10] for monitoring a cloud system
using stream processing. API requests are first logged in a fault-free system
execution to learn the correct behaviour of the system in form of rules to
observe later whether the system obeys the rules. Also in this approach, it is
not possible to specify properties that consider data in the API requests.
Also, monitoring results are not provided back to the distributed components,
to control the execution of the component based on the monitoring results.

Our monitoring approach is related to the authorization framework Open Policy
Agent (OPA)151515https://www.openpolicyagent.org, whose idea is to decouple
the enforcement of authorization policies from the application code. Policies
are specified in a declarative language called Rego which is similar to
Cyberlog. While OPA has an interface to integrate external data when applying
rules, processed and reasoned data is not auditable and no attestions are
supported. Since OPA is only an authorization framework, no monitoring of
workflow events is possible.

###  6.2 Auditability of system events

Google’s Certificate Transparency (CT)
framework161616https://certificate.transparency.dev is a prominent example for
making decisions based on auditable data: the goal is to identify falsely or
maliciously issued certificates, by recording all HTTPS certificates issued by
publicly trusted Certificate Authorities in a tamper-proof log called
Trillian171717http://transparency.dev/#trillian. Trillian is a scalable
alternative to distributed, tamper-proof data storages, such as the
blockchain. The concept has been adopted by several projects, whether for
supply chain security in sigStore, identification of malicious firmware with
Binary Transparency [13], for the detection of malicious software updates from
package managers [17] or the verifiable auditing of (confidential) data
accesses [16].

An extension of standard SIEM approaches is provided by syslog-
ng181818https://www.syslog-ng.com/ which assures the secure transmission and
tamper-proof storage of logged data to meet compliance requirements.
Nevertheless, it is not possible to monitor requirements which depend on this
data, so that components can incorporate the monitoring results in their
decisions.

Another work [19] has applied blockchain for the runtime monitoring of
business processes. Recorded events are auditable, but the transaction time
takes more than 7 minutes. This approach is therefore only applicable in a use
case with long-running tasks, such as logistics or manufacturing. Further, a
blockchain deployment is complex and requires a considerable amount of
resources, which is in contrast to our approach when using Trillian.

In [14] the authors have presented an approach making distributed system
auditable using tamper-proof storage focusing on the detection of byzantine
faults. Similiar to our approach, monitored system components have to
cryptographically sign messages, and events are recorded in the tamper-proof
log. However, the approach does not allow the monitoring of workflows based on
auditable claims as well as a flexible specification of monitored properties.

##  7 Conclusion

We have proposed a new monitoring approach for monitoring safety and data-
related properties in cloud applications. To satisfy the distributed nature of
deployed systems in a cloud, we have introduced Cyberlog for the specification
of monitored properties as an extension of Datalog by using attestations.
Monitoring is then performed by the execution of distributed Cyberlog
programs. The approach is non-intrusive because only communication between
services is processed. Further, we presented an approach for the revision of
claims inspired by file revision systems such as git, to handle the problem of
increasing storage of runtime events which comes by using a Datalog-based
language. To allow for the documentation of the processed information to
satisfy regulatory requirements our approach integrates a scalable tamper-
proof ledger to store all processed and reasoned monitoring claims. We
prototypically evaluated our monitoring approach on a cloud system for a
safety-critical use case. Nevertheless, the specification of Cyberlog programs
requires some effort. We are working on improving the syntax for the
specification of properties and investigating approaches for synthesizing
Cyberlog-programs from API specifications and use case information. We assume
that our monitoring approach and further research can increase the security of
cloud systems and promote the deployment of applications for mission-critical
systems in the cloud.

## Acknowledgment

This work was supported by the Federal Ministry for Economic Affairs and
Climate Action in the RTAPHM project, grant 20X1736M (Lufo V-3).

## References

  * [1] Ahmad, A., Saad, M., Mohaisen, A.: Secure and transparent audit logs with blockaudit. Journal of Network and Computer Applications 145, 102406 (2019)
  * [2] Alvaro, P., Condie, T., Conway, N., Elmeleegy, K., Hellerstein, J.M., Sears, R.: Boom analytics: Exploring data-centric, declarative programming for the cloud. In: European Conference on Computer Systems (EuroSys’10). p. 223–236. ACM (2010)
  * [3] Alvaro, P., Marczak, W., Conway, N., Hellerstein, J., Maier, D., Sears, R.: Dedalus: Datalog in time and space. In: Datalog Reloaded. pp. 262–281. Springer (2011)
  * [4] Bachane, I., Adsi, Y.I.K., Adsi, H.C.: Real time monitoring of security events for forensic purposes in cloud environments using SIEM. In: Systems of Collaboration (SysCo 2016). pp. 1–3 (2016)
  * [5] Bauer, A., Falcone, Y.: Decentralised LTL monitoring. Form. Meth. Syst. Des. 48 (2016)
  * [6] Bratanis, K., Dranidis, D., Simons, A.J.H.: An extensible architecture for run-time monitoring of conversational web services. In: Monitoring, Adaptation and Beyond (MONA 2010). p. 9–16. ACM (2010)
  * [7] Cassar, I., Francalanza, A., Aceto, L., Ingó lfsdóttir, A.: A survey of runtime monitoring instrumentation techniques. EPTCS 254, 15–28 (2017)
  * [8] Ceri, S., Gottlob, G., Tanca, L.: What you always wanted to know about datalog (and never dared to ask). IEEE Trans. Knowl. Data Eng. 1, 146–166 (1989)
  * [9] Chu, D., Popa, L., Tavakoli, A., Hellerstein, J.M., Levis, P., Shenker, S., Stoica, I.: The design and implementation of a declarative sensor network system. In: Embedded Networked Sensor Systems (SenSys 2007). p. 175–188. ACM (2007)
  * [10] Cotroneo, D., Simone, L.D., Liguori, P., Natella, R., Scibelli, A.: Towards runtime verification via event stream processing in cloud computing infrastructures (2020)
  * [11] Decker, N., Kühn, F., Thoma, D.: Runtime verification of web services for interconnected medical devices. In: ISSRE 2014. pp. 235–244 (2014)
  * [12] Gan, Y., Chechik, M., Nejati, S., Bennett, J., O’Farrell, B., Waterhouse, J.: Runtime monitoring of web service conversations. In: Computer Science and Software Engineering (CASCON 2017). p. 2–17. IBM / ACM, USA (2017)
  * [13] Google: Firmware transparency (2021), https://github.com/google/trillian-examples/tree/master/binary_transparency/firmware
  * [14] Haeberlen, A., Kouznetsov, P., Druschel, P.: Peerreview: Practical accountability for distributed systems. SIGOPS Oper. Syst. Rev. 41(6), 175–188 (Oct 2007)
  * [15] Harper, A., VanDyke, S., Blask, C., Harris, S., Miller, D.: Security information and event management (siem) implementation (2010)
  * [16] Hicks, A., Mavroudis, V., Al-Bassam, M., Meiklejohn, S., Murdoch, S.J.: VAMS: verifiable auditing of access to confidential data. CoRR abs/1805.04772 (2018)
  * [17] Hof, B., Carle, G.: Software distribution transparency and auditability. CoRR abs/1711.07278 (2017)
  * [18] Leveson, N.: A new accident model for engineering safer systems. Safety Science 42(4), 237–270 (2004)
  * [19] Prybila, C., Schulte, S., Hochreiner, C., Weber, I.: Runtime verification for business processes utilizing the bitcoin blockchain. Future Gener. Comput. Syst. 107, 816–831 (2020)
  * [20] Sen, K., Vardhan, A., Agha, G., Rosu, G.: Efficient decentralized monitoring of safety in distributed systems. In: ICSE 2004. pp. 418–427 (2004)
  * [21] Van Hoye, L., Maenhaut, P.J., Wauters, T., Volckaert, B., De Turck, F.: Logging mechanism for cross-organizational collaborations using Hyperledger Fabric. In: Int. Conf. on Blockchain and Cryptocurrency (ICBC 2019). pp. 352–359. IEEE (2019)