# Efficient Complex Event Processing in Information-centric Networking at the
Edge

Manisha Luthra*  — , Johannes Pfannmüller, Boris Koldehofe*  — Jonas Höchst*
— Artur Sterz*  — Rhaban Hark*  — and Bernd Freisleben*  — M. Luthra, B.
Koldehofe, J. Höchst, A. Sterz, R. Hark and B. Freisleben are with the
Department of Electrical and Computer Engineering at Technical University of
Darmstadt, Germany (email: manisha.luthra@kom.tu-darmstadt.de). J. Pfannmüller
was with Technical University of Darmstadt and is with Software AG, Germany.
Preprint submitted at IEEE IoT Journal on November 16, 2020.

###### Abstract

Information-centric Networking (ICN) is an emerging Internet architecture that
offers promising features, such as in-network caching and named data
addressing, to support the edge computing paradigm, in particular Internet-of-
Things (IoT) applications. ICN can benefit from Complex Event Processing
(CEP), which is an in-network processing paradigm to specify and perform
efficient query operations on data streams. However, integrating CEP into ICN
is a challenging task due to the following reasons: (1) typical ICN
architectures do not provide support for forwarding and processing continuous
data streams; (2) IoT applications often need short response times and require
robust event detection, which both are hard to accomplish using existing CEP
systems.

In this article, we present a novel network architecture, called INetCEP, for
efficient CEP-based in-network processing as part of ICN. INetCEP enables
efficient data processing in ICN by means of (1) a unified communication model
that supports continuous data streams, (2) a meta query language for CEP to
specify data processing operations in the data plane, and (3) query processing
algorithms to resolve the specified operations. Our experimental results for
two IoT use cases and datasets show that INetCEP offers very short response
times of up to $73\mu s$ under high workload and is more than $15\times$
faster in terms of forwarding events than the state-of-the-art CEP system
Flink. Furthermore, the delivery and processing of complex queries is around
$32\times$ faster than Flink and more than $100\times$ faster than a naive
pull-based reference approach, while maintaining $100\%$ accuracy.

###### Index Terms:

Complex Event Processing, Information-centric Networking, In-Network
Processing

##  I Introduction

Information-centric Networking (ICN) is aimed at simplifying the principles of
the Internet by changing its addressing scheme from _named hosts_ to _named
data_. It focuses on the question _what data is required_ rather than _where
to get the data from_. ICN evolved into a key architecture for the edge
computing paradigm towards realizing a content-centric Internet of the future
[49]. Nowadays, ICN is broadly adopted by academia and industry, e.g., by
Internet2, Cisco, and Intel, in real-world deployments [40]. Since ICN
inherently focuses on data, it particularly supports the utilization of in-
network processing functionality, e.g., to specify higher order functions on
the data already within the network, as provided by Complex Event Processing
(CEP). CEP is a powerful _in-network processing_ paradigm that takes a _query_
as input to describe correlations over incoming data streams in order to
deliver _complex events_ in the form of data notifications in response to the
_query_ [10]. An example for a CEP application is a disaster scenario, where
emergency management operations require a heat map of victim locations to
better coordinate the activities of rescue workers [7]. The heatmap can be
produced efficiently using a CEP query in the network based on a set of
incoming sensor data streams. However, existing ICN architectures like Named
Data Networking (NDN) [23] and Named Function Networking (NFN) [47] lack
mechanisms for in-network processing in the data plane. Particularly, these
architectures are restricted in terms of (i) the underlying communication
strategy, and (ii) higher level specification and data processing for
Internet-of-Things (IoT) applications. Thus, ICN architectures can benefit
from the CEP paradigm, but deploying CEP as part of an ICN architecture is
quite challenging.

A major challenge is the lack of support to forward and process incoming
periodic data streams due to ICN ´s underlying communication strategy. State-
of-the-art ICN architectures either rely on a _pull_ -based (_consumer-
initiated_) interaction pattern [23, 48] or a _push_ -based (_producer-
initiated_) interaction pattern [18, 25]. For periodic data streams, _pull-
based_ interaction poses several problems: (1) it causes significant overhead
in terms of request messages required to fetch each data object; (2) it causes
high delays until fresh data becomes available; (3) polling for data might
result into stale data served from an in-network cache; (4) once a producer
does not have any data to deliver, it is possible that the consumer ends up in
repeatedly requesting data and busy-waiting for data to arrive, thus,
unnecessarily allocating resources.

In contrast, changing to a _push-based_ interaction [18] as supported by CEP
systems is problematic for IoT applications that are based on request-reply
interactions. For example, traditional web applications still need
personalized request-reply interaction. Work on mitigating this problem [5]
relies on push-based beacon messages to inform consumers about arriving new
data such that consumers can initiate a request-reply interaction. Other
researchers [21] integrate the push-based semantics in a centralized control
plane. However, such mechanisms – besides adding complexity – either incur a
significant message overhead or might create a bottleneck for large-scale IoT
applications.

Ideally, ICN architectures should provide efficient support for both
interaction patterns as part of a unified communication enabling any type of
application. Initial work in the context of content routing has shown the
potential of a unified communication model [16, 39]. Another proposal solves
this problem by using long lived request packets [21]. Although this approach
provides scalable routing in a unified manner, the authors do not deal with
the side-effect of enabling _push-based_ interaction in ICN, i.e., _flow
imbalance_ [36] resulting in event loss. In the latter approach, long lived
requests result in a large in-network state and might still lead to stale
data.

We address these limitations by presenting a novel network architecture,
called INetCEP 111This article is an extension of the authors’ previous work
[31, 4]., for efficient CEP-based _in-network processing_ in combination with
_a unified communication model_ within ICN. INetCEP enables _efficient and in
order_ processing of data in the forwarding plane to support the requirements
of IoT applications, such as detecting complex events with short response
times (as low as a few milliseconds) in a highly accurate manner, while
potentially dealing with large numbers of events.

Our contributions are as follows:

  1. (1)

We propose a unified communication model that provides coexisting pull- and
push-based communication patterns to realize CEP applications while still
supporting classical request-reply interactions. Additionally, we provide a
rate-based flow control mechanism to mitigate the issue of flooded links to
ensure _flow balance_.

  2. (2)

We introduce a meta query language to express complex event derivations on the
data plane over the ICN substrate.

  3. (3)

We present centralized and distributed query processing algorithms that ensure
efficient and in order processing of queries in a distributed manner.

  4. (4)

Finally, we present an open source implementation [4] and evaluation of
INetCEP on state-of-the-art ICN architectures, namely NFN [48] and CCN-lite
[3], using two IoT case studies and open datasets from the DEBS Grand
Challenge 2014 [2] and a disaster communication scenario [7].

Our evaluation results show that INetCEP is more than $15\times$ faster in
terms of forwarding events compared to the state-of-the-art CEP system _Flink_
[13]. Furthermore, the delivery and processing of complex queries on the
mentioned datasets is around $32\times$ faster than _Flink_ and more than
$100\times$ faster than a naive _pull-based_ reference approach while
maintaining $100\%$ accuracy.

The paper is organized as follows. In Section II, we present related work. In
Section III, we describe the research challenges. In Section IV, we present
the design of INetCEP. In Section V, we provide a comprehensive evaluation of
INetCEP. Section VI concludes the paper and outlines areas for future work.

##  II Related Work

We provide background information and related work on the building blocks of
our proposal: Content-centric Networking (CCN) and Complex Event Processing
(CEP) in the following.

###  II-A Content-centric Networking

Jacobson et al. [23] proposed CCN222In the remainder of the paper, we will use
the terms ICN and CCN interchangeably., where communication is _consumer-
initiated_ , consisting of two packets: Interest and Data. A data object
(payload of a Data packet) satisfies an interest if the name in the Interest
packet is a prefix of the name in the Data packet. Thus, when a packet arrives
on a face333face stands for interface in CCN terminology. (identified by
$face\\_id$) of a CCN node, the longest prefix match is performed on the name
and the data is returned based on a lookup.

CCN data plane: Each CCN node maintains three major data structures:
Forwarding Information Base (FIB), Content Store (CS) (also known as in-
network cache), and Pending Interest Table (PIT). Once an Interest arrives on
a face, the node first checks its Content Store for a matching Data packet by
name. Upon a match, the Data packet is sent via the same face it arrived from.
Otherwise, the node continues its search in the PIT that stores all the
Interest packets (along with its incoming and outgoing face) that are not
satisfied. If an entry exists in the PIT, the face is updated and the Interest
is discarded, because an Interest packet has already been sent upstream.
Otherwise, the node looks for a matching FIB entry and forwards the Interest
to the potential source(s) of the data.

NDN and NFN: NDN [51] emerged as an architecture that is based on the
principles of CCN’s _named data_. NFN is another emerging architecture that
focuses on addressing _named functions_ in addition to _named data_ by
extending the principles of NDN. NFN blends [48] data computations with
network forwarding, by performing computational tasks across the CCN network.
It represents _named functions_ on the data as $\lambda-$calculus expressions.
We aim to encapsulate CEP operators (cf. next section) as NFN named functions
and hence resolve them in the network. However, NFN focuses mainly on
resolving discrete functions on top of the CCN substrate. In contrast, we
focus on continuous and discrete computations (push and pull), expressive
representation of the computation tasks, and their efficient distribution (cf.
Section IV).

###  II-B Complex Event Processing

CEP can process multiple online and unbounded data streams using compute units
called _operators_ to deliver meaningful events to consumers [10]. The
consumers specify interest in the form of a _query_ consisting of multiple
operators. Some of the commonly used operators are defined below [31]:

  1. (1)

Filter checks a condition on the attribute of an event tuple and forwards the
event if the condition is satisfied.

  2. (2)

Aggregate applies an aggregation function such as $max$, $min$, $count$,
$sum$, $avg$, etc., on one or more event tuples. Hence, the data stream must
be bounded to apply these operations. For this purpose, _window_ can be used.

  3. (3)

Window limits the unbounded data stream to a window based on time or tuple
size, such that operators like Aggregate can be applied on the selected set of
tuples444In this article, we consider sliding windows that are most widely
used, although other windows, such as tumbling windows, have been proposed..

  4. (4)

Join combines two data streams to one output stream based on a filter
condition applied on a window of limited tuples.

These operators can be _stateless_ or _stateful_. Filter and Aggregate are
stateless operators, while the other operators are stateful and maintain the
state of input tuples before emitting the complex event and therefore depend
on multiple input tuples to be accumulated before actual emission.

###  II-C ICN Architectures

We classify ICN architectures based on their communication capabilities as
pull-based, push-based, or both push- and pull-based.

Pull-based. Classical architectures in ICN like NDN [51] and NFN [48] use a
consumer-initiated communication strategy. This essentially means that the
consumer _pulls_ the data from the producer by sending an Interest (request)
packet to the network for each data item produced. The NDN network forwards
this request to one or more producers that satisfy the request and then
forward the Data (reply) back to the consumer. Such architectures do not
provide support for forwarding and processing of periodic data streams.
Several solutions building on top of NDN [44, 28] and NFN [43, 42, 47] try to
solve this problem by continuously polling the producer with Interest packets.
This approach works well when the producer sends the data stream at lower
rates, but at higher event rates a lot of traffic is generated. Furthermore,
the consumer can miss data objects or receive stale data.

Push-based. The second class of ICN architectures [25, 18] is based on _push-
based_ or producer-initiated communication similar to publish-subscribe
systems. Such architectures lack support for request-reply interactions such
as web applications. To solve this issue, some authors [5] use so-called
beacon messages to support request-reply interactions. Another work solves
this problem by using long lived request packets [21]. However, the former
approach incurs a significant message overhead, while the latter approach
poses a bottleneck for large-scale IoT applications.

Both push and pull. Some approaches like CONVERGENCE [34], GreenICN [1],
Carzaniga et al. [16] and HoPP [21] provide support for both _push-_ and
_pull-_ based communication strategies. The CONVERGENCE system combines the
publish/subscribe interaction paradigm on top of an information-centric
network layer. In contrast, we provide a unified interface such that pull and
push based interaction patterns can co-exist in a single network layer while
performing in-network computations. GreenICN is an ICN architecture by
combining NDN (pull-based) with COPSS [18] (push-based). However, their focus
is only on post-disaster [46] and video streaming scenarios [1], while we
focus on a wider range of IoT applications. Due to the limited range of
scenarios, their focus also is not on improving response times. Carzaniga et
al. [16] present a unified network interface similar to our work, but the
authors only propose a preliminary design of their approach without
implementing it in an ICN architecture, and subsequently focus on routing
decisions [15] rather than on distributed processing. HoPP [21] is closest to
our approach in implementing coexisting push and pull functionality, but the
authors have moved the push semantics to a centralized control plane which
could be a bottleneck for a large scale IoT network.

###  II-D CEP and Networking Architectures

We now discuss relevant work on CEP, emerging networking architectures, and
data plane languages.

CEP Architectures. Modern event processing architectures support low latency,
e.g., Apache Flink [13], Twitter’s Heron [29], and Google’s Millwheel [6].
However, these architectures operate in a middleware layer while we propose to
use the same functionality in the data plane with ultra-low latency in terms
of forwarding. Another possibility is to interface event processing
architectures with an ICN architecture. Initial work implemented Hadoop on NDN
[20] for datacenter applications. However, this requires changing the network
model to push, which would limit the support for request-reply interactions.
Reliable CEP architectures [24, 37] offer reliable event processing with
minimal overhead, for instance by checkpointing [24] and using buffers [37].
However, they suffer from high latencies either due to the round trip to the
overlay network or by the inherent overhead introduced by the reliability
mechanisms.

Networking Architectures. Another network architecture is Software-Defined
Networking (SDN) [27], which is gradually being deployed, e.g., in Google’s
data centers. It allows network managers to program the control plane to
support efficient traffic monitoring and engineering. The SDN architecture is
complementary to our work, since SDN empowers the control plane, while ICN
upgrades the data plane of the current Internet architecture.

Data Plane and Query Languages. Many CEP systems propose a query language that
supports users to specify and embed the features provided in their
applications [50]. A particular, challenge when supporting network
architectures is to be able to decompose the query specification such that it
can be mapped to the programming models provided by dedicated network
architectures. The novelty of our proposed query language is to provide a
mapping of operations to ICN’s data plane. Alternative designs build on P4
[9]. Initial work on programming ICN with P4 faced several difficulties due to
lack of language features and the strong coupling of the language to SDN’s
data plane model [45].

##  III Research Challenges

We have three main objectives: (1) with coexisting _pull-_ and _push-_ based
communication strategies, we aim to enable wide range of IoT applications over
ICN architectures; (2) we aim to provide low response times for IoT
applications, while maintaining throughput; (3) finally, we aim to provide
support for lossless communication, robust event processing, and hence highly
accurate complex events.

To better understand these objectives, consider as an example smart plugs that
are currently used with devices like Amazon Alexa and Google Assistant,
expected to be a $62 million market by 2023 [41]. One of the important
features of smart plugs is the information they provide to derive powerful
load predictions for power grid providers. By constantly predicting power
usage, providers can put contingency measures in place to manage huge changes
in demand in real time without any interruptions. These operations are
typically managed by control networks, where a machine connected to a control
network performs operations in place of a human. Such operations are time-
sensitive and need to be highly robust. In context of load prediction, the
machine might have to feed more power into the grid to prevent it from
collapsing. Such scenarios need continuous push notifications of load
predictions and, most importantly, the control networks require short response
times to plan for an action. Additionally, smart plugs must be able to take
requests from the end users. such as voice instructions by means of an app in
conjunction with Alexa or Google assistant. The example shows that CCN has to
support both types of communication strategies, pull- and push-bashed, at the
same time.

Beyond coexisting communication strategies, the CCN architecture must provide
means to consumers to specify and represent _queries_ to correlate data from
multiple primitive event producers and process the data. For example, to
specify load predictions on the events generated by smart plugs, there must be
a way to specify those predictions in a data plane query language and means to
derive them.

The third problem of IoT applications on top of CCN is the requirement of
highly _robust_ and _scalable executions_ while providing short response time.
As an example, smart plugs generate millions or even billions of power
readings that have to be processed simultaneously. There is a need to process
this scale of events efficiently in the data plane without overloading the
core network operations of a CCN. Furthermore, event loss is not acceptable in
control networks. Even a single wrong instruction may lead to power outage in
the smart plug example.

To achieve our objectives, we address two core challenges:

R1: Continuous data streams. Currently, there is no ICN architecture that
supports coexisting push- and pull-based communication patterns in a single
architecture. The fundamental reason why ICN architectures such as CCN are
_pure-pull_ -based is their implicit flow control. The one-to-one mapping of
Interest and Data packets along with the PIT table ensures flow balance in an
ICN network [36]. By introducing continuous data streams, the flow imbalance
between the producer and the broker network could cause congestion and data
loss. Although simple flow control mechanisms, e.g., restricting the sending
rate, are valid countermeasures, they cannot detect and actively react to
packet loss, which is crucial for many IoT applications.

R2: Efficient and in-order stream processing. The push nature of continuous
data streams introduces the challenge of providing efficient and in-order
processing of data in the forwarding plane of CCN. First, the outcome of the
CEP operators depends on the original order of events in the data stream as
they are produced. Hence, efficient and parallel processing of operators
requires coordination among the processing threads. Second, large numbers of
events and queries require the distribution of the processing queries into
graphs of operators, which leads to the well-known operator placement problem
[35, 32]. The deployment of operators in a CCN network introduces new
challenges in terms of fulfillment of Quality-of-Service (QoS) metrics like
short response times, throughput, and correctness in delivery of events.

Figure 1: INetCEP communication model supports both push- and pull-based
communication while maintaining a single PIT entry and fetching latest data
object for the query.

##  IV The INetCEP Architecture

In this section, we present the design of INetCEP. We first discuss the core
ideas of INetCEP and then provide details of its architectural components.

###  IV-A Core Ideas

The INetCEP architecture address the challenges discussed in the previous
section based on the following two core ideas:

####  IV-A1 A unified communication layer

We complement the current CCN architecture with three additional packet types
to support the handling of periodic data streams. We introduce the forwarding
plane logic for the newly proposed types and propose algorithms for their
handling. To enable flow control, we use a lossless rate-based flow control
mechanism [30] that balances the rate by using active feedback for the
producer to adjust the sending rate based on the available resources. This
ensures no reduction in response time and prevents packet loss.

####  IV-A2 Reactive and asynchronous handling of data streams

To provide efficient and in-order processing of periodic data streams in the
forwarding plane, we propose algorithms for centralized and distributed query
processing. We perform the decomposition of queries into primitive operations
that are executed in the CCN forwarding plane. An operator placement service
coordinates the deployment of operators onto paths that ensure the maximal
rate and minimal response time in the delivery of events. The processing of
the operators is planned by asynchronous parallel threads while maintaining
so-called sub interests in the PIT of the placed nodes. We offer coordination
of threads to guarantee in-order processing. The lossless rate based flow
control mechanism along with in-network caching mechanism of CCN ensures
correctness in the detection of complex events.

###  IV-B Architecture Overview

Every CCN node can act either as a _producer_ , a _consumer_ or a _broker_. A
CCN node here refers to a system entity that complies to the CCN protocol as
explained in Section II. A CCN broker is an in-network element, i.e., an
INetCEP aware CCN router, while a producer or consumer is an end device, e.g.,
a sensor or a mobile device. On the one hand, consumers can request a specific
data item using an Interest packet, where brokers forward the request received
by consumers to support request-reply interaction, as illustrated in Figure 1
on the left. The producer replies with a data object contained in a Data
packet. On the other hand, brokers process an unbounded and ordered _data
streams_ generated by producers, as illustrated in Figure 1 on the right and
explained below.

A producer multicasts the data stream (Data Stream packet) towards the broker
network, which disseminates the stream through the network towards consumers
that expressed their interest through Add Continuous Interest. A consumer
issues a query by sending an Interest or an Add Continuous Interest packet
comprising a query $q$ (top right Figure 1) identified by a $qname$. Depending
on the packet, the consumer receives the query result once or continuously.
The CEP query $q$ is processed by interconnected brokers forming a _broker
network_. The $qname$ is stored in the PIT just like the Interest name on the
receiving broker until Remove Continuous Interest packet is received that
triggers the removal of $qname$ or Continuous Interest from the PIT (bottom
right Figure 1).

The query $q$ induces a directed acyclic operator graph, where a vertex is an
operator $\omega\in\Omega$ and an edge represents the data flow of the data
stream $D$. Each operator $\omega$ dictates a processing logic $f_{\omega}$.
In the following, we explain the unified communication model, the operator
graph, and the quality of information model.

To achieve flow control, we model a flow from a CCN node defined as a uni-
directional data transfer from one node to another with feedback on the
opposite direction on the same route. Flows are considered to be independent
of each other and they can change dynamically. Furthermore, packets in a flow
are assumed to arrive in order (FIFO).

####  IV-B1 Unified Communication Model

The basic packet structure of a CCN packet includes a fixed-size header whose
size depends on the packet type, a variable length data object name, the
optional name’s TLV (type length value format), some optional TLVs, and
finally the payload555For more information, we refer the reader to the IETF
draft https://tools.ietf.org/html/draft-ccn-packet-header-00..

We provide five types of packets in CCN to support both kinds of interaction
patterns. Interest (_request_) packet is equivalent to CCN’s Interest packet
that is used by the consumer to specify interest in the name of a data object
or an operator ($\omega$). The Data (_reply_) packet is a CCN data packet that
satisfies an Interest or Add Continuous Interest detailed below. This packet
is similar to the Interest with additional payload that contains the content.
The Data Stream packet represents a continuous time series data stream of the
form $<ts,a_{1},\ldots,a_{m}>$. Here, $ts$ is the time at which a tuple is
generated and $a_{i}$ are the attributes of the tuple. It is similar to the
Data packet in terms of structure except the packet type and the payload
comprising of the data stream. The Add Continuous Interest packet represents
the interest in the form of a CEP query $q$. The structure is similar to
Interest packet and the query is encapsulated in the name. The Remove
Continuous Interest packet represents the CEP query that should to be removed
for the respective consumer, so that it no longer receives complex events. The
CCN forwarding or data plane is enhanced to handle these packets. We summarize
the differences to classical ICN architectures in Table I.

####  IV-B2 Quality of Information Model

One of our main objectives while unifying the communication strategies of ICN
is to minimize the delivery response time of the complex events, while
maintaining high throughput of events. On top of that, there must not be any
drop of events when delivering with short response time, which might result in
wrong results. We measure the drop of events or loss rate in forwarding as
$\frac{(\texttt{total events}-\texttt{processed events})}{\texttt{total
events}}$. Here, total events is the number of events that were produced by
the producers and should have been received at the consumers end. Processed
events is the number of events that were eventually received at the consumer.

Of course, operators like window and filter would eventually reduce the output
number of events based on the semantics of the operator. Thus, to measure the
loss for complex events, we refer to the _accuracy_ metric. It is defined as
the proximity of the derived complex events to the true complex events as a
result of false positives and false negatives. As a consequence of event
drops, the derived complex events could be false positives or false negatives.
On the one hand, a false positive (FP) is defined as a complex event that
should have been derived but is not or is falsely derived, while a false
negative (FN) is an complex event that is derived when it should not have been
derived. On the other hand, the true positives (TP) and true negatives (TN)
are correctly derived complex events, i.e., correct load predictions and not
detected, respectively.

Characteristic | Old Architecture | Our Architecture
---|---|---
ICN |  Description |  INetCEP |  Description
Packet Types (cf. § IV-C2) | Interest |  Interest packet to express a request on a data object |  Interest |  Interest packet to express a request on a data object
Data |  Data packet to satisfy a request |  Data |  Data packet to satisfy a request by an Interest or Add Continuous Interest
- |  - |  Data Stream |  Data Stream packet representing continuous time series data
- |  - |  Add Continuous Interest |  Continuous Interest packet to express a continuous interest in the data object
- |  - |  Remove Continuous Interest |  Remove Continuous Interest packet to remove interest on a data object
Data Structures (cf. § IV-C1) | PIT |  Stores pending interests of consumer |  PIT |  Stores pending interests and continuous interests
CS |  Stores data objects |  CS |  Stores data objects and buffers the data stream for stateful operators like window.
FIB |  Stores forwarding information towards producers |  FIB |  Stores forwarding information towards producers and consumers interested in CEP query
Data Processing (cf. § IV-D) | - |  - |  CEP engine |  Parse, place, process and derive complex events
TABLE I: Description of differences in traditional ICN vs INetCEP architecture
(”-” means no support).

###  IV-C Unified Communication Approach

In this section, we explain the extension of the CCN data plane to enable
coexistence of consumer and producer initiated communication. In our approach,
each CCN node $n\in N$ maintains a Content Store or cache (CS), a Pending
Interest Table (PIT), a Forwarding Information Base (FIB) and a CEP engine. In
the following, we explain the function of these main building blocks and the
data plane handling. Subsequently, we provide details on the newly introduced
packets Add Continuous Interest, Remove Continuous Interest and Data Stream.

####  IV-C1 Node Components

The CS stores all the data objects associated with the CCN Interest and the
time-stamped data objects associated with the Add Continuous Interest. The
data object either returns the result of an Interest or a CEP query $q$ in the
Interest or Add Continuous Interest packet. For instance, if the query is the
sum of victims in a disaster location, then the data object contains a value,
e.g., 20. Hence, if multiple consumers are interested in the same $q$, the
query is not reprocessed but the data is fetched directly from the CS. To
ensure that a fresh data object is returned from the cache once a Add
Continuous Interest is registered and new Data Stream packet is received the
query processing is triggered reactively. The reactive handling of the Data
Stream ensures that always up to-date data objects are stored in the cache,
while old entries are discarded (left cf. Figure 1).

The PIT stores the pending Interest name with $q$ so that the _complex event_
($ce$) could follow the path, i.e., the in-network state created in PIT to the
consumer. The $\mathit{face}$ information is also stored in the PIT entry to
keep track of consumers interested in $qname$. In contrast to consumer-
initiated interaction, the $ce$ must be notified to the interested consumers
as and when detected in real-time. Thus, as new data is received, the _qname_
(name prefix of the query) in the PIT are re-evaluated as explained later in
Algorithm 1. In the former, we are referring to only continuous Data Stream
packet since we also support fetching discrete data from producers (this is
handled similarly using a CCN Interest packet as explained in Section II).

The reasons why we distinguish between Add Continuous Interest ($qname$) and
CCN Interest packets are: (i) $qname$ is invoked on receipt of Add Continuous
Interest as well as the Data Stream packet, (ii) removal of the PIT entry is
not based on a Data packet retrieval but on the reception of the Remove
Continuous Interest packet, and (iii) $qname$ retrieves Data packets
asynchronously. In summary, by asynchronously handling the $qname$ instead by
3-way message exchange and efficiently managing PIT entries, respectively. We
store $qname$ in PIT and deliver complex events to the consumers when a new
data object is received in the Data Stream packet.

The FIB table is populated as the producer multicasts to the broker network
leaving a trail to the data source. Finally, the CEP engine holds the
processing logic $f_{\omega}$ for each operator $\omega$ and is responsible
for parsing, placement, processing, and returning the result to the next node
towards consumer (cf. Section IV-D).

####  IV-C2 Data Plane Handling

In Algorithm 1 (lines 1-1) and Figure 2, we define the handling of Add
Continuous Interest and Data Stream packets at the broker in a CCN network. In
INetCEP algorithms, we express _events_ , _packets_ , and their _handlers_
using the definition of the asynchronous event-based composition model [11].
Hence, events and packets are denoted as: $\langle\textit{Event or Packet
Type}|\textit{Attributes,...}\rangle$.

When an Add Continuous Interest arrives, the broker checks if the (up to-date)
data object corresponding to the $qname$ already exists in the CS. If this is
true, the broker forwards the data object to the consumer. In addition, the
broker checks the PIT for an entry of the face, in contrast to the handling of
a normal Interest packet for two reasons. (1) The data object found could
correspond to the Interest packet instead of an Add Continuous Interest so
there should be an entry related to the Add Continuous Interest packet in PIT.
It is important to note that the PIT entry has a flag indicating that the
entry is related to Add Continuous Interest or Interest packet. (2) The data
object found could be from another consumer, so still the face of the new
consumer needs to be added to the PIT (lines 1-1). If a PIT entry is not
found, then the broker discards the packet. This is because the $qname$ is
already processed at one or more brokers.

Figure 2: High level view of packet handling in INetCEP architecture.

If the cache entry is not found, the broker continues its search in the PIT
table (lines 1-1). If $qname$ is found in PIT and the face corresponding to
the query interest does not exist (lines 1-1), a new ${face\\_id}$ (from which
the interest is received) is added. Conversely, if the face entry is found in
PIT, this means the $qname$ is being processed and hence the packet is
discarded (lines 1-1). However, if no entry in PIT exists, this means that the
consumer’s interest reaches first time at the broker network. Therefore, a new
entry for $qname$ is created in the PIT and the Add Continuous Interest packet
is forwarded to the face found in the FIB (lines 1-1).

A Data Stream packet is also handled similarly to the Data packet (lines 1-1),
except for the fact that the query processing is triggered if Data Stream
satisfies a query interest in PIT and it is a new packet (lines 1-1). This
means that the $qname$ performs an operation on the data object contained in
the Data Stream packet. In this case, query processing is triggered because it
may contribute to the generation of a new $ce$. In addition, if the broker
does not have a matching $qname$ in PIT entry, this means it is not allocated
to operator graph processing and hence it is forwarded to the next broker
(line 1). The Data Stream is forwarded if there are consumers downstream by
looking at the FIB entry.

When a Remove Continuous Interest packet is received at a broker, the node
looks up its PIT table for an entry of the $qname$. If found, it removes the
PIT entry for $qname$ and the Remove Continuous Interest packet is forwarded
to the next node. It is done in a similar way as the PIT entry corresponding
to a CCN Interest packet is removed when a matching Data packet is found.

Variables : $\mathit{CS}\leftarrow\text{content store of current node}$

$\mathit{PIT}\leftarrow\text{pending interest table of current node}$

$\mathit{FIB}\leftarrow\text{forwarding information base}$

$\mathit{qname}\leftarrow\text{name prefix including query}$

$\mathit{result}\leftarrow\text{query result}$

$\mathit{facelist}\leftarrow\text{list of all faces in PIT}$

$\mathit{ds}\leftarrow\text{payload of {Data Stream} packet}$

$\mathit{data}\leftarrow\text{data object that resolves the qname}$

$\mathit{ts}\leftarrow\text{timestamp in event tuple {Data Stream}}$

1 upon  _ receipt  $\langle$ Add Continuous Interest $|qname\rangle$ packet _
do

2

3 if  _ $qname$ is found in $\mathit{CS.\textsc{lookup}()}$_ then

4 $\mathit{data}\leftarrow\mathit{CS}.\textsc{fetchContent}(\mathit{qname})$;

5 if  _ $qname\text{ is found in }\mathit{PIT.\textsc{lookup}()}$_ then

6 $\textsc{ProcessPIT}(\mathit{qname},\texttt{Add Continuous Interest})$;

7

8 $\text{return }\mathit{data}$;

9 (Discard Add Continuous Interest)

10 else if  _ $qname\text{ is found in }\mathit{PIT.\textsc{lookup}()}$_ then

11 $\textsc{ProcessPIT}(\mathit{qname},\texttt{Add Continuous Interest})$;

12

13 else if  _ $qname\text{ is found in }\mathit{FIB.\textsc{lookup}()}$_ then

14 $\mathit{PIT}.\textsc{addFace}(\mathit{qname})$;

15 Forward Add Continuous Interest ;

16

17 else

18 Discard Add Continuous Interest ;

19

20

21

22upon  _ receipt  $\langle$ Data Stream $|ds\rangle$ packet _ do

23 for each  _ $\mathit{qname}\in PIT$ _ do

24 if  _ ds satisfies  $qname$ _ then

25 $\textsc{ProcessPIT}(\mathit{qname},\texttt{Data Stream})$ ;

26

27 else

28 Forward DataStream;

29

30

31

32function  _$\textsc{ProcessPIT}(\mathit{qname},\mathit{packet})$_

33 if  _$\mathit{packet}\text{ is }\texttt{DataStream}\text{ and
}\mathit{packet}.\mathit{ts} >\mathit{qname.ts}$_ then

34 trigger _ $\langle\mathit{deployAndProcess}|qname\rangle$ _ (Refer Algo.
2);

35 Forward $\mathit{packet}$;

36

37 else

38
$\mathit{facelist}\leftarrow\mathit{PIT}.\textsc{getFaces}(\mathit{qname})$;

39 if  _ $qname.\mathit{face}$ is not found in $\mathit{facelist}$_ then

40 $\mathit{PIT}.\textsc{addFace}(\mathit{qname})$;

41

42 Discard $\mathit{packet}$;

43

44

45

Algorithm 1 Add Continuous Interest and Data Stream packet handling.

To summarize, in Table I, we show the differences of the INetCEP architecture
in comparison to the standard CCN architecture in terms of the packet types,
the data plane, and the processing engine. We show that with minimum changes
in the data plane, we support both _consumer-_ and _producer-_ initiated
traffic.

####  IV-C3 Rate-based Flow Control

Flow imbalance is a well-known problem also in IP-based _push_ approaches. The
link capacity or even the nodes processing capacity can be a bottleneck in
forwarding data streams which can cause heavy packet loss and in the worst
case can disrupt the system completely. CEP systems like Flink deal with this
issue using a credit based flow control mechanism 666A deep dive into Flink´s
network stack. https://flink.apache.org/2019/06/05/flink-network-stack.html
[Accessed on 31.10.2020]., although it has many similarities to rate-based
flow control, it is often regarded as lossy [30]. Rate-based flow control
mechanisms are lossless as they avoid buffer overflow by always keeping the
rate below or equal to what the network can offer [17]. In the context of
INetCEP, when continuous data streams are to be received, we propose a
distributed algorithm for rate-based flow control. Each producer maintains an
estimate of its optimal event rate. Initially, the producer uses its desired
sending event rate as an estimate and eventually updates this estimates based
on the periodic sending of management packets. The management packet contains
two fields, the first field is called the _underloading_ bit or in short _u-
bit_. The second field is used to contain the next event rate estimate for the
producer, known as the _stamped rate_.

Before sending the management packet, the producer puts its current rate
estimate in the stamped rate field, and clears the u-bit. If the demand of a
flow is below the allocation it receives from the network in the management
packet, the source sets the u-bit of the outgoing management packet to 1. Each
CCN router / producer monitors its event traffic and calculates its available
capacity per flow. This quantity is referred to as _advertised rate_. When a
management packet arrives, the CCN router compares the stamped rate with the
advertised rate. If the stamped rate is greater or equal to the advertised
rate, the stamped rate is set to the advertised rate and the u-bit is set in
the management packet. Otherwise, the CCN router does not change the fields of
the management packet.

When the rate control packet reaches the destination, i.e., the consumer, the
stamped rate contains the minimum of the producer’s rate estimate (at the time
the management packet was sent) and all rates that flow is allowed to have by
the CCN routers in its route. The consumer sends the management packet back to
the producer. After a full round trip of the network, the setting of the u-bit
indicates whether the flow is constrained along the data path. In other words,
the rate is limited by some CCN router in the path and cannot be increased. In
this case, the producer adjusts the stamped rate of its outgoing management
packets to the stamped rate of its incoming management packet. If the u-bit is
clear, the producer stamped rate is increased to the advertised rate value.

The advertised rate is computed as follows. The CCN routers maintain a list of
all seen stamped rates, referred to as recorded rates. The set of flows whose
recorded rate is higher than the advertised rate, are considered as
unrestricted flows and are denoted as $\mathcal{U}$. Similarly, the flows with
recorded rate below the advertised rate are regarded as restricted flows or
$\mathcal{R}$. The flows in restricted rate set $\mathcal{R}$ are assumed to
be the bottleneck links at the same or another CCN router. While the flows in
the unrestricted set $\mathcal{U}$ are those for which a restricted rate has
not been computed yet on this CCN router. Each CCN router, on receiving a
management packet from a flow which is currently unrestricted, will compute a
new stamped rate for this flow, under the assumption that this CCN router is a
bottleneck for this flow. This will cause the CCN router to recompute its
advertised rate as described and to insert a new rate into the stamped rate.
Given sets $\mathcal{U}$ and $\mathcal{R}$, the advertised rate ($\mu$) is
calculated as the link capacity not used by flows in $\mathcal{R}$ available
per flow in $\mathcal{U}$.

$\mu=\frac{\mathcal{C}-\mathcal{C}_{\mathcal{R}}}{n-n_{\mathcal{R}}}$ (1)

Here, $\mathcal{C}$ is the total capacity of the link,
$\mathcal{C}_{\mathcal{R}}$ is the total capacity consumed by all restricted
flows, $n=f+kb$ and $n_{\mathcal{R}}=f_{\mathcal{R}}+kb_{\mathcal{R}}$ with
$f,b,f_{\mathcal{R}}$ and $b_{\mathcal{R}}$ being the number of total and
restricted forward and feedback flows traversing the link respectively. For
$k=0$ and $k=1$, $n$ and $n_{\mathcal{R}}$ are the total number of flows and
the number and the number of restricted flows traversing the link.

Thus, by controlling the flow of events using the estimates provided by the
rate-based flow control algorithm we expect timely processing of events
without any event loss.

###  IV-D CEP Query Engine in the Data Plane

In this section, we present the proposed meta query language for CEP to
specify queries in CCN data plane and the query deployment process. We have
three main design goals for the meta query language and the query parser:  (1)
distinguishing between pull and push based traffic,  (2) translating a query
to an equivalent name prefix of the CCN architecture, and  (3) supporting
conventional relational algebraic operators and being extensible such that
additional operators can be integrated with minimum changes. This is to ensure
easy integration of existing and new IoT applications.  We provide the
definition of INetCEP query language in Section IV-D1 and the query engine in
Section IV-D2.

####  IV-D1 Meta Query Language for INetCEP

Each operator in a query behaves differently based on the input source type,
i.e., consumer- and producer-initiated interaction, which is done based on the
reception of a Data packet or a Data Stream packet, respectively. The Data
packet is processed and returned as a data object, as conventionally done in
the CCN architecture. For instance, a Join operator placed on broker $b$ can
join two data objects, $<lat1,long1>$ with name prefix $/node/node1$ and
$<lat2,long2>$ with $/node/node2$ to produce $<lat1,long1,lat2,long2>$ with
$/node/node3$. In contrast, a Data Stream is processed and transformed either
into an unbounded output stream (another Data Stream packet), or can be
transformed to derive a Data packet, containing, e.g., a boolean variable
depending on the CEP query.

We express the standard CEP operators as explained in Section II using the
INetCEP language below.

⬇

1WINDOW(GPS_S1, 4s)

⬇

1FILTER(WINDOW(GPS_S1, 4s),’latitude’<50)

⬇

1JOIN(

2 FILTER(WINDOW(GPS_S1, 4s), ’latitude’<50),

3 FILTER(WINDOW(GPS_S2, 4s), ’latitude’<50),

4 GPS_S1.’ts’ = GPS_S2.’ts’

5)

The stateful operators, e.g., Window and JOIN must store the accumulated
tuples in some form of a readily available storage. For this, we make use of
in-network cache, the CS, that readily provides data for the window operator.
This can be highly beneficial, e.g., in a dynamic environment where state
migration is necessary. For a complete discussion on the grammar of the
language and its design, we refer the reader to our previous work [31].

Variables : $query\leftarrow$ the input CEP query

$\tau\mathit{curList}\leftarrow$ top down list of 3 $\omega$ of tuple $\tau$

$\omega_{cur}\leftarrow$ current operator

$planNode\leftarrow$ logical plan node of G

$curNode\leftarrow$ current node

$allNodes\leftarrow$ list of all nodes

$paths\leftarrow$ list of possible paths in the topology

$opPath\leftarrow$ optimal path based on the objective

1

2upon  _ event $\langle\mathit{deployAndProcess}|qname\rangle$ _ do

// Query parsing

3 if  _ $query\in qname$ is new_ then

4 $\tau\mathit{curList}\leftarrow$ getCurList($query$);

5 $curNode\leftarrow$ parseQuery($\tau\mathit{curList})$;

6

// Query placement

7 if  _$query\in qname\text{ is not placed or }query\text{ must be replaced}$_
then

8 $allNodes\leftarrow\textsc{getNodeStatus}(curNode)$;

9 $paths\leftarrow\textsc{buildPaths}(curNode,allNodes)$;

10 $opPath\leftarrow\textsc{findOptimalPath}(paths,allNodes)$;

11 $\textsc{deployOperators}(optimalPath,allNodes)$;

12

// Query processing

13 for all  _$\omega_{cur}\in\tau\mathit{curList}$_ do in parallel

14 $\textsc{processOperator}(\omega_{cur},optimalPath)$;

15

16

17

18function  _$\textsc{parseQuery}(\tau\mathit{curList})$_

19 $\omega_{cur}\leftarrow$getOperator$(\tau\mathit{curList})$;

20 $\mathit{nfnExp}\leftarrow\textsc{constructNFNQuery}(\omega_{cur})$;

21 $planNode\leftarrow\text{new }\textsc{planNode}(\mathit{nfnExp})$;

22

23 if  _$size(\tau\mathit{curList})==1$_ then

24 return $node$;

25

26 else if  _$size(\tau\mathit{curList}) >1$_ then

27

28 parseQuery($\tau\mathit{curList}.left)$;

29 parseQuery($\tau\mathit{curList}.right)$;

30 return $planNode$;

31

32

33

Algorithm 2 Operator tree creation, placement and processing

####  IV-D2 Query Deployment

Given the constructs of the query language, we now explain how the queries are
processed in the CCN data plane. In INetCEP, the queries can be processed in a
centralized or a distributed fashion for better scalability. For instance,
when the amount of resources required to process the queries exceed (that can
easily happen in case of stateful operators like joins). Moreover, the core
in-network operations should not be disturbed by the execution of CEP
operators.

As shown in Figure 2, Add Continuous Interest and Data Stream packets invokes
the query engine and hence parsing, placement and processing of the queries in
the form of an operator graph. In Algorithm 2, we show this process. The query
parser parses and translates the query into an executable form. Furthermore,
it verifies the correctness of the query by validating the parameters. For
example, a join can be on a set of data streams and not a single data stream,
so it needs atleast two parameters. When a new query encapsulated in the name
field of Add Continuous Interest packet is received on a node, the recursive
function parseQuery performs the aforementioned checks (lines 2-2). It is
important to note that this function is invoked only when a new query is
received. This way, we provide query reuse, an important functionality of a
CEP system needed when same query is invoked from multiple consumers [38].

We express the INetCEP query parser as a recursive algorithm to map the query
in order to generate an equivalent NFN’s _ $\lambda$ expression_ which is
processed as follows. A CEP query is transformed into an operator graph $G$
(lines 2-2), which is an acyclic tree defined as a tuple $\tau=(L,S,R)$. Here,
$L$ (left sub tree) and $R$ (right sub tree) are binary trees or an empty set
and $S$ is a singleton set, e.g., a single operator ($\omega$). The query
parser starts parsing the query in a specific order, i.e., in a top-down
fashion that marks the dependency of operators as well. This implies each leaf
operator is dependent on its parent. Thus, the parser starts by iterating top
down the binary tree starting from the root operator $\omega_{cur}$ (line 2),
where $cur=root$ in the first step. The traversal is performed in a depth-
first pre-order manner (visit parent first, then left (L) and then right (R)
children) (lines 2-2).

To determine the query placement, we make use of Interest packets with a flag
indicating that they are for management (or internal) use. Furthermore, we
make use of a node status service that is probed to get the information on the
nodes as well as the links between them. This can run, for example, on a
monitoring server that collects information on the network. This information
includes distinct quality of service (QoS) metrics widely used for operator
placement [32] such as response times, bandwidth, energy information (crucial
for IoT), and finally, we also include the rate estimate from the rate-based
flow control mechanism (discussed in Section IV-C3). The response $r_{p}$ is
encapsulated in the payload of Data packet and is of the form
$start|b_{i}|end$ = $r_{p}|{b}_{p}|{qos}_{p_{i}}|\mu_{p}$, where $start$ is
the originating node of the path, $b_{i}$ are the intermediate nodes or the
brokers and $end$ is the end node of the path. On the right hand side, we have
the end to end response time (${r}_{p}$), bandwidth (${b}_{p}$), $qos_{p_{i}}$
for other QoS metrics and $\mu_{p}$ is the rate estimate of this flow in the
path $p$.

Hence, when a Add Continuous Interest encapsulating a query is received on a
node, it probes the decentralized service for its neighbor nodes status by
means of an Interest packet with name-prefix _node/nodeStatus_. The node
status service sends the information $r_{p}$ for each path including its
neighboring nodes. The placement coordinator (receiver of the query) then
depicts the optimal path where the query $q$ is deployed (Line 2–2). It is
important to note that the query placement is trigerred in the beginning for
each query and when a replacement request is received, for example, due to the
dynamic changes in the IoT network. This further accelerates the query
processing in the data plane.

We have now the formulated operator graph and the way it is placed on the
distributed nodes. However, the data stream still only reaches the node where
the query was triggered. The data stream and the intermediate events must also
flow down the operator graph to derive the complex event. For this purpose,
the so-called _sub_ continuous interests are sent down the optimal data path
such that the pending interest tables are updated for all nodes on the path.
Finally, the operator graph is processed in an asynchronous parallel manner as
seen in line 2–2.

Experiment time $t_{s}$ |  20 min
---|---
Warmup time $t_{w}$ |  60 s
Number of runs |  10
Topologies |  Manhattan graph (WSN), line and tree (edge-cloud) topology
Number of producers |  1, 2
Number of consumers |  1
Number of brokers |  5 (Manhattan), 3 (line), 8 (tree)
Queries |  Window, Filter, Join, Heatmap, Predict (cf Query LABEL:query:window–LABEL:query:predict)
Number of cloud instances for distributed setup |  1 GC c2-standard-8 and 7 AWS c5d.2xlarge instances
Input event rate |  1, 1000, 10,000 and 50,000 (Poisson distribution)
ICN and CEP systems |  INetCEP_UCL (ours), Apache Flink [13], Periodic Request (PR)  [43]
TABLE II: Configuration parameters for the evaluation. Default or commonly
used parameters are underlined.

(a) Single node with producer, consumer, and broker communicating via unified
communication layer. The metrics are annotated as they are measured.

(b) The different topologies Manhattan graph, line, and tree topology
representing the edge-cloud network (left to right).

Figure 3: Single and distributed nodes setup.

##  V Evaluation

We evaluate INetCEP by answering three questions:

EQ1: Can the unified communication approach support both consumer- and
producer-initiated communication and match the performance of state-of-the-art
CEP systems?

EQ2: What is the performance of INetCEP when the queries are processed on a
single node?

EQ3: What is the performance of INetCEP when the queries are processed on
distributed nodes in an edge / data-center topology?

To this end, we explain the evaluation setup in Section V-A, and answer EQ1 in
Section V-B, EQ2 in Section V-C1, and EQ3 in Section V-C2.

###  V-A Evaluation Environment

Implementation:  We selected the NFN architecture [48] to implement our
solution, due to its built-in support of resolving _named functions_ as so-
called $\lambda$ expressions on top of the ICN substrate. However, currently
NFN’s communication plane is purely consumer-initiated and there is no support
for CEP. To this end, we provide a unified communication layer for co-existing
consumer- and producer-initiated interactions, while doing CEP operations in
the network, on top of NFN. Therefore, we embedded CEP operators as _named
functions_ while leveraging NFN’s abstract machine to resolve them. NFN works
together with CCN-lite [3], which is a lightweight implementation of CCNx and
the NDN protocol. We developed unified interfaces of our design on top of NFN
(v0.2.1) and CCN-lite (v2.0.1) for the Linux platform [4]. We implemented
13,454 LOCs and 19,491 LOCs in CCN-lite and NFN-Scala, respectively.

We enhanced the NDN protocol implementation in the CCN-lite and the NFN
architecture by  (1) including the additional packet types and their handling,
as described in Section IV-C,  (2) implementing the extensible general CEP
query language, parser, and CEP operators as NFN services, as described in
Section IV-D, and  (3) implementing the parsing, placement and processing of
operator graphs in the data plane of CCN (cf. Section IV-D2).

Evaluation Platform:  We evaluated our implementation both using CCN-lite as
well as the Common Open Research Emulator (CORE) network emulator and MACI
[19] for distributed node evaluation in the cloud. We used the CORE emulator
in order to include different topologies like line and tree. We used virtual
machines from Google Cloud and Amazon Web Services (AWS) to introduce
heterogeneity in consistent to the IoT scenario. In Google Cloud we used the
c2-standard-8 machine with 8vCPUs, processing speed of upto 3.8 Ghz, and 32
GiB memory. Using AWS we chose instances of the type c5d.2xlarge with 8 vCPUs,
processing speed of upto 3.5 Ghz, and 16 GiB memory. All the instances run a
Ubuntu 18.04 server. Here, each VM is a CCN-NFN node, which hosts a NFN
compute server encapsulating the CEP operator logic. To run the experiments,
we first used a _Manhattan graph_ , a well-known WSN topology [8], and a tree
topology resembling edge and data-center networks with varying number of nodes
as illustrated in Figure 3. We deployed the INetCEP architecture that executes
on the NFN compute server, the CCN-NFN relay, as well as on the links. Here,
as intended, the nodes communicate using the NDN protocol instead of IP.

Datasets. We extended the INetCEP architecture to the two representative IoT
introduced in Section III, namely the load prediction query and the heat map
query. We extended the INetCEP query language and CEP operators to include the
heat map [26] and prediction operators [33] by making a few additions to our
implementation in extensible query language and parser [31].

Dataset 1. For the heat map query, we use a dataset [7] of a field test
mimicking a post-disaster situation. The collected dataset consists of sensor
data, e.g., location coordinates. Each sensor data stream has a schema
specifying the name of the attributes, e.g., the GPS data stream has the
following schema:
$<ts,s\\_id,latitude,longitude,altitude,accuracy,distance,speed>$

Query. We use the $latitude$ and $longitude$ attributes of this schema to
generate the heat map distribution of the survivors from the disaster field
scenario. A typical heat map application joins the GPS data stream from a
given set of survivors, derives the area by finding minimum and maximum
latitude and longitude values, and visualizes the heat map distribution of the
location of the survivors in this area. For simplicity, we consider one data
stream from one survivor. Here, GPS_S1 is the producer of GPS sensor, HEATMAP
operator represents the heat map generation algorithm [26].

(a) $\bar{x}(Flink)=792.19\mu s$, $\bar{x}(PR)=41.82\mu s$,
$\bar{x}(UCL)=38.77\mu s$.  (b) $\bar{x}(Flink)=982.16\mu s$,
$\bar{x}(PR)=39.06\mu s$, $\bar{x}(UCL)=38.20\mu s$.  (c)
$\bar{x}(Flink)=1162.53\mu s$, $\bar{x}(PR)=50.38\mu s$,
$\bar{x}(UCL)=73.57\mu s$.

Figure 4: Cumulative distribution function of end-to-end delay for Unified
Communication (UCL), Flink and Periodic Request (PR) approach. Our approach is
$20,25$ and $15\times$ faster in forwarding than Flink and mostly better than
the flooding (PR) approach for an event rate of 1000, 10,000 and 50,000 events
per second (left to right), respectively.

⬇

1HEATMAP(

2 ’cell_size’, ’area’,

3 WINDOW(GPS_S1, 4s)

4)

Query 1: Display the heat map distribution of GPS source 1 in the given area
with a given cell size.

Dataset 2. The second dataset comes from the 2014 DEBS Grand Challenge [2]
scenario focused on solving a short-term load forecasting problem in a smart
grid. The data for the challenge is based on real-world profiles collected
from smart home installations. The dataset captures load measurements from
unique smart plugs with the following schema:
$<ts,id,value,property,plug\\_id,household\\_id,house\\_id>$

Query. We apply an existing solution [33] to perform prediction by extending
the INetCEP architecture for smart plugs. In this query, a window of 4s of
plug sensor load is analysed to generate predictions every 30 seconds.

⬇

1PREDICT(30s, WINDOW(PLUG_S1, 4s))

Query 2: Performing a prediction every 30 seconds for 1 minute into the future
on the load observed by plug src 1.

Metrics. The main objective of INetCEP is to minimize response time, better
known as _end-to-end latency_ in the delivery of events, while maintaining a
high throughput. The latency is measured as the total time taken from the
generation of the event at the producer until the complex event reaches the
consumer. The second metric is _throughput_ which is measured as the number of
events received at the consumer compared to the number of produced events. In
an ideal case, the consumer should receive all the events sent by the producer
given that it expresses a continuous interest on all the events from the
producer. Another metric is the loss of events while forwarding or retrieving
the complex event. The _loss rate (%)_ is measured as $\frac{(\text{total
events}-\text{processed events})}{\text{total events}}\times 100$. Finally, we
are interested in the accuracy of the retrieved complex event as a consequence
of loss of primary events. In Figure 3a, we have illustrated the metrics for a
single node setup.

Baseline.  We consider Apache Flink [13], a state-of-the-art CEP system, which
so far has the best performance in terms of latency and throughput in CEP
systems, to be a valid baseline. Flink consumes the continuous data stream
from Apache Kafka, which acts as a data producer. Furthermore, we implemented
a periodic request approach (denoted as INetCEP_PR) equivalent to the
implementation of other work [43] in INetCEP and compare our solution against
it. A summary of the used parameters is provided in Table II.

###  V-B Performance of the Unified Communication Approach

To understand the performance of the unified communication approach (denoted
as INetCEP_UCL), we measure the metrics end-to-end latency, throughput, and
loss in forwarding the events. We consider a single node setup as shown in
Figure 3a, where producer, consumer and broker coexists on the same node. The
producer generates a continuous data stream of a GPS sensor from dataset 1
(above section). For this evaluation, we used one Google Cloud c2 instance.
The consumer places a continuous interest by sending a Add Continuous Interest
packet to the broker and starts receiving the data stream from the broker.
Each evaluation run is for 20 minutes, after which the consumer removes the
continuous interest by sending a Remove Continuous Interest packet. We repeat
the experiments 10 times to collect 12,029 data points for the metrics end-to-
end latency, throughput, and loss rate.

(a) $\bar{x}(Flink)=999.86ev/s$, $\bar{x}(PR)=999.63ev/s$,
$\bar{x}(UCL)=1000.0008ev/s$ (b) $\bar{x}(Flink)=9997.99ev/s$,
$\bar{x}(PR)=9999.65ev/s$, $\bar{x}(UCL)=10000.0008ev/s$ (c)
$\bar{x}(Flink)=50035.28ev/s$, $\bar{x}(PR)=19372.90ev/s$,
$\bar{x}(UCL)=49972.34ev/s$

Figure 5: Cumulative distribution function of throughput for Unified Communication (UCL), Flink, and Periodic Request (PR). Our approach is performing equally in terms of throughput in forwarding events compared to Flink and better than the flooding (PR) approach for an event rate of 1000, 10,000 and 50,000 events per second (left to right), respectively. | 1000 events per second | 10,000 events per second | 50,000 events per second
---|---|---|---
System | mean | max | min | quantiles (90, 95, 99) | mean | max | min | quantiles (90, 95, 99) | mean | max | min | quantiles (90, 95, 99)
Flink | 0.013 | 0.014 | 0.013 | 0.013, 0.014, 0.014 | 0.135 | 0.271 | 0.069 | 0.211, 0.241, 0.265 | 0.0135 | 0.020 | 0.003 | 0.016, 0.018, 0.019
INetCEP_PR | 0.038 | 0.044 | 0.036 | 0.04, 0.042, 0.043 | 0.003 | 0.004 | 0.001 | 0.004, 0.004, 0.004 | 61.185 | 61.226 | 61.153 | 61.216, 61.221, 61.225
TABLE III: Packet loss rate (in %) mean, min, max and quantiles (90, 95, 99)
in forwarding 1000, 10,000 and 50,000 events per second in Flink and PR.
INetCEP do not have suffer any loss thanks to the rate-based flow control and
in-network caching mechanism of INetCEP in ICN.

####  V-B1 Forwarding Latency

In Figure 4, we show the cumulative probability distribution function (CDF) of
the observed forwarding latency between sending at the producer and receiving
at the consumer. As expected, the forwarding of the data stream that happens
in the unified communication layer on the data plane is very fast with a mean
delay of $38.77\mu s$, $38.2\mu s$ and $73.57\mu s$ for an event workload of
1000, 10,000 and 50,000 events per second, respectively (cf. Figure 4a, b and
c left to right). With our approach, we are $20,25$ and $25\times$ better (in
terms of mean delay) for an event rate of 1000, 10,000 and 50,000 events per
second, respectively, than the baseline CEP system Flink due to several
reasons. (1) Flink relies on Apache Kafka as a streaming system or producer,
which inherently uses a _pull-_ based approach. Although it applies
optimizations like aggressive batching of events to be forwarded to the
consumer, it still falls short in forwarding latency because of too many
requests per event. (2) Flink operates in a middleware layer, while we benefit
from line rate speed of the data plane of CCN.

Furthermore, the periodic request (PR) approach, which is a _pull-_ based
implementation on top of CCN-lite, inspired by Shang et al. [43], performs
equally well in terms of latency in receiving the data streams, but on the
cost of event/packet loss, as explained later.

####  V-B2 Forwarding Throughput

In Figure 5, we show the CDF of the observed forwarding throughput for 1000,
10,000, and 50,000 dispatched events per second, respectively (left to right),
from producer to consumer. We use a single node setup as shown in Figure 3a.
As shown in the first plot (a), the three approaches perform equally well. A
step-wise progress of the distribution function results from the discrete
nature of the metric. In the second plot, we observe that both UCL and PR are
near to the optimal throughput of 10,000 events. However, for Flink we see a
divergence from the optimum. This can be explained by the batching mechanism
of Flink and Kafka that forwards the events in batches such that the events do
not arrive in a continuous flow but more in a batch form, which correlates
with the observation of this plot.

Finally, the third plot shows the throughput results for an event rate of
50,000. We see that for the PR approach there is a heavy drop of events. We
explain it as follows. The PR approach requires twice as many messages to
receive the continuous data stream. Thus, for this plot to retrieve 50,000
events per second, it will send 50,000 Interest packets. This simply floods
the network and eventually we see packet drops, when the UDP buffer is full.
This is not happening in Flink or in UCL, since UCL requires only a single Add
Continuous Interest packet to retrieve the continuous stream and uses a rate-
based flow control mechanism. Flink also manages the flow using a credit-based
flow control mechanism.

Figure 6: Cumulative distribution function of end-to-end latency of query processing on a single node compared to periodic request (PR) and Flink. Our approach outperforms both Flink and PR, with lowest $\bar{x}(UCL)=7.3ms$ for the window query and highest $\bar{x}(UCL)=14.48ms$ for the predict query. | window | filter | join | heatmap | predict
---|---|---|---|---|---
System | mean | quantiles (90,95,99) | mean | quantiles (90,95,99) | mean | quantiles (90,95,99) | mean | quantiles (90,95,99) | mean | quantiles (90,95,99)
Flink | 2.97 | 0.15, 0.16, 0.18 | 0.329 | 0.155, 0.17, 0.191 | 10.627 | 20.64, 41.52, 41.53 | 2.098 | 4.09, 4.128, 4.167 | 12.44 | 0.34, 0.36, 0.39
INetCEP_PR | 1.51 | 1.9, 1.96, 2.01 | 1.41 | 1.85, 1.92, 1.97 | 5.54 | 5.94, 5.99, 6.03 | 1.46 | 1.75, 1.85, 1.91 | 1.49 | 1.88, 1.95, 1.99
INetCEP_UCL | 0.007 | 0.009, 0.010, 0.014 | 0.010 | 0.014, 0.017, 0.020 | 0.011 | 0.017, 0.02, 0.025 | 0.010 | 0.013, 0.015, 0.019 | 0.014 | 0.019, 0.02, 0.024
TABLE IV: End-to-end latency in seconds for five queries. INetCEP is
$888\times$ (best case: predict query) and $32\times$ (worse case: filter
query) better compared to the Flink CEP system as well as $503\times$ (best
case: join query) and $141\times$ (worst case: filter query) better compared
to the PR approach.

####  V-B3 Event Loss during Forwarding

To further investigate the packet/event loss during forwarding, we measured
the total loss rate for UCL, PR, and Flink. In Table III, we present the
_mean_ , _min, max_ , and _quantiles (90, 95, 99)_ values of the loss rate (in
%). For a low event rate, we observe a slight loss in events for the PR
approach, which is expected due to the equal number of Interest packets sent
to retrieve the data stream. For a high event rate of 50,000 events per
second, the PR approach has more event loss with a mean up to $61.18\%$, due
to the UDP buffer overflow as well as the flood of Interest and Data packets.

Interestingly, we also observe event loss in Flink with Kafka as the data
producer. Although this contradicts to Kafka being highly resilient, Kafka
depends on many configuration parameters to prevent event loss, which are
potentially conflicting and hardly optimizable777Data loss in Flink with
Kafka. https://ci.apache.org/projects/flink/flink-docs-
stable/dev/connectors/kafka.html [Accessed on 1.11.2020]. With the default
configuration parameters, we experienced that there is event loss even while
using Flink and Kafka, as summarized in Table III.

Nevertheless, our approach showed _no event loss_ at all, thanks to the flow
control mechanism and the in-network caching mechanism of ICN.

###  V-C Performance Evaluation of INetCEP Query Engine

In this section, we evaluate the performance of the INetCEP query engine as
described in Section IV-D. We first evaluate on a single node setup similar to
the section above with the queries (cf. Table II). Next, we provide a
distributed evaluation of query engine, where operator placement is involved
as well.

####  V-C1 Centralized Processing

We measure the performance of the five queries on a single node using the
Google Cloud c2 instance. Here, the producer sends the data stream to the
broker while the consumer registers the query by means of a Add Continuous
Interest packet specifying the queries LABEL:query:window–LABEL:query:predict.
We re-implemented the same queries on the Flink engine and measured its
performance as well. Although in centralized processing, there is only one
possible broker for the placement of the operator, the process of parsing,
deployment and processing is time-consuming. In the PR approach, for each
trigger to a query, a new Interest packet is sent, and the placement is
triggered each time a complex event is generated. For example, in the window
query LABEL:query:window, when a new event arrives, a window of 4s is
released. This means, (i) the Interest packet is sent every time888For the
experiment, we have set the rate at which the consumer sends the Interest
equal to the producer sending rate, but in the real world, this is another
problem with PR approach. a new event arrives, (ii) the query is parsed,
placed, and processed, and (iii) finally, the complex event is released. For
this reason, we see that the performance of the PR approach is worse, which is
a consequence of the pull-based approach.

Impact on Latency. In Figure 6, we show the CDF of end-to-end latency for the
different queries. Similar to the forwarding results, the UCL approach is
$30\times$ better in latency than Flink. For example, the filter query UCL has
a mean delay of $10.72ms$, while Flink has $329.47ms$. Most importantly, for
the join query we see a substantial gain (about $100\times$ better). We
investigated the cause of high latency for query processing in Flink and we
saw that the serialization and deserialization of data stream takes a
significant amount of time. Furthermore, Flink suffers from the drawback of
Kafka’s pull-based approach, although with the batching mechanism Flink is
better than PR in some queries. The PR approach performs better than Flink for
some queries, for example, the heatmap query has a mean delay of $1.4s$ in
comparison to Flink, which has a mean delay of $2.0s$. In essence, we
outperform the PR approach by a substantial factor of $100\times$. In Table
IV, we summarize the mean delay and its quantiles (90, 95, 99). As we have
seen in the previous evaluations, the UCL approach exceeds the performance of
Flink by a high factor, so we will proceed with only the PR approach for our
comparison in the distributed evaluations.

####  V-C2 Distributed Processing

Similar to the single node evaluations, we measure the performance with the
same five queries in the distributed evaluations as well. We use different
topologies and numbers of nodes to investigate the behavior of operator
placement and processing of different queries as well as to show the
applicability of our approach to a wide range of scenarios. As shown in Figure
3b, we will first use the Manhattan graph to measure the performance of our
approach. In the topology, we consider a cloud server, edge switches, and end
nodes to process the queries. We used the AWS instances for the distributed
evaluation with seven nodes, as illustrated in Figure 3b on the left. All AWS
instances run the NFN compute server, ccn-lite relay, and the nodes
communicate using the CCN protocol.

Figure 7: Cumulative distribution function of end to end latency for query processing using an Manhattan graph (top) and tree topology (bottom) with 7 and 10 nodes, respectively.  | window | filter | join | heatmap | predict
---|---|---|---|---|---
System | mean | quantiles (90,95,99) | mean | quantiles (90,95,99) | mean | quantiles (90,95,99) | mean | quantiles (90,95,99) | mean | quantiles (90,95,99)
INetCEP_PR_M | 1.634 | 1.951, 1.98, 6.31 | 1.883 | 2.559, 2.702, 5.354 | 7.424 | 9.646, 9.652, 10.678 | 2.061 | 2.985, 2.991, 6.926 | 2.503 | 3.702, 5.38, 6.939
INetCEP_UCL_M | 0.006 | 0.008, 0.009, 0.012 | 0.011 | 0.014, 0.015, 0.021 | 0.011 | 0.015, 0.017, 0.023 | 0.011 | 0.014, 0.016, 0.020 | 0.018 | 0.027, 0.029, 0.035
INetCEP_PR_T | 1.59 | 1.764, 1.789, 1.886 | 1.542 | 1.664, 1.698, 1.797 | 6.955 | 9.527, 9.56, 9.594 | 1.599 | 1.768, 1.82, 1.943 | 1.556 | 1.704, 1.725, 1.784
INetCEP_UCL_T | 0.007 | 0.009, 0.010, 0.012 | 0.012 | 0.015, 0.017, 0.021 | 0.013 | 0.019, 0.022, 0.030 | 0.012 | 0.015, 0.016, 0.021 | 0.016 | 0.021, 0.023, 0.027
TABLE V: End-to-end latency in seconds for distributed evaluations of Periodic
Request (PR) and Unified Communication Layer (UCL). Table summarizes the
_mean_ and _quantiles (90,95,99)_. Here, INetCEP_PR_M stands for the PR
approach with a Manhattan topology and INetCEP_PR_T stands for PR with a tree
topology. Similar to the single node results, we supersede the PR approach by
a factor of $674\times$ (best case for join query) and $128\times$ (worst case
for filter query).

Manhattan Graph. In the Manhattan graph, the end node $n6$ acts as a producer,
$n5$ acts as a consumer that triggers the query, and nodes $n1,\dots,n4,n7$
acts as brokers, respectively. In Figure 7, we present the CDF on the latency
measurements for INetCEP with UCL as a communication strategy and the query
engine as well as INetCEP with PR as a communication strategy and the query
engine. In the PR approach, similar to the centralized evaluations, the query
parser and placement is invoked each time the complex event is retrieved. In
the top figure, we show the results for the Manhattan graph. Our approach
clearly outperforms the naive pull strategy also in distributed query
processing. It has the lowest mean delay $\bar{x}(UCL)=6.46ms$ for the window
query and the highest $\bar{x}(UCL)=18.08ms$ for the predict query due to its
high complexity [33].

Tree Topology. Next, we repeat the evaluations on the line and tree topology,
as illustrated in Fig. 3b (top and bottom on right) using CORE and MACI to set
up the topology and to run evaluations in multiple nodes. We used AWS cloud
instances as before and repeated the five queries. We observe identical
results for the line and tree topology. Hence, we only present the results of
the tree topology that shows only a slight difference, as shown in Figure 7
(bottom). Our approach is substantially better than the PR approach with a
lowest mean delay of $\bar{x}(UCL)=7.07ms$ for the window query and a highest
$\bar{x}(UCL)=16.65ms$ for the predict query, consistent with the results
before. In Table V, we summarize the mean and quantiles (90, 95, 99) for the
delay observed for the two topologies. While the query processing results are
similar, we expect the placement time to be different in the two topologies,
as evaluated below.

Figure 8: A high number of operators takes more time for placement in a high
number of nodes, but still in the order of a few milliseconds thanks to
parallel processing. Figure 9: End-to-end latency of query processing in the
tree topology with 10 nodes. Comparing application level queries with hybrid
(denoted as ”H_Q”, where $Q\in\\{window,filter,join,heatmap,predict\\}$) –
with a window operator deployed and processed in the network layer.

Placement Time. We now analyze the placement time for the different
topologies. It is defined as the time required to find an optimal path for the
deployment of operators. We consider the Manhattan graph with 7 nodes and the
tree topology with 10 nodes similar to the query processing evaluations. In
Figure 8, we present a point plot of the placement time with the median
represented by the point and 95% confidence interval represented by error bars
for the five queries. As expected, due to high number of nodes and the paths
to be discovered, the time to place operators in the tree topology is higher
than in the Manhattan graph, but there are higher confidence intervals in the
Manhattan graph in contrast to the tree topology. Further investigations
indicate that when farther away nodes are used for placement, it took longer
to place than with nearer nodes. Furthermore, the tree topology was evaluated
using the CORE emulator where everything ran on a single node with
virtualization for the 10 nodes using linux namespaces. Hence, due to no real
network latencies, there was no or a minimum difference in the placement time,
as indicated by the negligible confidence interval for the tree topology.

Another interesting observation is the time taken to place join query. In the
tree topology, it takes almost triple the time in comparison to the Manhattan
graph. This is because a higher number of operators is involved in a join
query (6 in total) than in other queries. The time to discover paths in the
tree topology is also higher than in the Manhattan graph and hence we see a
larger difference in placement time with join queries than for other queries.

Hybrid Processing. To further benefit from the performance of the data plane
of CCN, we implemented a window operator in the CCN-lite implementation of a
CCN router (network layer). While all the other operators executes on a NFN
compute server, we investigated the performance of the five queries when the
window operator executes on a CCN router. In Figure 9, we present the results
in a box plot for the end-to-end latency of queries running completely in the
application layer and partially running in the network layer (or hybrid and
denoted as H_Q). A box plot represents a five number summary of the latency
measurements, minimum, first quartile (Q1), median, third quartile (Q3), and
maximum. The vertical line in between sometimes appearing close to Q1 is the
median, we also report the median values written as an overlay in the figure.
The left bound represents the minimum, the start of the box plot Q1, the end
represents Q3 and the right bound represents the maximum. We have excluded the
outliers from the boxplot.

Hence, we clearly observe the benefit of moving the query operation from the
application layer to the network layer, as seen in the plot the hybrid queries
always outperforms the queries running at the application level. This is due
to the round trip to the NFN compute server. The minimum median is for the
hybrid window query $4ms$, which completely runs on top of a CCN router.

####  V-C3 Impact on Accuracy

In this paragraph, we evaluate the impact of event loss on the accuracy in the
delivery of the complex events. As indicated in the smart plug example for
power grid providers in Section III, such industrial IoT applications need to
be highly robust and hence cannot tolerate failures. Event loss can result in
false negatives and false positives in complex events, which eventually can
cause heavy loss if not complete failure. For this reason, we measure the
accuracy in the delivery of complex events for the five queries. We measure it
in terms of the F1-score that better represents accuracy in the results. In
Table VI, we present the F1-score of the two queries heatmap and predict that
are more prone to errors and are implicitly complex. As a consequence of event
loss, the PR approach either misses or leads to many false positives and
negatives, hence we see a drop in accuracy. Similar observation is seen for
Flink since it suffers from data loss as well. In contrast, INetCEP does not
suffer from any event loss and hence correctly identifies all the complex
events with a F1-score of 100%, thanks to the lossless flow control mechanism
in combination with the placement strategy.

| heatmap | predict
---|---|---
System | mean | quantiles (90,95,99) | mean | quantiles (90,95,99)
| Centralized,
---
INetCEP_PR
54.36 | 69.80, 70.90, 71.78 | 48.70 | 46.62, 51.88, 56.09
| Distributed,
---
INetCEP_PR
66.66 | 66.66, 66.66, 66.66 | 66.66 | 66.66, 66.66, 66.66
TABLE VI: Accuracy in complex events generated by the heatmap and predict
queries for the periodic request approach. INetCEP does not suffer from any
event loss and hence correctly identifies all complex events with a F1-score
of 100%.

##  VI Conclusion

We presented a novel in-network processing architecture, INetCEP, that
implements a unified communication layer for co-existing consumer-initiated
(pull-based) and producer-initiated (push-based) interaction patterns. In this
way, a wide range of IoT applications with push notifications and request-
reply interactions are supported. With the proposed meta query language, we
can express interest in aggregated data that is resolved and processed in a
distributed manner in an IoT network. In our evaluation, we showed that
INetCEP is more than $15\times$ faster in terms of forwarding events than the
state-of-the-art CEP system Flink. Furthermore, the delivery and processing of
complex queries on the mentioned datasets is about $32\times$ faster than
_Flink_ and more than $100\times$ faster than a naive _pull-based_ reference
approach while maintaining $100\%$ accuracy. There are several areas for
future work. For example, extending INetCEP to run on the CCN-lite kernel
module promises high performance benefits for our unified communication
layer999In our early efforts, we have updated CCN-lite to support the latest
Linux kernel. https://github.com/cn-uofbasel/ccn-lite/pull/375. [Accessed on
1.11.2020] when using the Linux kernel, similar to other work on CEP in the
kernel [22]. Another promising area of future work is to generate an optimal
operator graph, e.g., based on operator _selectivity_ or by partitioning the
operator graph by performing query optimization [12]. The placement module
should be extended to support other decentralized solutions and/or other QoS
metrics [14] like throughput and availability. Finally, eliminating DDoS
attacks for push-based communication is another research direction.

## Acknowledgements

This work is funded by the German Research Foundation (DFG) as part of the
project C2, A3 and C5 within the Collaborative Research Center (CRC) SFB 1053
– MAKI.

## References

  * [1] FP7 Green ICN project.  http://www.greenicn.org/, 2013.  Accessed on 01.11.2020.
  * [2] 2014 debs grand challenge.  https://debs.org/grand-challenges/2014/, 2014.  Accessed on 01.11.2020.
  * [3] Ccn-lite github.  https://github.com/cn-uofbasel/ccn-lite, 2019.  Accessed on 01.11.2020.
  * [4] Inetcep github.  https://github.com/luthramanisha/INetCEP, 2020.  Accessed on 01.11.2020.
  * [5] S. H. Ahmed and D. Kim.  Named data networking-based smart home.  ICT Express, 2(3):130 – 134, 2016.  Special Issue on ICT Convergence in the Internet of Things (IoT).
  * [6] T. Akidau, A. Balikov, K. Bekiroğlu, S. Chernyak, J. Haberman, R. Lax, S. McVeety, D. Mills, P. Nordstrom, and S. Whittle.  Millwheel: Fault-tolerant stream processing at internet scale.  Proceedings of the VLDB Endowment, 6(11):1033–1044, Aug. 2013.
  * [7] F. Álvarez, L. Almon, P. Lieser, T. Meuser, Y. Dylla, B. Richerzhagen, M. Hollick, and R. Steinmetz.  Conducting a large-scale field test of a smartphone-based communication network for emergency response.  In Proceedings of the 13th Workshop on Challenged Networks, CHANTS ’18, pages 3–10, 2018.
  * [8] B. J. Bonfils and P. Bonnet.  Adaptive and decentralized operator placement for in-network query processing.  Telecommunication Systems, 26(2):389–409, 2004.
  * [9] P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford, C. Schlesinger, D. Talayco, A. Vahdat, G. Varghese, and D. Walker.  P4: Programming protocol-independent packet processors.  SIGCOMM Computer Communication Review, 44(3):87–95, 2014.
  * [10] A. Buchmann and B. Koldehofe.  Complex event processing.  IT - Information Technology, 51(5):241 – 242, 2009.
  * [11] C. Cachin, R. Guerraoui, and L. Rodrigues.  Introduction to Reliable and Secure Distributed Programming.  Springer Publishing Company, Incorporated, 2nd edition, 2011.
  * [12] L. Cao and E. A. Rundensteiner.  High performance stream query processing with correlation-aware partitioning.  Proceedings of the VLDB Endowment, 7(4):265–276, Dec. 2013.
  * [13] P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas.  Apache flink: Stream and batch processing in a single engine.  IEEE Data Engineering Bulletin, 38:28–38, 2015.
  * [14] V. Cardellini, V. Grassi, F. Lo Presti, and M. Nardelli.  Optimal operator placement for distributed stream processing applications.  In Proceedings of the 10th ACM International Conference on Distributed and Event-Based Systems, DEBS 2016, pages 69–80, 2016.
  * [15] A. Carzaniga, K. Khazaei, M. Papalini, and A. L. Wolf.  Is information-centric multi-tree routing feasible?  SIGCOMM Computer Communication Review, 43(4):3–8, Aug. 2013.
  * [16] A. Carzaniga, M. Papalini, and A. L. Wolf.  Content-based publish/subscribe networking and information-centric networking.  In Proceedings of the ACM SIGCOMM Workshop on Information-centric Networking (ICN), pages 56–61, 2011.
  * [17] A. Charny, D. D. Clark, and R. Jain.  Congestion control with explicit rate indication.  In Proceedings IEEE International Conference on Communications (ICC), volume 3, pages 1954–1963, 1995.
  * [18] J. Chen, M. Arumaithurai, L. Jiao, X. Fu, and K. K. Ramakrishnan.  Copss: An efficient content oriented publish/subscribe system.  In ACM/IEEE Seventh Symposium on Architectures for Networking and Communications Systems, pages 99–110, 2011.
  * [19] A. Frömmgen, D. Stohr, B. Koldehofe, and A. Rizk.  Don’t repeat yourself: seamless execution and analysis of extensive network experiments.  In Proceedings of the 14th ACM International Conference on emerging Networking Experiments and Technologies (CONEXT), pages 20–26, 2018\.
  * [20] M. Gibbens, C. Gniady, L. Ye, and B. Zhang.  Hadoop on named data networking: Experience and results.  Proceedings of the ACM Measurement and Analysis of Computing Systems (POMACS), 1(1):2:1–2:21, June 2017.
  * [21] C. Gündoğan, P. Kietzmann, T. C. Schmidt, and M. Wählisch.  HoPP: Robust and Resilient Publish-Subscribe for an Information-Centric Internet of Things.  In Proceedings of the 43rd IEEE Conference on Local Computer Networks (LCN), pages 331–334, 2018.
  * [22] P. Graubner, C. Thelen, M. Körber, A. Sterz, G. Salvaneschi, M. Mezini, B. Seeger, and B. Freisleben.  Multimodal complex event processing on mobile devices.  In Proceedings of the 12th ACM International Conference on Distributed and Event-Based Systems (DEBS), page 112–123, 2018.
  * [23] V. Jacobson, D. K. Smetters, J. D. Thornton, M. F. Plass, N. H. Briggs, and R. L. Braynard.  Networking named content.  In Proceedings of the 5th ACM International Conference on Emerging Networking Experiments and Technologies (CONEXT), pages 1–12, 2009\.
  * [24] B. Koldehofe, R. Mayer, U. Ramachandran, K. Rothermel, and M. Völz.  Rollback-recovery without checkpoints in distributed event processing systems.  In Proceedings of the 7th ACM international conference on Distributed event-based systems (DEBS), pages 27–38, 2013.
  * [25] T. Koponen, M. Chawla, B.-G. Chun, A. Ermolinskiy, K. H. Kim, S. Shenker, and I. Stoica.  A data-oriented (and beyond) network architecture.  SIGCOMM Computer Communication Review, 37(4):181–192, 2007.
  * [26] C. Köpp, H.-J. von Mettenheim, and M. H. Breitner.  Decision analytics with heatmap visualization for multi-step ensemble data.  Business & Information Systems Engineering, 6(3):131–140, 2014\.
  * [27] D. Kreutz, F. M. V. Ramos, P. E. Verissimo, C. E. Rothenberg, S. Azodolmolky, and S. Uhlig.  Software-defined networking: A comprehensive survey.  Proceedings of the IEEE, 103(1):14–76, 2015.
  * [28] M. Król and I. Psaras.  Nfaas: Named function as a service.  In Proceedings of the 4th ACM Conference on Information-Centric Networking (ICN), pages 134–144, 2017.
  * [29] S. Kulkarni, N. Bhagat, M. Fu, V. Kedigehalli, C. Kellogg, S. Mittal, J. M. Patel, K. Ramasamy, and S. Taneja.  Twitter Heron.  Proceedings of the ACM International Conference on Management of Data (SIGMOD), pages 239–250, 2015.
  * [30] H. Kung.  Traffic Management for High-Speed Networks.  The National Academies Press, Washington, DC, 1997.
  * [31] M. Luthra, B. Koldehofe, J. Höchst, P. Lampe, A. H. Rizvi, R. Kundel, and B. Freisleben.  INetCEP: In-Network Complex Event Processing for Information-Centric Networking.  In 2019 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS), pages 1–13, 2019.
  * [32] M. Luthra, B. Koldehofe, P. Weisenburger, G. Salvaneschi, and R. Arif.  TCEP: Adapting to Dynamic User Environments by Enabling Transitions Between Operator Placement Mechanisms.  In Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems (DEBS), pages 136–147, 2018.
  * [33] A. Martin, R. Marinho, A. Brito, and C. Fetzer.  Predicting energy consumption with streammine3g.  In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems (DEBS), pages 270–275, 2014.
  * [34] N. B. Melazzi.  CONVERGENCE: Extending the Media Concept to Include Representations of Real World Objects.  The Internet of Things, pages 129–140, 2010.
  * [35] M. Nardelli, V. Cardellini, V. Grassi, and F. L. Presti.  Efficient operator placement for distributed data stream processing applications.  IEEE Transactions on Parallel and Distributed Systems, 30(8):1753–1767, 2019.
  * [36] D. Oran.  Maintaining CCNx or NDN flow balance with highly variable data object sizes.  https://tools.ietf.org/id/draft-oran-icnrg-flowbalance-04.html, July 2019\.  Internet Research Task Force (IRTF), Internet Draft, draft-oran-icnrg-flowbalance-00.
  * [37] B. Ottenwälder, B. Koldehofe, K. Rothermel, K. Hong, D. Lillethun, and U. Ramachandran.  MCEP: A Mobility-Aware Complex Event Processing System.  ACM Transactions on Internet Technology (TOIT), 14(1):1–24, 2014\.
  * [38] B. Ottenwälder, B. Koldehofe, K. Rothermel, K. Hong, and U. Ramachandran.  RECEP: Selection-Based Reuse for Distributed Complex Event Processing.  In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems (DEBS), page 59–70, 2014.
  * [39] M. Papalini, A. Carzaniga, K. Khazaei, and A. L. Wolf.  Scalable routing for tag-based information-centric networking.  In Proceedings of the 1st ACM Conference on Information-Centric Networking (ICN), page 17–26, 2014.
  * [40] A. Rahman, D. Trossen, D. Kutscher, and R. Ravindran.  Deployment Considerations for Information-Centric Networking (ICN), 2019\.  Internet Research Task Force (IRTF), Internet Draft, draft-irtf-icnrg-deployment-guidelines-04.
  * [41] P. M. Research.  Smart plugs market: Global industry analysis 2015 - 2019 and forecase 2020 - 2030, 2020.  Accessed on 31.10.2020.
  * [42] C. Scherb, C. Marxer, U. Schnurrenberger, and C. Tschudin.  In-network live stream processing with named functions.  In Proceedings of the IFIP Networking Conference and Workshops (IFIP Networking), pages 1–6, 2017.
  * [43] W. Shang, A. Bannis, T. Liang, Z. Wang, Y. Yu, A. Afanasyez, J. Thompson, J. Burke, B. Zhang, and L. Zhang.  Named data networking of things.  In Proceedings of the 1st IEEE International Conference on Internet-of-Things Design and Implementation (IoTDI), pages 117–128. IEEE, 2016\.
  * [44] M. Sifalakis, B. Kohler, C. Scherb, and C. Tschudin.  An information centric network for computing the distribution of computations.  In Proceedings of the 1st ACM Conference on Information-Centric Networking (ICN), pages 137–146, 2014.
  * [45] S. Signorello, R. State, J. François, and O. Festor.  Ndn.p4: Programming information-centric data-planes.  In Proceedings of the IEEE NetSoft Conference and Workshops (NetSoft), pages 384–389, 2016.
  * [46] A. Tagami, T. Yagyu, K. Sugiyama, M. Arumaithurai, K. Nakamura, T. Hasegawa, T. Asami, and K. K. Ramakrishnan.  Name-based push/pull message dissemination for disaster message board.  In Proceedings of the IEEE International Symposium on Local and Metropolitan Area Networks (LANMAN), pages 1–6, 2016.
  * [47] C. Tschudin and M. Sifalakis.  Named Functions for Media Delivery Orchestration.  In Proceedings of the 20th International Packet Video Workshop (PV), pages 1–8, 2013.
  * [48] C. Tschudin and M. Sifalakis.  Named Functions and Cached Computations.  In Proceedings of the IEEE Consumer Communications and Networking Conference (CCNC), pages 851–857, 2014.
  * [49] R. Ullah, M. A. U. Rehman, M. A. Naeem, B.-S. Kim, and S. Mastorakis.  Icn with edge for 5g: Exploiting in-network caching in icn-based edge computing for 5g networks.  Future Generation Computer Systems, 111:159 – 174, 2020.
  * [50] P. Weisenburger, M. Luthra, B. Koldehofe, and G. Salvaneschi.  Quality-aware runtime adaptation in complex event processing.  In Proceedings of the 12th IEEE International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS), pages 140–151, 2017.
  * [51] L. Zhang, A. Afanasyev, J. Burke, V. Jacobson, K. Claffy, P. Crowley, C. Papadopoulos, L. Wang, and B. Zhang.  Named data networking.  ACM SIGCOMM Computer Communication Review, 44(3):66–73, 2014.