11institutetext: Department of Computer Science, Malmö University
Internet of Things and People Research Center,
Malmö, Sweden
{radu.c.mihailescu, romina.spalazzese, paul.davidsson<EMAIL_ADDRESS>
#  A Role-Based Approach for Orchestrating Emergent Configurations in the
Internet of Things

Radu-Casian Mihailescu  Romina Spalazzese  Paul Davidsson  Clint Heyer

###### Abstract

The Internet of Things (IoT) is envisioned as a global network of connected
”things” enabling ubiquitous machine-to-machine (M2M) communication. With
estimations of billions of sensors and devices to be connected in the coming
years, the IoT has been advocated as having a great potential to impact the
way we live, but also how we work. However, the connectivity aspect in itself
only accounts for the underlying M2M infrastructure. In order to properly
support engineering IoT systems and applications, it is key to orchestrate
heterogeneous ’things’ in a seamless, adaptive and dynamic manner, such that
the system can exhibit a goal-directed behaviour and take appropriate actions.
Yet, this form of interaction between things needs to take a user-centric
approach and by no means elude the users’ requirements. To this end,
contextualisation is an important feature of the system, allowing it to infer
user activities and prompt the user with relevant information and interactions
even in the absence of intentional commands. In this work we propose a role-
based model for emergent configurations of connected systems as a means to
model, manage, and reason about IoT systems including the user’s interaction
with them. We put a special focus on integrating the user perspective in order
to guide the emergent configurations such that systems goals are aligned with
the users’ intentions. We discuss related scientific and technical challenges
and provide several uses cases outlining the concept of emergent
configurations.

##  1 Introduction

Connecting everything that can benefit from being connected, from both digital
and physical worlds, is becoming reality nowadays. IoT is already successfully
adopted in several application domains such as environmental monitoring,
healthcare service, transportation, inventory and production management among
others [41, 2]. Recent studies show that IoT changed the way people use the
Internet, mobile devices and sensors [18].

However, in order to reach its full potential, the IoT domain has to overcome
a number of shortcomings. On the one hand, in the enterprise space, the
majority of IoT solutions suffer from the so-called ’cloud silo’ problem. A
common practice in today’s IoT solutions is to have processes and data kept in
separate data centers, making it difficult to access it or to interact with
other systems. This design is often dictated by the companies’ business models
or due to certain security concerns. Nonetheless, these silos can cause real
issues by keeping data and services locked away and unavailable for an
efficient use by external third parties. On the other hand, in the retail
space of devising smart things as the building blocks of IoT, noticeably,
things are typically designed to follow a simple request-response pattern
enacted by the user. This clearly imposes many restrictions in terms of the
user experience and makes it impractical to specify requests and goals that
concern more than one device at a time.

We aim to address these challenges encountered across the entire IoT spectrum
in a unified manner. The focus of our current work is to provide means to
orchestrate heterogeneous things in a seamless, adaptive and dynamic manner,
such that the system can exhibit a goal-directed behaviour and take
appropriate actions, according to the user’s desires and intentions. As we
shall see in the following sections, this raises many interesting research
issues open for cross-pollination with the multiagent domain. Informally, we
term an Emergent Configuration (EC) as a set of things with their
functionalities and services that connect and cooperate temporarily to achieve
a goal [8]. Given the continuously changing character of many IoT systems, ECs
can change unpredictably. This tendency is even more pronounced in domains
that present pervasive characteristics such as mobile users, devices and
sensors. Thus, it is important to provide the user with a coherent system at
any point in time, capable to align to the user’s context and goals.

To this end, we put forward in this paper a general theoretical model for
designing Emergent Configurations. Our goal is to provide an encompassing
framework where new mechanisms addressing the different aspects outlined by
our model can be developed and deployed. Specifically, ECs set the stage for
addressing and reasoning about three key categories of challenges that
represent a barrier to effectively enable today’s IoT systems:

  1. (i)

challenges in interoperability \- include aspects of how can devices operate
across different types of networks, how to connect and deal with mobile
devices, how to integrate devices of different hardware and software
specifications, how to ensure ease of scalability?

  2. (ii)

challenges in coordination: are concerned with how can devices collectively
operate based on the user delegated goals to the system, how can devices
separately adapt to runtime conditions while a consistent view of the system
is maintained by the user, how to resolve data inconsistencies when retrieving
the data from various sources?

  3. (iii)

challenges in user interaction \- how to improve usability and provide novel
ways of interaction with sets of devices, how to provide a user-friendly
interface for a variety of devices, how to avoid overwhelming the user with
data, how to improve usefulness and automatically identify relevant
information, how to make the system proactive in supporting the user’s tasks?

The reminder of the paper is organized as follows. In Section 2 we present the
basics of our proposed model characterizing Emergent Configurations. In
Section 3 we outline a representative EC use-case developed in co-production
with industry partners. In Section 4 we discuss and point towards several key
enabling technologies. Section 5 concludes and draws future research
directions.

##  2 Emergent Configurations for the Internet of Things

###  2.1 Basics of the proposed model

Figure 1: Concept graph of the role-based model

In this section we lay the groundwork for providing a model of emergent
configurations in an IoT ecosystem. The proposed framework focuses in
particular on the importance of roles adopted by devices in various
configuration settings. The aim is to provide a framework that is general
enough to model different IoT scenarios and can capture the key aspects of
adaptability to dynamic environments, while maintaining a coherent user
perspective of the system according to its delegated objectives. This approach
makes it easier to reason about integrating algorithms, which deal with
different subproblems of the domain, as well as to reveal gaps in existing
research. In Figure 1 we give an overview of the different elements considered
in our model, which we describe hereafter.

###### Definition 1

A connected thing is an uniquely identifiable (embedded) computing system that
has at least the capability to exchange data.

Basically, a connected thing can represent anything, spanning from the
simplest RFID tag or sensor, to actuators, devices, computing-enabled physical
world objects, smartphones, computers, or even items of an increased
complexity such as modern fully equipped self-driving cars. It is further
assumed that things which are equipped with sensing capabilities can perceive,
collect and index timestamped signals received from the environment or
communication messages from other things in the environment. As a general
approach, signal sequences registered over a certain time-frame are aggregated
and classified to a higher-level representation, generating a new sequence of
events. Similarly, in a hierarchical manner, the system can construct higher-
level abstractions from the underlying sensing data.

###### Definition 2

A linear sequence of signals or events
$\epsilon_{1},\epsilon_{2},\cdots\epsilon_{t}$ obtained over the time period
$t$ is perceived as (part of) the activity performed by the user in the
environment.

Suppose as an example a scenario where a smartphone is used to track signals
from the accelerometer, gyro and GPS, inferring events such as accelerating
speed, entering a park and continue at increased speed, in order to
characterize a ’jogging in the park’ activity. The activity carried out by the
user can be further enriched with additional information (e.g. time of day,
weather, etc.), which captures the overall context of the user [13, 10].

###### Definition 3

Context is denoted here as the set of current statements about the user, which
are either directly sensed from the environment or inferred.

In fact, there is an extensive body of work focusing on the area of activity
recognition and context-awareness, as it pertains to the various sensing
modalities (e.g. smartphones [20], vision [31], ambient sensors [25, 23]) and
the different application domains (e.g. health [7], smart living spaces [22],
sports [5]). For a comprehensive survey on context-aware computing we further
refer the reader to [28, 3]. The underlying goal is to devise techniques that
seamlessly acquire context information without the explicit intervention or
feedback of the user. The use of semantics is one potential avenue for
introducing meta-mechanisms that capture the relation between contexts and
trigger the appropriate techniques for context inference, in a more general
setting that can cover a wider spectrum of senors and domains. In addition, an
important part of the context is the user profiling, which retains the user’s
preferences.

###### Definition 4

The environment $\mathcal{E}=\langle\Lambda,\mathcal{T}\rangle$ specifies a
set of constraints $\Lambda=\\{c_{1},\cdots,c_{n}\\}$ which induce the set of
things $\mathcal{T}$, providing a clear delimitation between the things in
$\mathcal{T}$ and those outside this set. Constraints are further
differentiated into physical constraints (e.g. devices at a certain location,
communication ports, resource configuration) and virtual constraints (e.g.
communication protocols, restrictions over virtual platforms, latency
requirements).

###### Definition 5

A role is a tuple $\Pi=\langle\varphi,\chi,\mathcal{S}\rangle$, which
specifies an invocation function $\varphi:\chi\rightarrow\mathcal{S}$, which
activates, for a given condition in the set $\chi$, an item $s\in\mathcal{S}$
from the restricted set of possible actions or service type interactions
$\mathcal{S}$ available for this role.

Items in the condition set $\chi$ can denote anything from incoming messages
to perceived events or activities. Note that, a role can also be interpreted
as a certain skill set $\mathcal{S}$, based on the actions and services that
the role can perform.

###### Definition 6

A configuration is a tuple
$\mathcal{C}=\langle\mathcal{R},\mathcal{T},\Delta,\beta,\mathcal{E}\rangle$,
where $\mathcal{R}$ is a collection of roles, $\mathcal{T}$ is a collection of
things and $\Delta:\mathcal{R}\rightarrow\mathcal{T}$ represents a mapping of
roles to things. With $\beta$ we denote the purpose of the configuration and
$\mathcal{E}$ represents the environment.

It is important to highlight that function $\varphi$ also specifies the type
of action or service invoked by a certain role $r$. On the one hand, we
distinguishing between services $E(r)$ that are available to that respective
role in the configuration, and services provided by this role, $P(r)$, to be
accessed by other roles in the configuration. On the other hand, we
differentiate between services $M(r)$ that are compulsory in the configuration
and those that are optional $O(r)$. The set of compulsory services in a
configuration $\mathcal{C}$ induces the set of compulsory roles
$\Gamma_{\mathcal{C}}(\mathcal{R})$, required in order to run the
configuration. Optional roles $\gamma_{\mathcal{C}}(\mathcal{R})$ can, for
instance, be used to map to things in the environment that are only temporary
part of the configuration.

In other words, roles provide the interaction constraints between the
different entities in the configuration and can be assimilated in a broader
acceptation to a workflow or (negotiation) protocol, that connected things
adopting different roles need to abide to. A simple example would be having a
configuration with the purpose of determining the device with the highest QoS
for a certain service among a list of possible candidates. This would entail
prescribing the roles of service provider and service consumer, whereas the
interaction could specify a simple Contract Net protocol [37], which follows,
on the provider side, a predefined sequence of broadcasting a call for bids,
followed by receiving and evaluating bids and finally, determining the winner.

Going back to the overview in Fig. 1, the inferred activity of the user
captures her underlying goals in a given context, which in turn dictates what
roles her associated connected things can adopt in a configuration. We use the
term associated connected things in relation to the user to denote any means
which interfaces the user with the configuration. They can range from
something as trivial as a PIR sensor detecting presence of a person, to more
advanced means of interaction, such as a smartphone, which can require some
form of user feedback. In the situation when different parts of the
configuration retain different information about the user, it is important to
maintain a coherent view and to resolve potential data inconsistencies, when
retrieving the data from various sources. Trust-based data fusion techniques
[42] can represent a viable manner for prioritizing, processing and
harmonizing data from heterogeneous sources. Note, that the purpose of the
configuration needs to be aligned with the user’s goals, otherwise its
significance to the user would be meaningless. This brings about a number of
interesting research questions, that situates the user as the focal point of
decision making in the system.

For instance, it is important to identify what is an appropriate way to
specify and represent user goals. The use of ontologies could provide a way
for representing, on the one hand, the user goals and on the other hand, the
purpose of the configuration. If the user and the configuration’s ontologies
align, then we could reason that the configuration can provide a relevant
function for the user. A dialog-based user-interaction can allow the user to
guide the actual execution run of the configuration. Also, to what extent
should the system use unobtrusive techniques in order to elicit the user’s
goals and obtain contextual information, and when should this information be
explicitly specified based on the user’s interaction with the system? There is
an underlying trade-off here between the proactiveness of the system and the
transparency of the process. If the system displays an increased level of
autonomy, which is not clearly understood by the user, this can in fact hinder
its usability and frustrate the user. In this regard, the recently introduced
research avenue of UbiCARS [21], which aims to characterize ubiquitous
recommender systems, can bring interesting contributions to the concepts
emphasized here.

###  2.2 Emergence

Given that the interaction between connected things in a configuration is
mediated through roles, another important aspect concerns the way according to
which roles are established. It is important to allow the users or the system
itself do modify the system behavior, in terms of prescribing new roles in a
configuration or evolving more specialized roles to carry out certain tasks.

###### Definition 7

Let $\mathcal{U}$ denote a user with goal $\mathcal{G}$. We term an Emergent
Configuration, the outcome of generating a configuration
$\mathcal{C}=\langle\mathcal{R},\mathcal{T},\Delta,\beta,\mathcal{E}\rangle$,
where the purpose $\beta$ of the configuration accommodates the goal
$\mathcal{G}$ of user $\mathcal{U}$, given the environment $\mathcal{E}$.

In this sense, we make the distinction between weak and strong  emergence. In
the former case, we have static roles and we are only concerned with
identifying a suitable mapping $\Delta$ from roles to things. In effect, this
represents an open-ended process where things can join the configuration on-
the-fly, adopting certain roles, or may choose to leave the configuration at a
certain point in time. This may also included situations where mobile software
agents, which are performing certain roles, migrate from device to device in
order to achieve the purpose of the configuration. For the later case, we
assume that roles are dynamic and can change over time in order to better
accommodate the goals of the user and the purpose of the configuration. A key
feature of the system is its capability to adapt roles in order to correspond
with the functionalities of the connected things adopting those roles.

On the one hand, role adaptation can be triggered by explicitly receiving
instructions from the user regarding the behavior of the configuration.
Interaction design can play an essential part in this regard. On the other
hand, the things associated to the user, through continuous monitoring, can
determine changes in the context or activities perform by the user, and
therefore infer a new user goal that sets in motion at run-time a role
adaptation procedure. In the more extreme case, we can envision a setting
where either the configuration is built in a top-down manner, solely from
specifying a certain user goal (which also represents purpose of the
configuration) or alternatively, a collection of roles are evolved to serve a
newly acquired purpose in a self-organizing fashion. This can be translated in
a simple example by considering a smart home environment where the user is
presented with a new type of functionality, such as begin able to use the
smartphone in order to control an operate a smart TV. In this instance, the
smartphone and the TV adopt the roles of master and slave respectively.

###  2.3 Managing emergent configurations

Another aspect that has to be considered is linked to the way according to
which a configuration is managed. A configuration $\mathcal{C}$, where
$\mathcal{R}=\Gamma_{\mathcal{C}}(\mathcal{R})$ and
$\gamma_{\mathcal{C}}(\mathcal{R})=\emptyset$, meaning that configuration
$\mathcal{C}$ includes only compulsory roles, is termed a centralized
configuration. For example, the leader role can be compulsory for initiating a
configuration and regulating the interaction between the other members of the
configuration. Then, if all the roles specified by the configuration are
compulsory, the configuration is a centralized one. At the opposite end of the
spectrum, in a decentralized configuration, we have
$\mathcal{R}=\gamma_{\mathcal{C}}(\mathcal{R})$ and
$\Gamma_{\mathcal{C}}(\mathcal{R})=\emptyset$. Here, there is no device that
assumes a compulsory role and the set of services provided by the
configuration depends solely on the optional roles adopted by devices in the
configuration at that time and the services they can receive and be expected
to deliver. We term a hybrid configuration one that involves both compulsory
and optional roles
($\mathcal{R}=\Gamma_{\mathcal{C}}(\mathcal{R})\cup\gamma_{\mathcal{C}}(\mathcal{R}),\Gamma_{\mathcal{C}}(\mathcal{R})\neq\emptyset,\gamma_{\mathcal{C}}(\mathcal{R})\neq\emptyset$),
in the sense that compulsory roles are instrumental and all the other roles
depend on the services provided by them.

The request for adopting a certain role can either be solicited by a member of
the configuration (e.g. the leader advertises roles), or proposed by a device
in the environment. This can take the form of a simple ’handshaking’ procedure
or, in more complex scenarios, it can imply that a negotiation occurs which
can result in a service level agreement (SLA) [1], that regulates future
interactions. A normative extension of roles using representations such as
obligations and prohibitions could further be used to enforce certain device
behaviors [9]. What is then an appropriate way to handle communication? An
option inspired from MAS is to communicate using ACL (Agent Communication
Language), a standard language for agent communication defined by FIPA (The
Foundation of Intelligent Physical Agents) [39]. Also relating to
implementation issues, we point towards model-driven engineering as a proposed
support technology for ECs and the IoT, and regard [8] as a possible
architecture provided for self-adaptive systems.

###  2.4 Discussion

Finally, perhaps nearest to our work, we can relate the model proposed here to
several efforts addressing the design of ontologies for ubiquitous and
pervasive systems (e.g. CoDAMoS ontology [32]). However, the research in this
area is focused on developing specialized ontologies dedicated to model
specific domains, such as BOnSAI [38], which addresses energy savings in smart
building environments and has a particular emphasis on modeling devices,
appliances and actuators. Another example is the OntoAMI model [36], which
presents a simplistic semantic model for universal use across ambient
intelligence applications, but fails to incorporate the user into their
approach. Alternatively, in this work, we put forward a novel concept of
Emergent Configurations to denote a set of things with their functionalities
and services that connect and cooperate temporarily to achieve a goal and
provide a framework based on the notion of roles to support reasoning and
developing such systems. Moreover, we put special emphasis on the user and
explicitly account for the user’s goals based on its current context and
activity, in order to guide the emergence of configurations, such that the
system’s goals are aligned with the users’ intentions.

It is important to point out a number of key benefits attained via our role-
based approach. From a development standpoint, roles enable a decoupling
between the interaction aspects, which prescribe how collaboration between
devices occurs, and the internal computational processes of each device, which
need not be exposed to other third parties. From the conceptual point of view,
roles can be referred to as design patterns, in the sense that a set of roles
that are used to fulfill the purpose of a configuration could further be
(adapted and) reused for similar purposes. That is, roles can promote quite
general, high-level solutions, which can then be parametrized and instantiated
in various specific applications. Roles encapsulate certain functionalities,
which equip devices with the necessary building blocks for establishing
dynamic interactions in continuously changing environments and which can be
directly related to the user’s goal. In this manner, devices can dynamically
adapt in a versatile manner to different coordination protocols and different
functionalities by assuming various roles in different configurations.

##  3 Use case of emergent configurations

Figure 2: Emergent Configuration use-case: Mesh Presenter

In this section we introduce the ”Mesh Presenter”
prototype111https://vimeo.com/130228635 which has been developed as part of
the ECOS project222http://iotap.mah.se/ecos/, where we investigate how to
model, interact with, and reason about ECs. We use this straightforward case
study (see Fig. 2) to emphasize the expressiveness of our formal model, which
clarifies based on precise definitions how to dynamically integrate different
entities and stakeholders to a configuration and how to regulate their
behavior.

The purpose of the ”Mesh Presenter” (MP) configuration is that of providing an
on-the-fly virtual collaborative workspace for sharing and generating content
on smart devices (e.g. smartphone, tablets, projectors) co-located in a
meeting room. In more detail, we describe the MP configuration according to
the definitions introduced in Section 2. Recall that roles represent one of
the key underpinnings of our approach, providing a way to mediate interaction
between different entities in the configuration in a dynamic manner.

Specifically, the MP configuration pertains that the set of roles
$\mathcal{R}$ includes two types of roles. On the one hand, there is a
compulsory role referred as $\Gamma(\mathcal{R})=\\{presenter\\}$ and an
optional role termed $\gamma(\mathcal{R})=\\{reviewer\\}$. Each of these roles
carries out a certain functionality, which can be described in terms of
services. According to our model, we differentiate between two categories of
services associated to each role. The set of services provided by the
presenter include P(presenter) = {share presentation; add/remove reviewer;
enable presenter control}. This means that the reviewer role can request the
presenter to execute any of these services. The presenter can grant or revoke
access to any of the above-mentioned services to the reviewer. Given the
collaborative context of this configuration, the reviewer can also actively
participate in the content generation process by exposing the following
service P(reviewer) = {share content}. However, the presenter is the one who
decides if the new data (e.g. image, slides) should be broadcasted on all
devices within the configuration. For the sake of simplicity, in this usecase,
having only two roles, the following sets coincide P(presenter) = E(reviewer)
and E(presenter) = P(reviewer), meaning that all services provided by one of
the roles are accessible by the other. In addition, it is required that all
services in P(presenter) are implemented for this particular role, thus
M(presenter)=P(presenter) and $O(presenter)=\emptyset$. The same applies for
the reviewer role.

In order to have a well-defined configuration several additional aspects need
to be specified. We assume that users interact with the MP configuration by
means of their associated smart devices. Importantly, the configuration
environment $\mathcal{E}$ determines the set of devices that the presenter can
allow to be part of the configuration. In this case, the inclusion constraint
is based on a proximity rationale (i.e. presence in the meeting room). Also,
the environment restricts the manner in which communication can take place, in
this case being mesh connectivity. Contextualization also plays a significant
part here. The real-time incoming data from the sensors of the device, such as
a smartphone, are used to infer the user’s activity. Once the context of
’being present in a meeting’ is recognized, this acts as a precondition for
the reviewer role to invoke the service of joining the configuration.
Similarly, when the presenter determines that another device has entered its
environment $\mathcal{E}$ it will forward an inquiry to that device to join
the configuration. User feedback is required in both of these instances in
order to finalize the procedure. The preconditions for invoking all of the
remaining services, such as sharing a presentation or granting presenter
control to another device are all triggered based on a simple user interface,
which basically summaries function $\varphi$, that is responsible for service
invocation.

Note that, in the case of the MP configuration, we are dealing with a weak
type of emergence, as we are essentially assigning fixed roles to devices that
are joining the configuration. Alternatively, a scenario that would exhibit a
strong emergence would entail that roles evolve and specialize over time, such
that they can provide, new functionalities for the configuration, new services
or new types of interaction (e.g. a new reviewer role that can connect
remotely to the configuration). Finally, with respect to managing the
configuration, we point out that MP is a hybrid configuration as it consists
of both compulsory and optional roles. A configuration can exist even in the
absence of a reviewer, which can assume a temporary role, but not in the
absence of the presenter which takes here a coordinator type of role.

##  4 Enabling technologies

In our work, we are taking a user-centric approach, by placing the users at
the very center of the IoT ecosystem, which they have to manage in an
efficient and effective way such that it can service their needs. We identify
in the following, several key enabling technologies for IoT and point towards
the contribution they can bring in the context of our model. Also, we relate
them back to the three classes of problems outlined in Section 1. Notably, we
identify multiagent systems as arguably the main area of interest for
advancing the EC concept presented in this work. Likewise, the user-centric
approach advocated here, deploying a role-based solution for orchestrating IoT
environments, provides an interesting perspective for new developments in MAS.

###  4.1 Middleware

The large majority of the research in IoT literature has focused thus far on
dealing with the first class of challenges identified in the introduction,
namely issues in M2M interoperability. In this sense, significant progress has
already been made in terms of middleware platforms (see [34, 12] for a
comprehensive survey) designed to abstract hardware-specific issues and
facilitate interoperability of various components, such as ad-hoc
communication, discovery of new devices or establishing new communication
links. These solutions also simplify the development process at the
application level and exempts developers from handling implementation at the
lower layers. In particular, service-oriented architectures (SOA) [17, 27]
have emerged as a popular way for providing a middleware with standard
protocols, common interfaces and well-defined components. This can be regarded
as an approach by means of which connected things can easily interact and
collaborate such that certain ECs are formed, according to the virtual
constraints of the environment. Fog computing [4] can also be regarded as an
architectural blueprint for ECs, given its distributed infrastructure
comprising of heterogeneous resources, which needs to be managed in a
distributed fashion.

Another important aspect in our model that defines an emergent configuration
is the mapping function $\Delta$, which relates roles to things. A good
candidate for implementing this function is given in [30], where the authors
propose a method for automatically deploying and configuring applications onto
a heterogeneous hardware infrastructure, by taking into account the hardware
properties and characteristics. Alternatively, a multiagent system can act as
an intermediary layer, providing the necessary level of autonomy for reasoning
about adopting different roles in the configuration. At the same time, agent-
based platforms, such as Calvin [29], provide a lightweight distributed
runtime, allowing the system to dynamically deploy agents according to
requirements and hardware constraints. Along the same lines, self-adaptive
systems (SaS) [33] can play an important role in enabling run-time
reallocation, reconfiguration and increasing reuse (e.g. thanks to model-
driven technologies [8]). SaS are concerned with modifying the system’s
behavior at runtime in response to their perception of the environment or the
system itself. In the last decade, the research in SaS has mainly focused on
technical solutions, while less effort has been devoted to the human
involvement that is the special perspective we advocate here.

###  4.2 Multiagent-based coordination

Regarding the second class of challenges outlined in the introduction, in
order to address the problem of device coordination, multi-agent systems (MAS)
[40] is yet another important area we draw inspiration from in the realization
of the concept of emergent configurations in IoT. In particular, we draw the
parallel between ECs and an open MAS, which assumes a high level of
interaction between a large number of highly stochastic, heterogeneous and
possibly self-interested agents operating within a dynamic environment. In
such a system agents may form groups to solve more complex problems via
cooperation, given that many tasks cannot be completed by a single agent
because of limited resources or capabilities. A relevant example in this sense
is given in [24], where the authors propose an online coalition formation
scheme, having agents negotiate and formulate speculative coordination
solutions, with respect to the estimated behavior of the system. This type of
learning-based coordination can be instrumental for ECs in the way that roles
evolve over time (representing in our scenario a case of strong emergence).

Importantly, agents running on devices can also act as a middleware in a
configuration, enabling the interaction between devices, while at the same
time adopting different roles in a versatile manner according to the user’s
goals. To tackle the problem of today’s IoT platforms designed as vertical
silos, we can envision a scenario, where at least one agent assumes a special
broker role, responsible for interfacing with each IoT platform, while the
configuration emerges at a cross-platform level. This means that we could have
for instance a situation where the configuration consumes data from a number
of sensors registered to one IoT platform, then applies a data analysis
component from another big data platform and visualizes the result on the
projector that is managed by yet another local cloud platform. Notably, since
a role-based approach is particularly aimed at providing a high-level
description focusing on the interaction between things, this has the advantage
of simplifying and supporting the development of large scale applications.

In [26] the authors introduce the notion of emergent coordination and discuss
its potential for efficiently handling coordination in open environments. The
authors advocate that a careful intentional design of a limited set of
interaction rules and a common communication protocol could produce the
desired emergent property, which is a higher level property caused by the
interaction of its lower level components. This brings about the question as
to what are the key ingredients required in order to engineer a system such
that it displays certain emergent properties, how does this relate to the
types of emergence identified in ECs and what is the role of the user in this
context?

Moreover, MAS allows to address challenges of autonomous and decentralized
decision-making in a flexible manner, by decomposing complex tasks and
assigning subproblems to loosely coupled components that interact and
coordinate autonomously to solve system-level design goals. In a similar
manner, ECs enhance open MAS by factoring in the user perspective. That is,
ECs explore the extent of users involvement in the system not only as final
user, but also as an active support for the system to guide or enable the
execution of specific goals. An interesting preliminary work is [6], in which
human participants are explicitly modeled and the authors reason about human
involvement in self-adaptation, focusing on the role of human participants as
effectors during the adaptation execution phase.

###  4.3 Interaction Design

Finally, from a user interaction perspective, again, people need to be
involved not just as end-users, but as co-constructors of the system itself. A
common strategy for co-construction has been designing for ’end-user
programmability’, such as rule based systems [11] and ’programming-by-example’
[14]. Here, devices are often used as the ’control panel’ to represent,
configure and control a complex system and its constituent components ([15],
[19]). The disadvantage of this paradigm is that it draws attention away from
the task-at-hand, making it difficult to continue the flow of activity [35]
and can suffer from device-centric rather than activity-centric design [16].
Thus, the challenge of designing IoT systems and applications that take a
user-centric approach in order to guide the emergent configurations, such that
the system’s goals are aligned with the users’ intentions, is still an open
research question. Introducing roles takes one step further in this direction
by means of binding the users’ goals to certain roles assumed by their
devices. Moreover, roles can be evolved in order to match changes in the
user’s goals. The framework described in this paper represents a vision that
has to be tackled gradually, encompassing all of the identified challenges.

##  5 Conclusions and future work

In this paper we defined and characterized emergent configurations of
connected systems, as an approach to model, manage, and reason about
collaborative IoT systems consisting of a number of autonomous yet coordinated
devices and the user’s interaction with them. We propose herein a role-based
approach to ECs that captures a key missing aspect in today’s IoT systems,
namely, adaptability to dynamic environments, along with maintaining a
coherent user perspective of the systems aligned with the user’s objectives.
The exploitation of roles at run-time enables things to dynamically adopt
specific roles in order to satisfy the user’s goal or cope with dynamic
environments. Our aim is to utilize a common framework, which can support a
wide range of IoT applications, instead of targeting specific scenarios, each
of which can exercise particular aspects of the framework. We discussed
scientific and technical challenges involving several research disciplines at
the same time and we provided concrete cases showing how ECs can be used and
help the IoT in practice.

As future work we plan to continue the development of the proposed framework,
with a special emphasis on interaction design and the ways in which we can
deliver an enhanced user experience providing functionality even in the
absence of intentional commands, as well as on designing techniques to ease
the development and deployment of ECs in real case scenarios. Moreover, in
this work we are strictly concerned with generating a configuration with
respect to the goal of a single user. In order to accommodate multiple user,
conflict resolution mechanism (such as negotiation procedures) may need to be
included, in the event that the goals of different users cannot co-exist in a
given configuration. Regarding the evaluation, we would also like to identify
and develop more full-fledged usecases that fully exploit the expressiveness
of our model (e.g. configurations that display strong emergence). It would be
interesting to see the level of reusability we can attain by casting roles to
new usecases and the impact of using the concept of roles throughout the
analysis, design and implementation phases. Besides the development of
additional cases and validation against domain-specific settings, the
framework introduced here is also well suited for the integration, validation
and comparison of already existing algorithm and mechanism or future
proposals. Among them, we mention dynamic discovery of things and services,
goal adaptation mechanisms, approaches for configuration monitoring, context
acquisition and inference, or novel ways for user interaction.

Acknowledgments. This work is partially financed by the Knowledge Foundation
(KKS) through the Internet of Things and People research profile (Malmö
University, Sweden).

## References

  * [1] A. Andrieux, K. Czajkowski, A. Dan, K. Keahey, H. Ludwiga, T. Nakata, J. Pruyne, J. Rofrano, S. Tuecke, and M. Xu.  Web services agreement specification (WS-agreement).  Technical report, 2007.  https://www.ogf.org/documents/GFD.107.pdf.
  * [2] Luigi Atzori, Antonio Iera, and Giacomo Morabito.  The internet of things: A survey.  Computer Networks, 54(15):2787 – 2805, 2010.
  * [3] Claudio Bettini, Oliver Brdiczka, Karen Henricksen, Jadwiga Indulska, Daniela Nicklas, Anand Ranganathan, and Daniele Riboni.  A survey of context modelling and reasoning techniques.  Pervasive and Mobile Computing, 6(2):161 – 180, 2010.
  * [4] Flavio Bonomi, Rodolfo Milito, Jiang Zhu, and Sateesh Addepalli.  Fog computing and its role in the internet of things.  In Proceedings of the First Edition of the MCC Workshop on Mobile Cloud Computing, pages 13–16, New York, NY, USA, 2012. ACM.
  * [5] Andreas Bulling, Ulf Blanke, and Bernt Schiele.  A tutorial on human activity recognition using body-worn inertial sensors.  ACM Computing Surveys, 46(3):33:1–33:33, 2014.
  * [6] Javier Cámara, Gabriel A. Moreno, and David Garlan.  Reasoning about human participation in self-adaptive systems.  In 10th IEEE/ACM International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS 2015, Florence, Italy, May 18-19, 2015, pages 146–156, 2015.
  * [7] Saisakul Chernbumroong, Shuang Cang, Anthony Atkins, and Hongnian Yu.  Elderly activities recognition and classification for applications in assisted living.  Expert Systems with Applications, 40(5):1662 – 1674, 2013.
  * [8] Federico Ciccozzi and Romina Spalazzese.  MDE4IoT: Supporting the Internet of Things with Model-Driven Engineering.  In Proc. of the 10th International Symposium on Intelligent Distributed Computing (IDC’16). Springer - (To appear), 2016.
  * [9] N. Criado, E. Argente, and V. Botti.  Open issues for normative multi-agent systems.  AI Commun., 24(3):233–264, 2011.
  * [10] Valentin Cristea, Ciprian Dobre, and Florin Pop.  Internet of Things and Inter-cooperative Computational Technologies for Collective Intelligence, chapter Context-Aware Environments for the Internet of Things, pages 25–49.  Springer Berlin Heidelberg, 2013.
  * [11] Luigi De Russis and Fulvio Corno.  Homerules: A tangible end-user programming interface for smart homes.  In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pages 2109–2114. ACM, 2015\.
  * [12] Jose Delgado.  Internet of Things and Inter-cooperative Computational Technologies for Collective Intelligence, chapter Service Interoperability in the Internet of Things, pages 51–87.  Springer Berlin Heidelberg, 2013.
  * [13] Anind K. Dey.  Understanding and using context.  Personal Ubiquitous Computing, 5(1):4–7, 2001.
  * [14] Anind K Dey, Raffay Hamid, Chris Beckmann, Ian Li, and Daniel Hsu.  a cappella: programming by demonstration of context-aware applications.  In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 33–40. ACM, 2004.
  * [15] Nicolas Ducheneaut, Trevor F Smith, James Bo Begole, Mark W Newman, and Chris Beckmann.  The orbital browser: composing ubicomp services using only rotation and selection.  In CHI’06 Extended Abstracts on Human Factors in Computing Systems, pages 321–326. ACM, 2006.
  * [16] W Keith Edwards, Mark W Newman, Jana Z Sedivy, and Trevor F Smith.  Experiences with recombinant computing: Exploring ad hoc interoperability in evolving digital networks.  ACM Transactions on Computer-Human Interaction (TOCHI), 16(1):3, 2009.
  * [17] Thomas Erl.  Service-Oriented Architecture: Concepts, Technology, and Design.  Prentice Hall PTR, Upper Saddle River, NJ, USA, 2005.
  * [18] Dave Evans.  How the next evolution of the internet is changing everything.  Technical report, Cisco Internet Business Solutions Group, 2011.  http://www.cisco.com/web/about/ac79/docs/innov/IoT_IBSG _0411FINAL.pdf .
  * [19] Yucheng Jin, Chi Tai Dang, Christian Prehofer, and Elisabeth André.  A multi-display system for deploying and controlling home automation.  In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces, pages 399–402. ACM, 2014.
  * [20] Jennifer R. Kwapisz, Gary M. Weiss, and Samuel A. Moore.  Activity recognition using cell phone accelerometers.  SIGKDD Explorations Newsletter, 12(2):74–82, 2011.
  * [21] Christos Mettouris and George A. Papadopoulos.  Ubiquitous recommender systems.  Computing, 96(3):223–257, 2014.
  * [22] Radu-Casian Mihailescu and Paul Davidsson.  Integration of smart home technologies for district heating control in pervasive smart grids.  In Proceedings of the First International Workshop on Pervasive Smart Living Spaces, 2017.
  * [23] Radu-Casian Mihailescu, Paul Davidsson, and Jan A. Persson.  Multiagent model for agile context inference based on articial immune systems and sparse distributed representations.  In Multi-Agent Systems and Agreement Technologies - 13th European Conference, EUMAS 2015, and Third International Conference, AT 2015, Athens, Greece, December 17-18, 2015, Revised Selected Papers, pages 82–87, 2015.
  * [24] Radu-Casian Mihailescu, Matthias Klusch, and Sascha Ossowski.  e COOP: privacy-preserving dynamic coalition formation for power regulation in the smart grid.  In Proceedings of the Second International Conference on Agreement Technologies, AT 2013, Beijing, China, August 1-2, pages 19–31, 2013\.
  * [25] Radu-Casian Mihailescu, Jan A. Persson, Paul Davidsson, and Ulrik Eklund.  Towards collaborative sensing using dynamic intelligent virtual sensors.  In Intelligent Distributed Computing X - Proceedings of the 10th International Symposium on Intelligent Distributed Computing - IDC 2016, Paris, France, October 10-12 2016, pages 217–226, 2016.
  * [26] Sascha Ossowski and Ronaldo Menezes.  On coordination and its significance to distributed and multi-agent systems.  Concurrency and Computation: Practice and Experience, 18(4):359–370, 2006.
  * [27] Mike P. Papazoglou and Willem-Jan Heuvel.  Service oriented architectures: Approaches, technologies and research issues.  The VLDB Journal, 16(3):389–415, July 2007.
  * [28] C. Perera, A. Zaslavsky, P. Christen, and D. Georgakopoulos.  Context aware computing for the internet of things: A survey.  IEEE Communications Surveys Tutorials, 16(1):414–454, 2014.
  * [29] Per Persson and Ola Angelsmark.  Calvin – merging cloud and IoT.  Procedia Computer Science, 52:210 – 217, 2015.
  * [30] Ferdinand Piette, Cédric Dinont, Amal El Fallah Seghrouchni, and Patrick Taillibert.  Deployment and configuration of applications for ambient systems.  In Proceedings of the 6th International Conference on Ambient Systems, Networks and Technologies, ANT-2015, pages 373–380. Procedia Computer Science, 2015.
  * [31] Ronald Poppe.  A survey on vision-based human action recognition.  Image and Vision Computing, 28(6):976 – 990, 2010.
  * [32] Davy Preuveneers, E Berbers, Davy Preuveneers, and E Berbers.  Automated context-driven composition of pervasive services to alleviate non-functional concerns.  International Journal of Computing and Information Sciences, 3:28–38, 2005.
  * [33] R. de Lemos et al.  Software engineering for self-adaptive systems: A second research roadmap.  In Rogério de Lemos, Holger Giese, Hausi A. Müller, and Mary Shaw, editors, Software Engineering for Self-Adaptive Systems II, volume 7475, pages 1–32. Springer-Verlag, 2013.
  * [34] Mohammad Abdur Razzaque, Marija Milojevic-Jevric, Andrei Palade, and Siobhan Clarke.  Middleware for internet of things: A survey.  IEEE Internet of Things Journal, 3(1):70–95, 2016.
  * [35] Christof Roduner, Marc Langheinrich, Christian Floerkemeier, and Beat Schwarzentrub.  Operating appliances with mobile phones–strengths and limits of a universal interaction device.  In Pervasive Computing, pages 198–215. Springer, 2007.
  * [36] Maria J. Santofimia, Francisco Moya, Felix J. Villanueva, David Villa, and Juan C. Lopez.  An agent-based approach towards automatic service composition in ambient intelligence.  Artificial Intelligence Review, 29(3):265–276, 2009.
  * [37] R. G. Smith.  The contract net protocol: High-level communication and control in a distributed problem solver.  IEEE Transactions on Computers, C-29(12):1104–1113, 1980.
  * [38] Thanos G. Stavropoulos, Dimitris Vrakas, Danai Vlachava, and Nick Bassiliades.  BOnSAI: A smart building ontology for ambient intelligence.  In Proceedings of the Second International Conference on Web Intelligence, Mining and Semantics, WIMS ’12, pages 30:1–30:12, New York, NY, USA, 2012. ACM.
  * [39] The Fundation of Intelligent Physical Agents.  FIPA specifications.  Technical report, 2002.  http://www.fipa.org/repository/standardspecs.html.
  * [40] Michael Wooldridge.  An Introduction to MultiAgent Systems.  Wiley Publishing, 2nd edition, 2009.
  * [41] L. D. Xu, W. He, and S. Li.  Internet of things in industries: A survey.  IEEE Transactions on Industrial Informatics, 10(4):2233–2243, 2014\.
  * [42] Zheng Yan, Peng Zhang, and Athanasios V. Vasilakos.  A survey on trust management for internet of things.  Journal of Network and Computer Applications, 42:120 – 134, 2014\.


# Perspectives on Negative Research Results in Pervasive Computing

Ella Peltonen  University of Oulu, Finland  Nitinder Mohan  Technical
University Munich, Germany  Peter Zdankin  University of Duisburg-Essen,
Germany  Tanya Shreedhar  IIIT-Delhi, India  Tri Nguyen  University of Oulu,
Finland  Suzan Bayhan  University of Twente, The Netherlands  Jon Crowcroft
University of Cambridge, United Kingdom  Jussi Kangasharju  University of
Helsinki, Finland  Daniela Nicklas  University of Bamberg, Germany

(2022)

###### Abstract

Not all research leads to fruitful results; trying new ways or methods may
surpass the state of the art, but sometimes the hypothesis is not proven or
the improvement is insignificant. In a systems discipline like pervasive
computing, there are many sources of errors, from hardware issues over
communication channels to heterogeneous software environments. However,
failure to succeed is not a failure to progress. It is essential to create
platforms for sharing insights, experiences, and lessons learned when
conducting research in pervasive computing so that the same mistakes are not
repeated. And sometimes, a problem is a symptom of discovering new research
challenges. Based on the collective input of the First International Workshop
on Negative Results in Pervasive Computing (PerFail 2022), co-located with the
20th International Conference on Pervasive Computing and Communications
(PerCom 2022), this paper presents a comprehensive discussion on perspectives
on publishing negative results and lessons learned in pervasive computing.

Research in fields such as pervasive computing, networking, systems, etc., is
usually driven by results that push forward (or improve efficiency of) the
state-of-the-art or provide demonstrable contributions. To achieve novel and
practical solutions, researchers often invest significant efforts into
designing and evaluating several iterations of design choices with hopes of
achieving the desired performance. However, the probability of such efforts
leading to a negative output can be quite high due to a lack of complete
understanding/hindsight or unconvincing design choices, some of which may be
harder to rectify at later stages. Such efforts are rarely rewarded since
negative results are usually hard to publish [1, 2], being considered not
novel or lacking significant new knowledge. However, we argue that negative
results, if properly leveraged, can be quite beneficial to the research
community at-large in the perspective of lessons learned and knowledge
transfer between the research groups tackling the same topics. While our
arguments in this article are from the perspective of pervasive computing
research, many of our insights are valid in other fields of computing, such as
human-computer interaction, network/mobility measurements, security and
privacy, etc., thanks to the inherently interdisciplinary nature of the
pervasive computing.

Most pervasive computing systems require multiple, often non-standard
components, like hardware prototypes for sensing, developing computation
paradigms, novel energy-saving communication protocols, adaptive middleware,
or multi-modal user interfaces. Often, a new research idea or hypothesis
covers only one part of the system, but to complete the experiment, you must
(at least partially) get everything to work. Each component might be a source
of error or the bottleneck that prohibits the expected result. However,
sometimes the interplay of different components highlight the symptom of an
underlying, more significant challenge that could lead to new research ideas.
Hence, in this article, we assert that we _must_ treat negative results as a
potential source for new scientific insights.

While negative results might not directly contribute to the advancement of the
state-of-the-art, the wisdom of hindsight could be an essential contribution
in itself, such that other researchers could avoid falling into similar
pitfalls. Such “what not to do?” insights can also foster good research
practices, especially for handling gray areas like ethical boundaries [3]. We
consider negative results to be outcomes of studies that are run correctly (in
the light of the current state-of-the-art) and in good practice but fail in
proving the hypothesis (statistically significantly). The “badness” of the
work might also emerge from properly but unfittingly designed data collection
or (non-trivial) lapses of hindsight, especially in studies involving real-
world uncontrolled measurements. Furthermore, the experiences that help spot
negative outcomes from intermediate results benefit the systems and networking
community at large.

Besides, it is crucial to share (especially for junior researchers) the
knowledge that not all experiments will be successful, failures might happen,
and knowing how to overcome unexpected outcomes is a skill to develop as a
researcher. The interest in this field was evident from the participation in
the questionnaire for Percom 2022 participants, in which $\approx$70 attendees
participated. The result of the questionnaire revealed that $\approx$85% of
the conference participants already failed in their research work. The
perspectives noted in this article are drawn from discussions with the
pervasive computing community during the First International Workshop on
Negative Results in Pervasive Computing (PerFail 2022)111https://perfail-
workshop.github.io/2022/ and from authors’ past experiences. Specifically,
this paper discusses failure taxonomy in pervasive computing, considers
mitigation and avoidance strategies, and presents our call-for-arms for
healthy failure culture in pervasive research.

## What is a Failure?

Traditional applied research aims to fundamentally understand how systems work
or how to make them work. Within the pervasive computing, systems and
networking research, solutions that _demonstrably_ work are naturally more
attractive to the community. Therefore, it becomes essential to gauge the
usefulness of research that failed to work. Failure can be viewed as a
necessary catalyst to research – if there is no possibility to fail, the
novelty of the research is also questionable.

Within the research ecosystem, the definition of “failure” is often
misconstrued, which makes many researchers incorrectly label unexpected
outcomes of research studies as failures. Since the underbelly of the
ecosystem is somewhat driven by “publish or perish” ideology [4], researchers
tend to inherently add significant weight on the number (and quality of venue)
of publications. However, we argue (on logical grounds) that publications are
means for disseminating new-found results; hence failing to publish in a
competitive venue or securing funding should not be considered as failed
research. While the scientific community inadvertently uses certain criteria
to measure a researcher’s success – citations, publications at high-impact
venues, funding acquired – these may not fully reflect the success of the
conducted research. Instead, the criteria reflect how the output is perceived
by the scientific community in the short term. For example, a publication
about a specific failure may be successful on its own by preventing others to
repeat said failure, hence having a big impact on the research that is
performed. But this publication has a low chance of being cited in the
hopefully successful research that it lead towards. As such, we first discuss
different categories of results obtainable from research studies.

### Categories of Research Results

I. Positive results are the results that show the validity of the research
hypothesis. Studies with positive results are more likely to be cited,
indicating a bias in community in publishing works showcasing positive results
[5, 6].

II. Negative results are the results that contradict previously-established
knowledge (e.g., lack of statistical significance to prove the hypothesis of
these earlier studies or statistically-significant results contradicting the
earlier positive results). These negative results, if not emerging due to some
flaws in
research222https://www.nature.com/articles/nature.2012.10249,333https://www.nature.com/articles/d41586-019-01675-9,
can play a significant role in advancing science and avoiding waste of
resources (e.g., researchers’ time or public money). Hence, before disclosing
such results, it is crucial to have confidence that the research is rigorously
designed and executed [7]. Additionally, negative results can prove a unique
collaboration opportunity for involved research groups to understand the
possible reasons for the discrepancy between the obtained results. Negative
results are also related to reproducibility. Recreation of previous studies
might be affected by uncontrollable configuration changes, design and
assumptions, which can lead to different conclusions. Therefore, it is
paramount to follow open science practices to facilitate easier investigation
of the root causes of discrepancy [8]. According to a small-scale survey with
96 researchers found that the majority of the respondents saw value in sharing
the negative research outcomes, but do not attempt publishing due to
considerable time and effort investment with potentially less number of
citations compared to the studies with positive results [9].

III. Unexpected or unexplainable results are results that cannot be explained
with the hypothesis being investigated, requiring further exploration and re-
assessment of problem formulation, assumptions, or design choices. Unexpected
results can be likely cause of flaws in methodology (e.g. software errors) or
unrealistic assumptions, but might also emerge due to the complex system
interactions not captured by the hypothesis or designed experiments. For
example, in pervasive computing, many factors from the used sensors, their
calibration and the composition of study participants play a significant role
and can lead to a mismatch between expected and obtained results.

Not all research studies contain results that can be proved or disproved.
Vision papers raise important questions pointing to new possibilities, and
draw an agenda pioneering a field or vision for researchers to pursue. For
opening new avenues and out-of-box thinking beyond the state-of-the-art,
vision papers such as [10] play a crucial role in advancing science.
Similarly, survey studies and tutorial papers can provide a good understanding
of the field and assist researchers in identifying the knowledge gaps.

### The Cycle of Research

Figure 1: The cycle of research.

We now discuss the cycle of research that can potentially lead to such
different achievable results from a study, illustrated in Fig. 1. To conduct
research that advances the field and is impactful, researchers should raise
the “right” questions to work on. Finding good research questions that fill a
knowledge gap may be challenging, as research directions that lie at the
intersection of being solvable and pertinent to the community can be limited
or require a deep understanding of the literature. At this stage of
formulating the research questions, apart from performing extensive literature
review with an inquisitive mind, researchers can benefit from discussing
hypotheses with colleagues (in addition to advisors) or in discussion forums
such as Ph.D. schools, workshops, and community meetings.

Note that performing actions within this stage in conjunction with the next
stage of the cycle, i.e., plans to execute the research, is likely to yield
desired positive outcomes. For example, suppose the research question involves
understanding user behavior over certain stimuli. In that case, it is
beneficial to prototype design/methodology for conducting the study, enumerate
parameters that need to be recorded and the expected results from the
campaign. It is pertinent that researchers are not rigid with their initial
assessment of the suitability of the research questions and means to achieve
it to avoid cognitive bias – that can significantly impact the outcome of the
research. As such, they should be willing to consider preliminary feedback and
_re-assess_ their approach, formulation and (if required) research direction –
since re-aligning at these initial stages can maximize research impact with
the least negative consequences.

Once the research formulation is finalized, researchers should periodically
assess the intermediary results from iterative prototypes for revisions, if
necessary. Most importantly, it is essential to keep in mind that the
experiments can likely yield negative or unexpected results at the end of the
study. At this stage, researchers must discover the reason behind their
unexpected results – is it due to wrongful formulation or lack of collected
parameters, or is it challenging the widespread assumption within the field?
As such, a negative outcome can result in a successful study with a
breakthrough, albeit a strong validation proves no shortcomings/bias in
methodology and problem formulation. Similarly, a positive result from a study
does not immediately equate to a successful outcome and requires statistically
significant validation to prove no bias in the experiment formulation.

In the next step of the cycle, the researchers usually formulate their
findings into research articles and submit them to peer-review venues. While a
significant portion of the scientific community questions the accuracy of the
current peer-review systems, often terming it to be “broken” [4], the core
utility of the process is to ensure that the submitted research is free from
overlooked biases/errors and incorrect information.

It must be noted that despite widely accepted notions, publications are not
the _goal_ of a research study but merely means for disseminating the new-
found knowledge or experiences gained during the course of the research.
Successful research can also produce practical systems and artifacts which do
not require traditional peer-review conference/journal publications but open-
source code with descriptive documentation, website or even blog post. As a
result, researchers should aim to engage with the community in alternate
methods (in addition to research publications) to receive a holistic peer
review and feedback on improving the research. For example, networking and
systems research outcomes can include RFC drafts to be discussed in venues
like IETF/IRTF. Researchers can give formal or informal talks at conferences
and research institutes and advertise early results in mailing lists,
community forums, Slack channels, and so on.

Expert peer review of any form (traditional publication as well as via
alternate means) must be considered with due diligence, which might uncover
earlier overlooked sources of errors in the research methodology. As a result,
the researchers should be open to _revise_ their research methodology and
formulation to address correctness concerns.

Despite repeated improvements, it might be possible for the manuscript to be
repeatedly rejected from multiple publication venues for various reasons. The
phenomenon is often misrepresented as a “failed research”. However, as
previously stated, failing to publish a paper at the desired venue does not
necessarily point to insufficient quality of the research study. Different
publication venues have different approaches and understanding of the
methodology and research approaches; hence, choosing the venue correctly and
ensuring that the manuscript is well-written can be crucial.

Researchers should also consider submitting to less competitive venues to
maximize the impact and timeliness of a research study. For example, a study
that facilitates further research with more efficiency (e.g., via developing a
measurement tool) can be as impactful as studies with groundbreaking ideas.
Moreover, the impact of many studies can become evident over time with the
advances in supporting sciences and technology. It might turn out that peer-
review feedback or methodology revisions reveal that research has failed or
obtained results are insignificant. Sharing lessons learned from such failed
experiments can also help advance the understanding of the field as others can
avoid similar pitfalls that impacted the study. However, care must be taken to
accurately summarize the sources of errors/learning and potential future
directions for improvement which must not be mistaken with publishing
incorrect results. If the researchers correct the study’s errors, such
learnings can be shared in the appendix of the accepted paper for
completeness. The researchers can also look for venues that actively publish
and discusses failure results catering to their research field, e.g. PerFail,
Insights, and JOURNE. Knowing the possible failure points in the research
cycle, knowing the right community for the work in question, and writing an
article that is acceptable by the relevant community only comes by
experiencing research. As a result, learning from failures is one of the key
skills supervisors can deliver to their juniors.

Table 1: Summary of potential failure points at different stages of research study. | Failure Points | Recommendations
---|---|---
Problem Formulation |  | Wrong hypothesis
---
Being ahead of time
Unclear research questions
Low relevance to venue
Lack of novelty or delta increment to existing works
| Clear demarcation from literature
---
Vision papers
Extensive literature review
Study Design and Execution |  | Theory vs. practice
---
Unrealistic assumptions
Too simple or too complex abstraction
| A clearly-described methodology section
---
Citing relevant resources
Detailed model descriptions
Measurements |  | Software/Hardware errors
---
Human/sensors errors
| Avoiding ambiguity
---
Open datasets
Community accepted methodology
Performance Analysis | Wrong metrics and methodology | Comparison with state-of-the-art
Publication | Repeated rejections | Re-assess relevance and publication venue

## Failure Taxonomy

In this section, we investigate the process of scientific failures. We
identify what is a failure and its key characteristics that hint at research
outcomes that might fail. Recognizing these specific situations in research
might be an early warning sign that a particular research direction might
fail. We present methods and preventive actions that can be taken to avoid and
mitigate possible failures. The measures include changing the research
direction or methodology early on, gathering insights, and publishing the
negative results if the research fails at a later stage.

### The Origin of Failures

Research should start with a clear problem statement and a verifiable
hypothesis, as a lack of either is a well-known predetermined breaking point
for the subsequent research. One of the most critical steps in producing
“good” research is a comprehensive understanding of the problem that is to be
solved. If the problem is not understood well enough, it is possible to
research something that lies close to a problem and perhaps is an interesting
field on its own but fails to produce fruitful results that can be published.
While open-ended research in a preliminary phase is useful to collect ideas,
it eventually should be fitted into a more solid framework with clearly-
defined goals. Experience says that the sooner the failure is identified, the
easier it is to fix.

Related work is critical to advance in science, as this step encourages
findings on this topic made by other researchers. This could include positive
or negative results on a similar topic, to either be a warning of spending
time and effort on a problem that is either unsolvable or already solved. Even
more important, the time frame of scanning the related work should not be too
small, and perhaps 10, 20, or more years ago, a problem was already solved.

### New Research Topics

It is easy to focus on the wrong part of the problem or misunderstand the
significance of the results, either because you are looking at incomplete or
incorrect information or trying to interpret different things from the results
from what they actually show. Being early on a topic, especially a topic that
later becomes _hot_ , is what all researchers dream of doing. But being early
on a new topic means there is not an established community around, so there
might be no general understanding of the significant results or the critical
questions in this area. This might seem counter-intuitive because research is
about discovering new things, and somebody always has to write the first paper
on a new topic. There is an excellent scope of vision papers in such
scenarios.

### Failure Types

Failures come in all kinds of shapes and forms (see Table 1). Following
previous overviews of the experiences and lessons learned while conducting
research [11, 12], we identify the following set of failure types relevant in
pervasive computing field:

  * •

Unconvincing results which could not be verified because it is not even
possible, for example, due to lack of datasets, real-life experiments, or
ground truth.

  * •

Under or over-performing experiments that could have been possibly fixed, for
example, oversights in system design, inadequate or misconfigured
infrastructure, or just buggy code.

  * •

Setbacks resulting in lessons learned and acquired hindsight, including
hypotheses with too limiting or too broad assumptions.

  * •

Unconventional, abnormal, or controversial results that contradict community
expectations and are thus extremely hard to reliably prove.

  * •

“Non-publishable” or “hard-to-publish” side-outcomes of the otherwise
correctly run studies, including mistrials of experiment methodology or
design, preparations for proof-of-correctness of results, and so on. Limited
contribution might not be a failure itself, but might become hard to publish
as a full research article.

  * •

Unexpected roadblocks affecting publications, including ethical concerns,
institutional policy breaches, and difficulties in collaboration with
commercially-bound parties of the research projects. Ethical and privacy
hurdles might only be encountered after a series of measurements have already
been conducted, forcing an unwanted halt to the entire research, including any
observations made prior to any private data being collected.

## Failure-Avoidance and Mitigation

In previous sections, we have discussed where, when, and how failures in
pervasive computing research may occur. Next, we discuss methods and best
practices to avoid failures while conducting research. The studies focusing on
simulations, performance analysis, and the more technical side of the field
are not free from typical problems either. These topics were also covered
during the PerFail workshop [13, 14, 15, 16]. Some lessons learned discussed
in the workshop include:

  * •

Use step-wise study procedures, failures and setbacks are easier to recognize
during the study.

  * •

Design for the target group and consult domain experts, especially if the
study concerns human subjects. Seek feedback from domain experts for any
modification to the questionnaire.

  * •

Prioritize collecting high-quality data in the first-hand. Consider the
quality of labeling the data items and events, especially when collecting the
data in the wild. The sensor devices have to ensure the quality of the signal,
even though existing noise.

  * •

Ensure validity of the ground truth data by considering “gold standard”
devices, especially with sound studies evaluating self-gathered field data
with smartphones and wearable devices. The ground truth data must be validated
in the lab and in-the-wild environments. Data collection can be monitored via
a dashboard to filter noise and abnormality.

  * •

Avoid ambiguous questionnaires. If the study includes real people, accommodate
your research for the situation. If using questionnaires, create them in a
self-explainable way to avoid misunderstandings. Keep studies and
questionnaires as short as possible to improve acceptance and participation
rates.

  * •

Relation between data selection and experimental plans. Based on the
requirements and practical techniques, selecting a fit dataset affects the
final results of the research.

  * •

Analyze previous methodologies for deep comparison and understanding. After
deciding on a fit dataset, understanding the dataset needs an effort in
analyses from earlier works for data processing and methodologies.
Understanding the dataset for experiments is to avoid a lack of important
features that fit the experimental requirements.

  * •

Consider resources related to the experiments. The experimental results mainly
indicate the boost in a feature; however, the trade-off from other aspects or
resources needs to be carefully considered, especially when transforming
methodologies into a new environment.

Additionally, a careful consideration of ethical guidelines is essential [17].
This concern may not only protect the integrity of subjects of a study but may
also prevent the halt of entire studies. To better safeguard any research
involving personal data, the use of Institutional Review Boards is strongly
advised and even often required by outlets [18].

Exciting and risky research will necessarily have more failures than
successes; this needs to be understood and normalized in scientific
publications. Unlike industry and startup projects, not every research project
must yield a success story, as even the insight of a methodology not working
for a given problem is a revelation on its own. There are some research
directions that are both risky and expensive, where a registered experiment
publication (e.g., RSOS444https://royalsocietypublishing.org/journal/rsos) or
any publication about negative results is vital to prevent such sunk costs
from happening again.

In addition, it is notorious that there are unexpected outcomes or a
hypothesis that does not hold [19]. In these cases, it is crucial to show the
importance of negative results and lessons via parameters, including the
design of experiments, evaluation of performance, comparison of models,
statistical hypothesis testing, and so on. It needs to be emphasized that
there is a difference between negative results and no results; that the no-
results indicate incomplete or unreliable works. Meanwhile, the negative
results contain valuable information; they should be welcomed to save efforts
in redundant works and contribute intellectual side related to models,
methods, and datasets.

### When to Fix Failures

During the research work or latest in the review phase, the modifications in
the original manuscript are still possible and even desirable for the
readability and cohesion of the article. However, identifying the failures
after the manuscript has passed the review cycle and got published is not only
more than welcome but also crucial for the reliability of the research.
Publishing a retraction, errata, or follow-up are all critical tools to avoid
just “living with” an error that could have been fixed.

Failure should obviously be avoided despite the pressure to publish and
approaching deadlines. This makes lessons learned so critical – learning from
one’s own experience and the community as a whole. “Intelligence is the
ability to adapt to change”, which also applies to the research plans and how
it actually turns out. This means that research work should be refactored in
even late stages if preliminary results necessitate this action.

## A Call-for-Arms for Healthy Failure Culture in Research

Even the most experienced researchers can come with a research question and
hypothesis that proves wrong. If the outcome is not expected or aimed for, is
this a failure? More importantly, how should it be reported in the community
that the first hypothesis did not work? Publishing failures results enable
others to avoid making the same mistakes, yet there are significantly more
non-working solutions than good, efficient ones.

Most of research is what is commonly called “incremental”, i.e., it involves
small perturbations of previous research. Done correctly, it entails
reproducing the earlier results and adding some variation. Even if the
variation does not yield the expected results, the work has successfully
reproduced the earlier work, and thus has validated that work. In our opinion,
this is definitely not a failure. Unfortunately often such “failed” variations
are reported as successes, by sugarcoating the results or trying to spin them
as positive results. Partly this is because we, as a research community,
expect good and positive results, so we should be more tolerant and recognize
that reproducing earlier results also carries significant value.

Some research aims at creating something radically new and different; in most
cases, these end up failing. Some funding agencies, e.g., European Research
Council, explicitly call for high risk, high gain types of proposals, while
others focus (implicitly) on more incremental work with more guaranteed but
smaller successes. We should understand that both kinds of research have value
and that failures may simply be indicators of the level of ambition. To
propagate a healthier failure culture in research, we propose a call-for-arms
action in the community (with a broader relevance outside pervasive
computing).

### “Guardians of Publication Quality”

The review process itself is not free of criticism. What will be accepted is,
by human nature, dependent on the opinion of reviewers, accompanied by the
game of chance to get the reviewers assigned that view the publication most
positively [4]. The same game of chance is present in the unregulated nature
of paper reviews, which allows for claims and unfairness by the reviewers if
the editors do not secure uniform quality reviews.

The task of a reviewer or editor _should not_ be about improving the prestige
of the publication venue, but instead checking the correctness of the work -
to the extent feasible. Of course, not all problems in the review process are
due to “malicious” reviewers. The difficulties are real in conveying the
applicability of studies regardless of their limited metrics and comparing
publications against each other, as the performance metrics applied to decide
the fate of one submission can not be used for another. In addition, there is
a constant need for novel openings instead of building on old theories,
despising potential but risky new research fields. Highly selective
conferences tend to favor “exciting” results, leaving little to no room for
the “correct but boring” work that has been done rigorously yet falls under
the not-so-successful variation of earlier work.

Yet, an interesting question is how much publication space should failures or
negative results get? If the result refutes earlier work, it deserves its spot
in the limelight. However, if it simply highlights that a particular
variation/configuration did not work, how should it be communicated? Should
publication venues dedicate some space to “failures”, dictating authors to
report such unexpected results in appendix or a specifically reserved section?
Or should we have dedicated publication venues for publishing such negative
results? The former allows readers to learn the underbelly evolution of the
work alongside positive results, providing a new learning perspective to the
research study. On the other hand, the latter would give a general prominence
to “things that did not work” in various open problems in the field,
encouraging people to take more risks, knowing they have a good chance of
advancing the state-of-the-art through unexplored approaches.

### Normalize Failures as Researchers

Our aim, through this paper, is to normalize failures as part of the research
process and motivate their means for communication. Optimally, this would be
through encouraging publication venues to integrate lessons learned from
negative results as part of their program. Other means, such as social media
or reports on arXiv could also serve as a venue for bringing information about
negative outcomes to the public. We also encourage the research community to
organize workshops, PhD forums, and other early-stage work events for getting
feedback and sharing information on how different researchers – also those in
senior positions – have overcome the setbacks they have faced.

Communicating stories of the research process as a whole should be seen as
equally important as communicating the final results. This could be a section
at the end of the article, similarly to “limitations” section currently
included in the discussion of the results. In addition, communicating the
story of the work, from the initial hypothesis through failures to the final
decision, should become a standard when discussing the results in conferences
or sharing results on (research) social media, such as Twitter, ResearchGate,
or LinkedIn. Here, encouraging senior researchers as role models to
communicate about setbacks and failures – and how they did or did not overcome
them – is crucial. Some could even consider publishing a list of rejections
along with acceptances. Normalizing failure could even encourage researchers
to take more risks in research without the fear of not having to show
anything.

## Conclusion

Failures and setbacks in research work are natural and will occur for
everyone, from junior PhD students to senior professors. In pervasive
computing research, reasons for failures can become apparent quite late and
might be unrecoverable, including overlooked aspects in data gathering or
instrumentation, wrong assumptions, trust, and ethical considerations. Causes
for failures can be common but relatively unknown as published research tends
to highlight novel findings and overlook negative results, setbacks, and
stories of difficulties. Sometimes writing and getting negative results
published can be, indeed, much more difficult than publishing a paper with
minor incremental results.

As we have highlighted in this paper, venues for discussing experiences from
failures must become more mainstream. Shared experiences and lessons learned
will help others to avoid failures; this should be done by presenting setbacks
and talking to other members of the community starting from the early stages
of the study. It would be exceptionally important to have a platform where
people can publish the stories behind successful papers describing the
problems they ran into. This could also be a specific section in each paper
for failed trials, similarly to how limitations are currently discussed. For
our own part, we will continue running the PerFail workshop on negative
results in pervasive computing. We encourage increasing the number of similar
workshops also in other fields of the computer science.

## ACKNOWLEDGMENT

Authors are thankful for the participants of the First International Workshop
on Negative Results in Pervasive Computing (PerFail2022) and the organizers,
especially workshop chairs, of the 20th International Conference on Pervasive
Computing and Communications (IEEE PerCom 2022). The paper only reflects on
opinions of the authors.

## References

  * [1] N. Matosin, E. Frank, M. Engel, J. S. Lum, and K. A. Newell, “Negativity towards negative results: a discussion of the disconnect between scientific worth and scientific culture,” pp. 171–173, 2014.
  * [2] D. Fanelli, “Negative results are disappearing from most disciplines and countries,” _Scientometrics_ , vol. 90, no. 3, pp. 891–904, 2012.
  * [3] A. Mlinarić, M. Horvat, and V. Šupak Smolčić, “Dealing with the positive publication bias: Why you should really publish your negative results,” _Biochemia medica_ , vol. 27, no. 3, pp. 447–452, 2017\.
  * [4] P. M. Dahlgren, “Instructions for reviewer 2: How to reject a manuscript for arbitrary reasons,” May 2022. [Online]. Available: osf.io/t8jsm
  * [5] S. Gaillard and S. Devine, “Systemic Problems in Academia: The Positive Publication Bias and Solutions from a Human Factors Perspective,” _Journal of Trial and Error_ , vol. 2, no. 1, 2022, https://assets.pubpub.org/ydsc2fci/11647096441542.pdf.
  * [6] S. Nimpf and D. A. Keays, “Why (and how) we should publish negative data,” https://doi.org/10.15252/embr.201949775, accessed: 2022-05-24.
  * [7] A. Bespalov, T. Steckler, and P. Skolnick, “Be positive about negatives–recommendations for the publication of negative (or null) results,” _European Neuropsychopharmacology_ , vol. 29, no. 12, pp. 1312–1320, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0924977X19317195
  * [8] V. Bajpai, A. Brunstrom, A. Feldmann, W. Kellerer, A. Pras, H. Schulzrinne, G. Smaragdakis, M. Wählisch, and K. Wehrle, “The dagstuhl beginners guide to reproducibility for experimental networking research,” _SIGCOMM Comput. Commun. Rev._ , vol. 49, no. 1, p. 24–30, feb 2019. [Online]. Available: https://doi.org/10.1145/3314212.3314217
  * [9] Lucia Echevarria, and Alberto Malerba, and Virginia Arechavala-Gomeza, “Researcher’s Perceptions on Publishing “Negative” Results and Open Access,” _Nucleic Acid Therapeutics_ , vol. 31, no. 3, pp. 185–189, 2021, accessed: 2022-05-24.
  * [10] D. L. Tennenhouse and D. J. Wetherall, “Towards an active network architecture,” _ACM SIGCOMM Computer Communication Review_ , vol. 37, no. 5, pp. 81–94, 2007.
  * [11] N. Howell, A. Desjardins, and S. Fox, “Cracks in the success narrative: Rethinking failure in design research through a retrospective trioethnography,” _ACM Trans. Comput.-Hum. Interact._ , vol. 28, no. 6, nov 2021. [Online]. Available: https://doi.org/10.1145/3462447
  * [12] S. Honig and T. Oron-Gilad, “Understanding and resolving failures in human-robot interaction: Literature review and model development,” _Frontiers in Psychology_ , vol. 9, 2018. [Online]. Available: https://www.frontiersin.org/article/10.3389/fpsyg.2018.00861
  * [13] B. van Berlo, T. Ozcelebi, and N. Meratnia, “Insights on mini-batch alignment for wifi-csi data domain factor independent feature extraction,” in _2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)_ , 2022\.
  * [14] A. Das, Y. D. Kwon, J. Chauhan, and C. Mascolo, “Enabling on-device smartphone gpu based training: Lessons learned,” in _2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)_ , 2022.
  * [15] M. Heck, J. Edinger, and C. Becker, “Lessons learned from an eye tracking study for targeted advertising in the wild extraction,” in _2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)_ , 2022.
  * [16] S. Liaqat, D. Liaqat, T. Son, A. Gershon, M. Gabel, R. Wu, and E. de Lara, “Hindsight is 20/20: Retrospective lessons for conducting longitudinal wearable sensing studies,” in _2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)_ , 2022.
  * [17] a. s. franzke, A. Bechmann, M. Zimmer, and C. Ess, “Internet research: Ethical guidelines 3.0,” _Association of Internet Researchers_ , 2020.
  * [18] D. R. Thomas, S. Pastrana, A. Hutchings, R. Clayton, and A. R. Beresford, “Ethical issues in research using datasets of illicit origin,” _Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC_ , pp. 445–462, 2017.
  * [19] A. Borji, “Negative results in computer vision: A perspective,” _Image and Vision Computing_ , vol. 69, pp. 1–8, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0262885617301671