# Efficient micro data centers deployment for mobile healthcare monitoring
systems in IoT urban scenarios

Kevin Henaresa, José L. Risco-Martína, José L. Ayalaa and Román Hermidaa
CONTACT José L. Risco-Martín. Email<EMAIL_ADDRESS>aDept. of Computer
Architecture and Automation, Complutense University of Madrid, Calle Prof.
José García Santesmases, 9, 28040 Madrid, Spain

###### Abstract

In the last decade, the Internet of Things paradigm has caused an exponential
increase in the number of connected devices. This trend brings the Internet
closer to everyday activities and enables data collection that can be used to
create and improve a great variety of services and applications. Despite its
great benefits, this paradigm also comes with several challenges. More
powerful storage and processing capabilities are required to service all these
devices. Additionally, the need to deploy and manage the infrastructure to
efficiently support these resources continues to pose a challenge. Modeling
and simulation can help to design and analyze these scenarios, providing
flexible and powerful mechanisms to study and compare different strategies and
infrastructures. In this scenario, Micro Data Centers (MDCs) can be used as an
effective way of reducing overwhelmed Cloud Data Center infrastructures. This
paper explores an M&S methodology to study the overall power consumption of a
healthcare IoT scenario. The patients wear non-intrusive monitoring devices
that periodically generate tasks to be executed in MDCs. We extract the layout
of existing urban infrastructures, simulate the monitored population’s
behavior, and compare the power consumption of several data center
configurations.

###### keywords:

Internet of Things; model-based systems engineering; edge and fog computing;
discrete-event system specification; clustering

††articletype: ARTICLE TEMPLATE

##  1 Introduction

The data generation ratio worldwide is increasing exponentially over the
years. In every scope, the domain-specific information is progressively taken
into account to create valuable knowledge that can help us understand our
reality and make our procedures and technologies more effective and efficient.
According to the International Data Corporation (IDC), worldwide created,
captured, and replicated data will grow to 175 zettabytes by 2025. Since 2018
(in which 33 zettabytes were registered), this represents an increment of 530%
and a compound annual growth rate of 61% (Reinsel ., 2018).

This increment has been favored by the growing number of connected devices and
the development of new and improved IoT ecosystems. IDC predicts that by 2025
there will be 55.7 billion connected devices worldwide, 75% of which will be
connected to an IoT platform. They estimate data generated from connected IoT
devices to be 73.1 ZB by 2025, growing from 18.3 ZB in 2019 (Afuang, 2020).
Moreover, this growth in the total number of IoT devices is projected to
provide substantial economic and social benefits in the way of cost savings,
value creation, and productivity improvements (Castillo O’Sullivan Thierer,
2015). Improved industrial monitoring and automation techniques will help to
minimize failures, reduce waste, and optimize processes. Smart cities are
expected to contribute to this trend, paving the way for the introduction of
intelligent and unmanned transportation (Tokody ., 2017) and optimized
deliveries (F. Wang ., 2019), and optimizing their infrastructures (Barba .,
2012) (Masera ., 2018) (Dey ., 2000).

In healthcare, accurate patient monitoring and pharmaceutical management,
added to predicting risk factors in highly-impact diseases, will result in
substantial cost savings. These predictions will be facilitated by the
implementation of Wireless Sensor Networks (WSNs), where patients with
specific target diseases wear unintrusive wearable devices that allows to
continuously monitor patients state, registering variables like heart rate,
Electrocardiogram (ECG), electrodermal activity, Electroencephalography (EEG),
or oxygen saturation. These variables, related to the autonomic nervous system
(ANS), can be modeled for associating them with specific disease symptoms or
outcomes, generating useful predictions for patient diagnosis and treatment
(Pagán ., 2015; Ali ., 2020).

These scenarios come accompanied with a need for storage infrastructures and
computing capabilities. This trend has led to the increasing use of data
centers to store and process data traditionally located at endpoints. Cloud
computing is one of the enabling platforms to support these needs. The
National Institute of Standards and Technology (NIST) (Mell Grance, 2011)
defines cloud computing as ”…a model for enabling ubiquitous, convenient, on-
demand network access to a shared pool of configurable computing resources
(e.g., networks, servers, storage, applications, and services) that can be
rapidly provisioned and released with minimal management effort or service
provider interaction”. Armbrust et al. (Fox ., 2009) simplify this concept
defining cloud as the “datacenter hardware and software that provide
services”. However, in addition to raw computing and storage, cloud computing
providers usually offer multiple software services, APIs, and development
tools that allow developers to build seamlessly scalable applications upon
their services (Voorsluys ., 2011). This Infrastructure as a Service (IaaS)
model approaches large and powerful cloud infrastructures to individuals,
allowing them to develop solutions to simplify and optimize different aspects
of everyday life through a new range of innovative services and applications.
Through its regular use, it is possible to improve the systems’ performance,
easily make use of these storing and processing capabilities, and reduce
systems overall cost.

However, there are also some drawbacks when using these services. Some
applications may not assume the latency derived from the intermediate
communications, or may be restricted by security or privacy concerns. In these
cases, Fog Computing is often used. This paradigm tries to solve these aspects
by making available these infrastructures to end-users that provides several
advantages. In addition to the lower latencies derived from the tightly
coupled infrastructures, it enables the use of mid-range IoT protocols and
reduces problems related to bandwidth bottlenecks. Also, the reliability of
the connection is increased due to the existence of multiple interconnected
channels. Some fog nodes include industrial controllers, Micro Datacenters
(MDCs), and video surveillance cameras.

All these technologies enable the development of connected healthcare
scenarios, where networked sensors are being placed either in the body or in
the living environments of patients to continuously extract different types of
data. These data, combining several technologies like Cloud Computing, Big
Data, the Internet of Things, and Machine Learning, are being aggregated and
processed to create valuable and meaningful insights regarding the patients’
lifestyle, habits, and health conditions. In this way, these proactive systems
are helping to evolve the traditional concept of medicine, favoring the
prognosis over the classical post-facto reactive paradigm (Saha ., 2017).
However, while incorporating all these technologies, healthcare scenarios are
becoming larger and more complex. They often include many heterogeneous and
dynamic components, that constantly evolve to incorporate new features over
time. Modeling and simulation (M&S) is progressively gaining acceptance in the
healthcare field for studying and tackling such complexity. M&S shows
excellent potential for comparing methodologies and scenarios, visualizing
workflows, or even monitoring and increasing the performance of health care
procedures through their integration with information system applications
(Gaba, 2007). Moreover, M&S can be used to simulate large amounts of
configurations in manageable times, which would be too expensive or infeasible
to execute in real-world settings (Erdemir ., 2020).

This paper uses these benefits to perform an M&S-driven analysis of the
energetic impact of MDCs geographical location in an urban healthcare
scenario. This scenario includes a population of migraine patients, being
continuously monitored by non-intrusive and portable devices. These devices
periodically send information to MDCs to retrieve predictions generated with
the corresponding patient-oriented models. These predictions can help migraine
patients avoiding the pain by taking measures before the pain starts.

Migraine is one of the most disabling neurological diseases. It affects around
10% of population worldwide (Linde ., 2012) and 15% in Europe (Stovner Andree,
2010). Apart from the pain phase, migraine disease includes other less-known
symptoms. Premonitory or prodromic symptoms may occur from three days to hours
before the pain starts (Giffin ., 2003). They are subjective, varied, and
include changes in mood, appetite, sleep, etc. Auras occur in one-third of the
cases (Olesen, 2004) and appear within 30 minutes before the onset of pain. It
consists of a short period of visual disturbance. Postdrome are symptoms that
occur after the headache. Some of the most common are tiredness, head pain, or
cognitive difficulties. They are present in 68% of the patients, and they have
an average duration of 25.2 hours (Kelman, 2006). Moreover, migraine sufferers
are more prone to suffer from other diseases such as fatigue, anxiety, or
cardiovascular problems, which leads to high costs for private and national
health systems. In Europe, it is estimated that migraine leads to direct and
indirect costs of €1,222 per patient per year (Linde ., 2012). It is difficult
to estimate the onset of pain to make an adequate intake of drugs. The time
response of the drugs’ pharmacokinetics (the mechanisms of absorption and
distribution of substances in an organism) does not match the long times of
the vague predictive symptoms, or the short times of the urgent auras. So,
most migraine sufferers wait until the onset of pain to take the rescue
medication. The delayed intake reduces the effectiveness of the treatment.

In the remainder of this article, Section 2 reviews existing work related to
healthcare monitoring simulation scenarios, and Cloud/Fog-related analysis and
processing methodologies and platforms. Section 3 gives further details about
the use case and the goals covered by this article. The results are then
discussed in Section 4. Finally, Section 5 present the conclusions and provide
some suggestions for future work.

##  2 State of the art

When designing and analyzing data-driven IoT scenarios, multiple factors must
be considered. Aspects such as the application’s placement, the system’s
architecture, the storing and processing resources, the communication networks
and protocols, the job allocation policies, or the energy constraints, may
have critical importance in the full service’s usability and performance. To
ease the development of such systems, several simulation frameworks have been
developed over the years. Section 2.2 describes some of the most popular IoT
simulation frameworks. Section 2.3 discusses several principles and approaches
chosen by other authors when developing this type of scenario.

###  2.1 M&S for the development of complex systems

Complex systems are usually understood as organizations including a large
number of interacting components. They often have a heterogeneous and dynamic
nature, and progressively increase their complexity over time. Some examples
of complex systems are communication infrastructures, the Earth climate
system, biologic organisms, or socio-economic organizations as cities.
Mathematical models are often used as an effective tool to study and tackle
their inner complexity, providing precise mechanisms to represent real problem
situations and helping to make decisions faster and more accurately.
Furthermore, when analytical solutions are not enough to study such systems,
simulation models can describe systems’ behavior. These computerized models
extend these analytical properties through the use of simulation methodologies
or formalisms (e.g., Cellular Automata, Machine Learning, fuzzy models) and
algorithms reproducing their life-cycle based on these descriptions.

The development of such simulation models starts with analysis over real-life
phenomena to extract different types of knowledge (i.e., mathematical
properties, behavioral rules) and create a conceptual model. Simulation models
derive from this specification, being implemented in specific programming
languages following the guidelines described by the selected M&S framework.
Through its execution with a simulation engine and varying the input
parameters and configurations, it is possible to obtain a wealth of
information that helps modelers understand reality and create more accurate
specifications. Over time, models increase their complexity in many ways. They
accumulate knowledge of several disciplines and increase the level of detail
of certain phenomena representations as the development iterations are
completed. The same model can combine low-level and high-level descriptions
and analyze a problem with different levels of detail. Moreover, the
combination of the size and diversity of these models often leads to large-
scale behaviors known as emergent behaviors. Therefore, as the systems grow,
the properties of the entire system become very different from those of its
components. Although several authors have reviewed the theoretical foundation
for modeling emergent behavior in the literature and several tools have been
created to facilitate the specification of these phenomena, reproducing
emergent behaviors in artificial environments is still a challenging task
(Mittal ., 2018). As the models produced in the modeling workflow represent
simplified versions of the existing systems, the inner omission of some low-
level details implies an information loss that may cancel these emerging
behaviors. To avoid this, the model has to be supported by solid hypotheses or
theoretical constructs and be extensively simulated to check its correctness.
The use of M&S formalisms contributes to this objective, offering clear,
reusable, and unambiguous specifications that deal with the complexity and
particularities of these systems. Some examples, matured in academia for
decades, are Petri Nets, Markov Chains, Cellular Automata, or Discrete Event
System Specification (DEVS).

###  2.2 IoT simulation frameworks

Some simulators allowing the specification of IoT scenarios are:

  * •

CloudSim (Calheiros ., 2011) is an open-source simulation framework developed
at the Cloud Computing and Distributed Systems (CLOUDS) laboratory of the
University of Melbourne. It allows modeling, simulation, and experimentation
of Cloud computing infrastructures and application services. It also includes
an end-to-end Cloud network architecture that utilizes BRITE topology (Medina
., 2001) for modeling link bandwidth and associated latencies. This framework
has been used as a basis to develop a multitude of specialized frameworks.

  * •

EdgeCloudSim (Sonmez ., 2018) is an edge-oriented simulator developed at the
Department of Computer Engineering at the Bogazici University. It extends the
functionality of CloudSim so that it can be used for Edge Computing scenarios.
Through a modular architecture, it provides support for a variety of
functionality such as network modeling specific to WLAN and WAN, device
mobility model, realistic and tunable load generator. It also supports the
definition of scenarios with several Edge server layers, properly coordinated
with different cloud resources. To facilitate these definitions, it also
includes orchestration modules to model the organization of the different
types of resources.

  * •

EmuFog (Mayer ., 2017) is an an extensible emulation framework for the
definition of Fog Computing scenarios. It allows the definition of fog
infrastructures and emulates real applications and workloads by embedding
Docker images in the scenario nodes. All components of EmuFog are extensible
and replaceable by custom-built components designed for specific scenarios.
Also, it allows loading the designs performed in network topology generators
as BRITE (Medina ., 2001), facilitating the import of real-world topology
datasets.

  * •

FogNetSim++ (Qayyum ., 2018) is a toolkit for the modeling and simulation of
distributed fog environments. It is built on the top of OMNeT++, a discrete-
event simulator oriented to the development of network simulators. It allows
us to incorporate customized mobility models and fog node scheduling
algorithms, and manage handover mechanisms. It bases the construction of IoT
scenarios on three main types of modules: (i) end devices, (ii) fog nodes, and
(iii) brokers. In these scenarios, the end devices contain the actual sensors
and generate the processing requests to upper layers. The broker receives
these requests and sends them to the suitable fog nodes. The broker nodes are
also connected to a backbone network which connects them to cloud data
centers. FogNetSim++ allows researchers to incorporate their request
scheduling and handover algorithms, simplifying the study of the energetic
impact of different delivery strategies.

  * •

FogTorch (Brogi Forti, 2017) is a Java tool for the definition of QoS-aware
IoT applications to Fog infrastructures. It divides the definition of the
scenario into three levels: (i) IoT devices, (ii) one or more layers of Fog
computing nodes, and (iii) one or more cloud data centers. The Cloud concept
is simplified in the FogTorch simulation model, understood as a virtually
unlimited amount of hardware capabilities. This limitation constrains the
scenario definition, but eliminates the need to describe particular cloud
topologies and infrastructures and simplifies the definition of any SaaS,
PaaS, or IaaS service. FogTorch allows specifying different Quality of Service
(QoS) profiles based on pairs of latency and bandwidth values, associating
them with specific network links.

  * •

GloudSim (Di Cappello, 2015) is a distributed cloud simulator that aims to
reproduce the Google cloud environment, allowing to define its cluster
infrastructures and simulate different types of events. It allows defining
dynamic resource consumption and priority levels for the emulated jobs and
reproducing kill/evict events. It is compatible with the Google traces format
and includes several interfaces and scripts to extract the information from
these CSV trace files. Among other parameters, it allows us to obtain the CPU
and memory consumption, the number of simultaneous active jobs, and the
workload processing ratio. GloudSim has been published under the GNU GPL v3
license.

  * •

iCanCloud (Núñez ., 2012) is a simulation platform aimed to model and simulate
cloud computing systems. The main objective of iCanCloud is to predict the
trade-offs between cost and performance of a given set of applications
executed in specific hardware, and then provide users helpful information
about such costs. It allows modeling the cloud architecture, includes a
hypervisor module for managing and comparing different brokering policies, and
enables the extraction of detailed energy consumption information for each
hardware component of the whole infrastructure. It is also possible to specify
custom policies to analyze the impact of energy consumption on the overall
system performance, facilitating the study of trade-offs between performance
and energy consumption.

  * •

iFogSim (Gupta ., 2017) is a toolkit for modeling and simulation of resource
management techniques in the Internet of Things, derived from the CloudSim
framework. It allows the study of different resource management policies
applicable to fog environments concerning their impact on latency
(timeliness), energy consumption, network congestion, and operational costs.
It simulates edge devices, cloud data centers, and network links to measure
performance metrics. One of its main contributions is the Sense-Process-
Actuate, which allows defining scenarios where sensors publish data
periodically or based on events, and devices in the fog layer subscribe to
these data streams.

  * •

IoTSim (Zeng ., 2017) is a CloudSim-based IoT simulator that allows to specify
and execute IoT big data scenarios. It organizes this specification in six
layers: (i) the Core Simulation Engine Layer provides core functionalities as
the creation of cloud elements, communication among components, and management
of the simulation time, (ii) the CloudSim Simulation Layer provides support
for modeling and simulation of Cloud-based simulation environments, (iii) the
Storage Layer includes different storage types as Amazon S3, Azure Blob
Storage, and HDFS, (iv) the Big Data Processing Layer the processing of the
data generated by the IoT devices through Map-Reduce or a streaming computing
model, and (v) the User Code Layer includes several utilities to facilitate
the definition and validation of IoT scenarios.

  * •

Mercury (Cárdenas ., 2020) is a Modeling, Simulation, and Optimization
framework to analyze the dimensioning and the dynamic operation of real-time
fog computing scenarios. It has been developed by the Integrated Systems
Laboratory at the Technical University of Madrid, upon the Python xDEVS API.
It allows to specify 2D Mobility scenarios and includes a 5G-based model. It
organizes the scenario definition in six layers: (i) IoT devices layer, (ii)
edge federation layer, (iii) access points (AP) layer, (iv) radio interface
layer, (v) core network layer, and (vi) Crosshaul layer. It also includes
utilities to easily selecting the APs and Micro Data Centers (MDCs) optimal
location and generating different output plots to study the results of the
simulations.

  * •

SFIDE (Penas ., 2017) is a simulation framework developed by the Embedded
Systems Laboratory at the École Polytechnique Fédérale de Lausanne (EPFL). It
allows the customization of the data centers architecture, including both the
servers arrangement in the room, the cooling equipment, and the workloads’
characterization to be executed. The real benefit of SFIDE resides in its
capability to implement, test, and assess arbitrary workload allocation
strategies and cooling control policies.

  * •

YAFS (Yet Another Fog Simulator) (Lera ., 2019) is a fog computing simulator
developed at the Department of Mathematics and Computer Science of the
University of the Balearic Islands (Spain). It is implemented through Simpy, a
simulator for the generation of discrete-event scenarios. It focuses on the
analysis of applications design and deployment through the use of customized
and dynamic strategies. YAFS allows representing the relationships among
applications, network connections, and infrastructure elements, enabling the
integration of application modules, workload location strategies, and path
routing and scheduling. The resulting computational and transmission results
can be exported as CSV files.

Table 1 summarizes these IoT simulators, showing some of their features and
capabilities. After the simulator name column, it shows the programming
language in which they are implemented, and its support for defining Edge,
Fog, and Cloud computing infrastructures. The next columns show if they
include a graphical GUI to graphically design the scenarios, if they support
Map-Reduce computing strategies, and if they allow defining the network
components and links. Finally, the last two columns indicate if the simulators
allow extracting raw data or visualizations related to the power consumption
and network communication latency, respectively. Among these simulators, it is
worth noting that only a few base their development on simulation formalisms.
Among these, we found Mercury and SFIDE, which are built upon the xDEVS
framework, and YAFS, which uses SimPy for its construction. A formal
simulation basis offers many advantages in developing simulators and systems,
providing unambiguous specifications that help manage the complexity of large
systems and the interaction among all their heterogeneous components.
Moreover, they often offer mechanisms to separate the model definition from
the simulation itself. In this way, they facilitate the scalability of
simulators’ maintainability while favoring more robust and reliable
developments.

In this paper, we use the Mercury framework to analyze the power consumption
of data centers receiving processing requests of IoT devices in a Wireless
Sensor Body Network (WSBN). We selected this framework since it includes the
IoT layers we need to incorporate and avoids complexities related to the cloud
layer and network infrastructures, not needed in this research. Additionally,
Mercury is the only one that provides support for 2D mobility, facilitating
our development. With this framework, we model a Micro Data Centers (MDCs)
architecture and components, as well as the services related to the data
processing task requested by the agents. Also, it provides mechanisms to
estimate the optimal locations of the data centers based on the patients’
movement over the scenario, and includes mechanisms to export and visualize
the simulations’ results.

Table 1: Comparison of Fog/Cloud simulators for defining IoT applications and scenarios. Simulator |  | Programming
---
Language
Edge | Fog | Cloud | GUI | Map-Reduce |  | Network
---
configuration
| Power
---
consumption
analysis
| Communication
---
latency
analysis
CloudSim | Java | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✓
CloudExp | Java | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
EdgeCloudSim | Java | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓
EmuFog | Python | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓
FogNetSim++ | C++ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓
FogTorch | Java | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓
GloudSim | Java | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗
iCanCloud | C++ | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗
iFogSim | Java | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓
IOTSim | Java | ✗ | ✗ | ✓ | ✗ | ✓ | ✓ | ✗ | ✓
Mercury | Python | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓
SFIDE | Java | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓
YAFS | Python | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓

###  2.3 Data-driven IoT scenarios: related work

When considering the usability and performance of an IoT scenario, several
parameters may be considered. From an end-user point of view, some aspects
like the perceived latency, the battery life of the nodes, and the overall
cost, have a critical impact on the acceptance of IoT products. The
optimization of these key aspects can be tackled when developing the IoT
scenario by optimizing the power consumption, the processing workflows, and
the required bandwidth. The application placement, communication protocols, or
job allocation policies are some of the decisions that can alter the usability
and performance of the final product. For instance, assigning more processing
responsibilities to the end-nodes can highly reduce the latency perceived by
the users, but significantly impact the battery life and cost of the product.
On the contrary, delegating processing to cloud services can lead to
substantial cost savings and higher reliability and performance, but its use
implies the acceptance of higher latency and more complex system architecture.
Usually, fog computing offers a good trade-off between these two approaches,
and it is especially suitable when developing low latency or real-time
applications. A good example is healthcare IoT systems, which are usually
latency-sensitive, show low response time, and produce a large amount of data
(Mutlag ., 2019). As a result, intensive use of fog computing has been made in
this area. For instance, Tuli et al. (Tuli ., 2020) presented a fog-based
smart healthcare system for automatic diagnosis of heart diseases using deep
learning techniques. The data coming from different IoT devices is processed
through a fog service for deducing the health status of patients and identify
heart disease severity. Sahoo et al. (Sahoo ., 2016) designed a stochastic
prediction model to foresee the future health condition of the groups of
correlated patients based on their current health status. Aazam et al. (Aazam
Huh, 2015) presented a resource management model for IoTs based on MDCs,
covering resource prediction, customer type based resource estimation and
reservation, and pricing estimations. Jararweh et al. (Jararweh ., 2017)
developed a software-defined based framework to allow mobile cloud computing
(MCC) services to integrate different Software-Defined Systems (SDSys) in a
Mobile Edge Computing context. This software-based specification of systems
abstracts the complexity of large infrastructure deployments and is especially
useful for content delivery networks, crowdsourcing, traffic management, or
E-health, among others.

In IoT healthcare systems, one of the most important involved methods is
monitoring (Mutlag ., 2019). A typical paradigm in this kind of scenario is
the Modeling and Simulation (M&S) of crowds. The representation of the
behavior of a population or a group of individuals allows the development of
realistic scenarios as a basis for evaluating new technologies or
methodologies. Multiple techniques are used (S. Zhou ., 2010) to represent
this behavior, such as agent-based models (B. Zhou ., 2012), flow-based models
(Tripathi ., 2019), or particle system models (Liu ., 2018). Currently,
several platforms exist to simplify this crowd representation. Simulation of
Urban MObility (SUMO) (Behrisch ., 2011) (shown in Figure 1b) is an open-
source traffic simulation package including net import and demand modeling
components. It provides direct compatibility with OpenStreetMap shapefiles and
allows modeling of vehicles, public transport, and pedestrians. Pedestrian
Dynamics (Schijve, ) is a simulation environment designed to model pedestrian
infrastructures or environments. It allows importing buildings geometry from
multiple formats and generating 2D, 3D, and VR virtual environments. Figure 1a
shows an scenario example developed with this software. Simulation of such
large scenarios involving crowds usually requires intensive computational
resources to be processed. These resources, in some cases, far exceed the
capabilities of single workstations, being more suitable for its execution in
clusters or data centers.

(a) Pedestrian Dynamics.

(b) Simulation of Urban MObility (SUMO).

Figure 1: Example scenarios designed with urban simulation packages.

In order to model the patients’ displacement in our IoT scenario, we have
selected Pedestrian Dynamics, as it allows high configuration of the
pedestrians’ behavior and provides mechanisms to easily import the layout of
the infrastructures. The dataset of moving pedestrians generated by this
simulation platform is used as an input to Mercury’s scenario to determine to
which data center the patients’ monitoring devices have to send their
processing requests.

##  3 Experimental setup

In this paper, we consider an ambulatory monitoring system scenario where a
population of migraine patients wearing health monitoring devices periodically
sends some hemodynamic variables to Micro Data Centers (MDCs). These MDCs use
per-patient predictive models to estimate the probability of new pain phases,
generating an alert in specific patients’ monitoring devices when a new pain
phase is approaching. In this context, we have selected an actual urban area
and modeled its infrastructures and population movement, seeking to locate the
best data center locations to optimize the energy consumption of the system.

Figure 2: Scenario architecture. A network of Edge Data Centers communicated
with a distributed database provide connectivity to end devices through
several Access Points.

The architecture of the ambulatory monitoring scenario is represented in
Figure 2. The monitoring devices carried by migraine patients periodically
obtain several hemodynamic variables and send them to their smartphones.
Through an app and with a predefined frequency, these data are sent
periodically to the MDCs to get the predictions. This connection is performed
through a network of Access Points that provide coverage to an entire
metropolitan area. In this way, each phone sends the data through the nearest
AP, and each AP is connected with the nearest MDC. The MDCs evaluate the data
packet using per-patient custom models and return the probability of a new
migraine pain phase. When this probability exceeds a threshold, the patient is
alerted through the monitoring device. Moreover, there exists a shared
distributed database when the different customized models are stored. Hence,
when an MDC receives a prediction request corresponding to a new patient it
loads the suitable model and performs the inference over it.

Each MDC consists of a set of racks, each one containing several servers.
Sometimes, as part of the efforts to reduce the high energy consumption of
data centers, idle servers are shut down until end-users request extra
resources. In our simulation, for simplicity purposes, we do not use these
types of strategies. This facilitates the interpretation of the results,
allowing us to focus on the energy dynamically consumed by the processing
units. For a realistic characterization of the predictive models, we set some
scenario parameters based on the research conducted by Pagan et al. In their
study, several types of custom migraine models have been developed, using
algorithms like Subspace State Space System Identification (N4SID) ((Pagán .,
2015)) and Grammatical Evolution ((Pagán ., 2016)) for training the models.
For the generation of these per-patient models, they perform ambulatory
monitoring of the patients during a period from two weeks up to a month. The
five variables implied in this process are: (i) electrodermal activity, (ii)
heart rate, (iii) oxygen saturation, (iv) surface skin temperature, and (v)
subjective pain level for each migraine event. This last variable is
registered through an app, where the patient tracks the progress of migraine.
The rest of them are registered with a portable medical monitoring device.
After the model is trained in this offline phase, it is ready to be uploaded
to the servers and start generating runtime predictions of the migraines.

Figure 3: Information workflow used in the process of modeling and simulating
the pedestrians scenario.

Figure 3 summarizes the information workflow followed in the modeling and
simulation of these scenarios. We start extracting an urban area for our
scenario from OpenStreetMaps (OSM) (M. Wang ., 2013). This collaborative
project gives access to detailed information of map data across much of the
world, containing information such as the layout of roads and paths, buildings
topology, place names, and points of interest. For these scenarios, we have
selected a metropolitan area of $75km^{2}$ belonging to Madrid, Spain’s
capital city. Although OSM provides a native mechanism to extract its data, it
is limited to 50000 map nodes. As this scenario corresponds to a large
metropolitan area, we have used the web platform HOT Export Tool to obtain a
dataset containing the layout of all the buildings in the selected area as a
KML map file. This map is then converted for compatibility reasons to CityGML
(Kolbe ., 2005), an open standardized data model and exchange format to store
digital 3D models of cities and landscapes, using Autodesk Map 3D. This
CityGML definition is then loaded into Pedestrian Dynamics, a simulation
environment designed to model pedestrian infrastructures or environments.

(a) Map view.

(b) Simulation scenario.

Figure 4: Geographical area considered for the simulation. It corresponds to
Madrid (Spain), has an area of around $75km^{2}$, and includes 9 hospitals.
Figure 5: Intermediate presence grid generated by Mercury with a time window
of 60s and a grid resolution of 40. This grid is used to find the best
locations of APs and MDCs through KMeans clustering.

The information of these buildings is used in the simulations as obstacles
that the pedestrians have to avoid to reach their goals. The main streets and
parks of the remaining space are manually marked as activity areas. These
areas are used by Pedestrian Dynamics to determine the locations to which the
pedestrians go during the simulation. Moreover, 139 stops of the Madrid metro
network extracted from the Madrid Open Data portal were included in the map as
entry/exit points. With these infrastructures already loaded into Pedestrian
Dynamics, we configured the pedestrians’ simulation as follows. Every 180
seconds, 200 pedestrians agents are instantiated in the entry/exit points,
simulating the arrival of new trains in the metro stations according to
average values recorded in the city of Madrid during peak hours (Álvarez-
Sánchez, 2018). Each of these pedestrians is configured to walk until a random
location included in the main streets or parks defined before, wait for a few
seconds, and exit the scenario for the nearest metro stop. This simulation is
performed until 10 hours of virtual time are completed (from 7 A.M. to 12 A.M.
and from 4 P.M. to 9 P.M., which are the peak intervals). As a result, we
obtain a dataset containing the XY coordinates of each pedestrian agent for
each simulation time step, that can be used as a reference to model the
behavior of the population. We have selected these parameters because they
represent the most dynamic scenario: (i) the frequency of trains during the
intervals selected is maximum, (ii) we have reduced the waiting time per
pedestrian to a few seconds, maximizing the number of sessions opened at the
MDCs, and (iii) these peak hours coincide with the maximum demand of the
health system.

The 10-hour dataset generated using Pedestrian Dynamics is then loaded into
the Mercury DEVS framework (Cárdenas ., 2020). Apart from the modules involved
in the IoT scenarios simulation itself, this framework also includes some
additional tools to generate datasets with optimized locations of APs and
MDCs. First, this allocation method partitions the scenario map using a grid,
registering the maximum presence of pedestrians in each cell considering a
specific time window and grid resolution. Figure 5 depicts the intermediate
grid. Mercury uses this information to calculate the optimized locations of
APs and MDCs through a KMeans clustering process. We use this method for
placing the APs location, which is common to all the scenarios. For this
study, we define four alternative scenarios:

  * •

Clustering scenario (C3): both the APs and the MDCs are allocated based on the
KMeans clustering. The resulting locations of these elements are depicted in
Figure 6a. This Figure depicts the paths chosen by the pedestrians with points
of different colors, corresponding with the coverage area of each MDCs. The
different APs representing the mobile communication network are drawn as
squares and the MDCs as circles. As a reference, public hospitals of this
metropolitan area are marked with crosses. In this scenario we consider a
fixed amount of 3 MDCs to illustrate the validity of the analysis.

  * •

Central hospital (H1): only one MDC is placed in the scenario, corresponding
with the nearest hospital to its center. As a result, all the tasks generated
by the monitorization devices are sent to this single data center in this
scenario.

  * •

Nearest hospitals (H3): the MDCs of the clustering scenario (C3) are moved to
the nearest hospitals. Figure 6b represent the resulting scenario.

  * •

All the hospitals (H9): an MDC is placed in each of the nine hospitals present
in the selected metropolitan area.

As we aim to compare how the power consumption is distributed among the MDCs
of these scenarios, we preserve the same configuration for all the MDCs. Each
MDC is composed of 10 processing units (PUs), allocated in a single rack. Each
of these PUs represents an Intel Core i9900K processor, operating at 3.6 GHz
and composed of 8 cores (16 threads). For calculating the power consumption of
the PUs, we selected the IdleActive power model included in the Mercury
framework. This simple model considers two different power consumption values.
The idle power consumption applies when no task is being executed in a
specific PU. The active power consumption specifies the power consumption
registered when the PU is executing one or more tasks. In this case, according
to the selected processor, the idle power is set to 47W and the active power
to 95W.

(a) Scenario 1: APs and MDCs are allocated using KMeans clustering.

(b) Scenario 2: MDCs location are changed so that the correspond to the
hospitals closest to the MDCs generated for Scenario 1. The same APs remain.

Figure 6: Scenarios considered in the simulation.

Moreover, it is worth noting that the selected dispatching strategy searches
the first PU sequentially with enough free resources to allocate new tasks.
Therefore, a PU does not receive a task until the previous PU is unable to
allocate it. Although Mercury allows configuring the shutdown of inactive
processing units, all the PUs are powered on during all the simulations to
reduce variability and facilitate the interpretation of the results.

For each pedestrian in the 10-hour dataset, an agent is created in the
simulation. For this, Mercury sorts all the agents’ entry time and creates and
destroys them dynamically so that only the active agents are loaded in memory.
We consider that all these pedestrian agents carry a monitoring device, and
configure two types of services. The inference service requests periodically
to the nearest MDC an updated estimation of the probability of a new onset of
pain, based on the previously trained predictive models. The training service
requests a new training of the patient model and aims to generate more
efficient models with the updated patient data iteratively.

Table 2: Statistics of the monitoring devices services. Service |  | Generation
---
period
| PU Util.
---
(%)
| Operation
---
time (s)
| Packet payload
---
size (B)
| Total packet
---
size (B)
Inference | 60s | 6.25 | 1.17 | 65 | 119
Training | unif(1s, 1d) | 6.25 | 18 | 20 | 74

Table 2 shows more detailed information of these services. The inference
service requests each minute an updated probability of pain onset, while the
subsequent training request is calculated with a uniform distribution from 1
second to 1 day. Both services use 6.25% of a PU, since they fully use one of
its 16 threads. The operation times for the inference and training services,
1.17s and 18s respectively, were extracted from actual training times of
migraines models using similar processors. The packet payload size is
determined according to the BSON encoding of the dictionaries exemplified in
Figure 7. Both contain the patient identifier and a boolean indicating to the
data center if new training is needed. Additionally, the inference packet
contains the average values of the last minute for the implied hemodynamic
variables. The total packet size of Table 2 corresponds to the sum of the
payload size and the header of the TCP packet (fixed to 54B).

Table 3: Mercury configuration parameters. Parameter | Value
---|---
Mercury mode | Lite
Hot standby | Off
PUs per MDC | 10
Threads per PU | 16
Threads per service | 1
| Simultaneous requests
---
(per service and UE)
1
Power model | Idle/Active
Idle power | 47W
Active power | 95W

⬇

{

"id":348345,

"train":false,

"spo2":98,

"eda":19.7,

"temp":34.23,

"hr":67

}

(a) Inference service sample payload.

⬇ { "id":348345, "train":true }

(b) Training service sample payload.

Figure 7: Sample packets generated by the simulation services.

Table 3 contains summary information regarding the specific configuration
values used for this Mercury scenario. We used Mercury’s Lite mode, which
speeds up the execution time by avoiding fine-grained calculations regarding
intermediate network communications. To improve the interpretability of the
results, we do not active the hot standby configurations provided by Mercury,
which allows dynamically change PUs status based on MDCs utilization ratios.
Training and inference tasks are executed in a single thread, so 16 tasks can
be run in parallel by each processing unit. Further information regarding the
configuration of this scenario can be found in the project’s official GitHub
repository (Henares, ), which includes the code and datasets used to deploy
it.

Through this methodology, we can analyze the deployment of MDCs, studying how
their location can alter the overall energy consumption of the scenario. In
the next section, we present results regarding the optimization of energy
consumption and the distribution of tasks for each of the mentioned MDCs
organizations.

##  4 Results

The four scenarios defined in the previous section were executed in the ETNA
cluster of the Arcuitecture and Technology of Computing Systems (ArTeCS)
research group of the Complutense University of Madrid. Each simulation run
provides results like power consumption, utilization ratios, sessions status,
etc. Figure 8 shows how the average utilization of each MDCs evolves as
simulation time progresses, and includes shaded areas whose edges indicate the
minimum and maximum utilization over time. In this Figure, the solid lines
represent the evolution of the utilization factor, and the dashed line
represents the evolution in the number of simultaneously monitored
pedestrians. We can see how the H1 hospital gets quickly overloaded, being
unable to serve all incoming requests. This situation is depicted in Figure 9,
showing its number of rejected sessions over time. H3 and C3 scenarios produce
very similar results, being its evolution lines overlapped in the plot.
Finally, because the requests are distributed among a greater number of MDCs,
the H9 scenario obtains the least average utilization factor.

For a better understanding of how the requests are distributed over the MDCs,
we represent (see Figure 10) the percentage of utilization of the individual
MDCs with respect to the total utilization of each scenario. In the case of H3
and C3 scenarios, we can see an even distribution of the requests, sending
about a third of them to each MDC. As expected, the percentage of individual
usage differs considerably in the H9 scenario since the distance among data
centers is not optimized. As a result, a third of data centers have less than
3% usage while the two most used data centers add up to around 44%.

Figure 8: MDCs average utilization for the different scenarios, compared with
the number of simultaneous pedestrian agents. H3 and C3 produce similar
results (overlapped). Figure 9: Sessions rejected by the MDC of the H1
scenario due to resource overload .

(a) C3 scenario.

(b) H3 scenario.

(c) H9 scenario.

Figure 10: Distribution of computational tasks over the Micro Data Centers.

Figure 11 depicts the mean power consumption for the different MDCs after
15000s of simulation time, when the number of simultaneous pedestrians is
stabilized. The blue bars indicate the power consumption derived from the
infrastructure itself, while the orange ones represent the consumption due to
the training and inference services’ execution. Consequently, we can see how
H3 and C3 scenarios present an equivalent power consumption with the
utilization ratios. Also, we can see how around a third of the power
consumption corresponds to the execution of the services. While looking at the
H1 scenario, it is worth noting that even though the energy consumption only
represents around 47% compared to the H3 and C3 scenarios, its MDC gets
quickly overloaded before reaching a third of the total simulation time.
Hence, this scenario does not provide a convenient service to the training and
inference tasks and does not accomplish the scenario’s requirements. The H9
scenario is not a good solution either, since it presents a total consumption
almost 2.5 times higher than the scenarios with 3 MDCs. Moreover, as its
processing units show a less exhaustive use, the resulting non-idle power
consumption is 45% more than the H3 and C3 scenarios.

Figure 11: Average power consumption of the different scenarios after the
number of simultaneous pedestrians is stabilized (after 15000s of simulation
time).

##  5 Conclusions

We are in the midst of the data era, where the information generated worldwide
and the number of connected devices keeps increasing exponentially. Such
devices are helping to improve the efficiency of a large variety of procedures
and activities. A highly relevant example is the field of healthcare, where
non-intrusive monitoring devices are promoting a move from the traditional
post-facto diagnose-and-treat paradigm to prognosis and predictive approaches.
However, monitoring large populations with these networked devices demands
vast storage and processing capabilities, together with efficient strategies
for the deployment and management of the underlying IoT infrastructures. In
this paper, we have presented a M&S-driven methodology that can be used to
analyze and deploy healthcare IoT infrastructures involving networks of
monitored patients. We have simulated patients’ movement using a well-known
crowd simulator, over an urban scenario based on actual building layouts and
metro stops. The architecture and functionality of data centers have been
modeled with the use of the Mercury framework, defining specific APs over the
city to serve as intermediate connection infrastructures for the communication
between monitoring devices and data centers, and defining specific data center
architectures and processing patterns. Also, we have compared different
scenarios based on clustered and hospital-based Micro Data Centers locations,
analyzing how the placement of the processing resources impacts the overall
energy consumption of the system. As a result, we have obtained several
comparisons of utilization ratios and power consumption, providing insights
that can help in decision-making when optimizing the location of IoT
infrastructures. Moreover, this methodology can be easily adapted to suit
additional use cases, analyze other parameters as communication delays or
suitability of the selected processing units, and perform an in-depth analysis
of the optimal number of MDCs based on the chosen data center architectures.

## References

  * Aazam Huh (2015) aazam2015fogAazam, M. Huh, EN.  2015\.  Fog computing micro datacenter based dynamic resource estimation and pricing model for IoT Fog computing micro datacenter based dynamic resource estimation and pricing model for iot.  2015 IEEE 29th International Conference on Advanced Information Networking and Applications 2015 ieee 29th international conference on advanced information networking and applications ( 687–694).
  * Afuang (2020) idc2020iotAfuang, A.  2020\.  IoT Growth Demands Rethink of Long-Term Storage Strategies IoT growth demands rethink of long-term storage strategies .  International Data Corporation.
  * Ali . (2020) ali2020smartAli, F., El-Sappagh, S., Islam, SR., Kwak, D., Ali, A., Imran, M. Kwak, KS.  2020\.  A smart healthcare monitoring system for heart disease prediction based on ensemble deep learning and feature fusion A smart healthcare monitoring system for heart disease prediction based on ensemble deep learning and feature fusion.  Information Fusion63208–222.
  * Álvarez-Sánchez (2018) AlvarezSanchez2018Álvarez-Sánchez, C.  2018\.  Deployment and management of platforms based on fog computing within an interconnected city Deployment and management of platforms based on fog computing within an interconnected city .  Universidad Complutense de Madrid.
  * Barba . (2012) barba2012smartBarba, CT., Mateos, MA., Soto, PR., Mezher, AM. Igartua, MA.  2012\.  Smart city for VANETs using warning messages, traffic statistics and intelligent traffic lights Smart city for vanets using warning messages, traffic statistics and intelligent traffic lights.  2012 IEEE intelligent vehicles symposium 2012 ieee intelligent vehicles symposium ( 902–907).
  * Behrisch . (2011) behrisch2011sumoBehrisch, M., Bieker, L., Erdmann, J. Krajzewicz, D.  2011\.  SUMO–simulation of urban mobility: an overview Sumo–simulation of urban mobility: an overview.  Proceedings of SIMUL 2011, The Third International Conference on Advances in System Simulation. Proceedings of simul 2011, the third international conference on advances in system simulation.
  * Brogi Forti (2017) brogi2017fogtorchBrogi, A. Forti, S.  2017\.  QoS-aware deployment of IoT applications through the fog Qos-aware deployment of iot applications through the fog.  IEEE Internet of Things Journal451185–1192.
  * Calheiros . (2011) calheiros2011cloudsimCalheiros, RN., Ranjan, R., Beloglazov, A., De Rose, CA. Buyya, R.  2011\.  CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms.  Software: Practice and experience41123–50.
  * Cárdenas . (2020) cardenas2020mercuryCárdenas, R., Arroba, P., Blanco, R., Malagón, P., Risco-Martín, JL. Moya, JM.  2020\.  Mercury: A modeling, simulation, and optimization framework for data stream-oriented IoT applications Mercury: A modeling, simulation, and optimization framework for data stream-oriented iot applications.  Simulation Modelling Practice and Theory101102037.
  * Castillo O’Sullivan Thierer (2015) castillo2015projectingCastillo O’Sullivan, A. Thierer, AD.  2015\.  Projecting the growth and economic impact of the internet of things Projecting the growth and economic impact of the internet of things.  Available at SSRN 2618794.
  * Dey . (2000) dey2000contextDey, AK., Abowd, GD. Salber, D.  2000\.  A context-based infrastructure for smart environments A context-based infrastructure for smart environments.  Managing Interactions in Smart Environments Managing interactions in smart environments ( 114–128).  Springer.
  * Di Cappello (2015) di2015gloudsimDi, S. Cappello, F.  2015\.  GloudSim: Google trace based cloud simulator with virtual machines Gloudsim: Google trace based cloud simulator with virtual machines.  Software: Practice and Experience45111571–1590.
  * Erdemir . (2020) erdemir2020credibleErdemir, A., Mulugeta, L., Ku, JP., Drach, A., Horner, M., Morrison, TM.Myers, JG.  2020\.  Credible practice of modeling and simulation in healthcare: ten rules from a multidisciplinary perspective Credible practice of modeling and simulation in healthcare: ten rules from a multidisciplinary perspective.  Journal of translational medicine1811–18.
  * Fox . (2009) fox2009aboveFox, A., Griffith, R., Joseph, A., Katz, R., Konwinski, A., Lee, G.others  2009\.  Above the clouds: A berkeley view of cloud computing Above the clouds: A berkeley view of cloud computing.  Dept. Electrical Eng. and Comput. Sciences, University of California, Berkeley, Rep. UCB/EECS28132009.
  * Gaba (2007) gaba2007futureGaba, DM.  2007\.  The future vision of simulation in healthcare The future vision of simulation in healthcare.  Simulation in Healthcare22126–135.
  * Giffin . (2003) giffin2003premonitoryGiffin, N., Ruggiero, L., Lipton, RB., Silberstein, S., Tvedskov, J., Olesen, J.Macrae, A.  2003\.  Premonitory symptoms in migraine: an electronic diary study Premonitory symptoms in migraine: an electronic diary study.  Neurology606935–940.
  * Gupta . (2017) gupta2017ifogsimGupta, H., Vahid Dastjerdi, A., Ghosh, SK. Buyya, R.  2017\.  iFogSim: A toolkit for modeling and simulation of resource management techniques in the Internet of Things, Edge and Fog computing environments ifogsim: A toolkit for modeling and simulation of resource management techniques in the internet of things, edge and fog computing environments.  Software: Practice and Experience4791275–1296.
  * Henares () github2021mdcsHenares, K.  .  MDCs optimization scenario repository. MDCs optimization scenario repository.  https://github.com/khvilaboa/mdcs_optimization.  [Online; accessed 07-October-2021]
  * Jararweh . (2017) jararweh2017softwareJararweh, Y., Alsmirat, M., Al-Ayyoub, M., Benkhelifa, E., Darabseh, A., Gupta, B. Doulat, A.  2017\.  Software-defined system support for enabling ubiquitous mobile edge computing Software-defined system support for enabling ubiquitous mobile edge computing.  The Computer Journal60101443–1457.
  * Kelman (2006) kelman2006postdromeKelman, L.  2006\.  The postdrome of the acute migraine attack The postdrome of the acute migraine attack.  Cephalalgia262214–220.
  * Kolbe . (2005) kolbe2005citygmlKolbe, TH., Gröger, G. Plümer, L.  2005\.  CityGML: Interoperable access to 3D city models Citygml: Interoperable access to 3d city models.  Geo-information for disaster management Geo-information for disaster management ( 883–899).  Springer.
  * Lera . (2019) lera2019yafsLera, I., Guerrero, C. Juiz, C.  2019\.  YAFS: A simulator for IoT scenarios in fog computing Yafs: A simulator for iot scenarios in fog computing.  IEEE Access791745–91758.
  * Linde . (2012) linde2012costLinde, M., Gustavsson, A., Stovner, LJ., Steiner, TJ., Barré, J., Katsarava, Z.others  2012\.  The cost of headache disorders in Europe: the Eurolight project The cost of headache disorders in europe: the eurolight project.  European journal of neurology195703–711.
  * Liu . (2018) Liu2018Liu, H., Xu, B., Lu, D. Zhang, G.  2018\.  A path planning approach for crowd evacuation in buildings based on improved artificial bee colony algorithm A path planning approach for crowd evacuation in buildings based on improved artificial bee colony algorithm.  Applied Soft Computing68360–376.
  * Masera . (2018) masera2018smartMasera, M., Bompard, EF., Profumo, F. Hadjsaid, N.  2018\.  Smart (electricity) grids for smart cities: Assessing roles and societal impacts Smart (electricity) grids for smart cities: Assessing roles and societal impacts.  Proceedings of the IEEE1064613–625.
  * Mayer . (2017) mayer2017emufogMayer, R., Graser, L., Gupta, H., Saurez, E. Ramachandran, U.  2017\.  Emufog: Extensible and scalable emulation of large-scale fog computing infrastructures Emufog: Extensible and scalable emulation of large-scale fog computing infrastructures.  2017 IEEE Fog World Congress (FWC) 2017 ieee fog world congress (fwc) ( 1–6).
  * Medina . (2001) medina2001briteMedina, A., Lakhina, A., Matta, I. Byers, J.  2001\.  BRITE: An approach to universal topology generation Brite: An approach to universal topology generation.  MASCOTS 2001, Proceedings Ninth International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems Mascots 2001, proceedings ninth international symposium on modeling, analysis and simulation of computer and telecommunication systems ( 346–353).
  * Mell Grance (2011) mell2011nistMell, P. Grance, T.  2011\.  The NIST definition of cloud computing The NIST definition of cloud computing .  National Institute of Standards and Technology.
  * Mittal . (2018) Mittal2018Mittal, S., Diallo, S. Tolk, A.  2018\.  Emergent behavior in complex systems engineering: a modeling and simulation approach Emergent behavior in complex systems engineering: a modeling and simulation approach.  John Wiley & Sons.
  * Mutlag . (2019) mutlag2019enablingMutlag, AA., Abd Ghani, MK., Arunkumar, Na., Mohammed, MA. Mohd, O.  2019\.  Enabling technologies for fog computing in healthcare IoT systems Enabling technologies for fog computing in healthcare iot systems.  Future Generation Computer Systems9062–78.
  * Núñez . (2012) nunez2012icancloudNúñez, A., Vázquez-Poletti, JL., Caminero, AC., Castañé, GG., Carretero, J. Llorente, IM.  2012\.  iCanCloud: A flexible and scalable cloud infrastructure simulator icancloud: A flexible and scalable cloud infrastructure simulator.  Journal of Grid Computing101185–209.
  * Olesen (2004) headache2004internationalOlesen, J.  2004\.  The international classification of headache disorders. The international classification of headache disorders.  https://ichd-3.org/.
  * Pagán . (2015) pagan2015robustPagán, J., De Orbe, MI., Gago, A., Sobrado, M., Risco-Martín, JL., Mora, JV.Ayala, JL.  2015\.  Robust and accurate modeling approaches for migraine per-patient prediction from ambulatory data Robust and accurate modeling approaches for migraine per-patient prediction from ambulatory data.  Sensors15715419–15442.
  * Pagán . (2016) pagan2016grammaticalPagán, J., Risco-Martín, JL., Moya, JM. Ayala, JL.  2016\.  Grammatical evolutionary techniques for prompt migraine prediction Grammatical evolutionary techniques for prompt migraine prediction.  Proceedings of the Genetic and Evolutionary Computation Conference 2016 Proceedings of the genetic and evolutionary computation conference 2016 ( 973–980).
  * Penas . (2017) penas2017sfidePenas, I., Zapater, M., Risco-Martín, JL. Ayala, JL.  2017\.  SFIDE: a simulation infrastructure for data centers Sfide: a simulation infrastructure for data centers.  Proceedings of the Summer Simulation Multi-Conference Proceedings of the summer simulation multi-conference ( 1–12).
  * Qayyum . (2018) qayyum2018fognetsim++Qayyum, T., Malik, AW., Khattak, MAK., Khalid, O. Khan, SU.  2018\.  FogNetSim++: A toolkit for modeling and simulation of distributed fog environment Fognetsim++: A toolkit for modeling and simulation of distributed fog environment.  IEEE Access663570–63583.
  * Reinsel . (2018) reinsel2018dataReinsel, D., Gantz, J. Rydning, J.  2018\.  Data age 2025: the digitization of the world from edge to core Data age 2025: the digitization of the world from edge to core.  Seagate, https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf.
  * Saha . (2017) saha2017healthSaha, HN., Auddy, S., Pal, S., Kumar, S., Pandey, S., Singh, R.Saha, S.  2017\.  Health monitoring using internet of things (IoT) Health monitoring using internet of things (iot).  2017 8th Annual Industrial Automation and Electromechanical Engineering Conference (IEMECON) 2017 8th annual industrial automation and electromechanical engineering conference (iemecon) ( 69–73).
  * Sahoo . (2016) sahoo2016analyzingSahoo, PK., Mohapatra, SK. Wu, SL.  2016\.  Analyzing healthcare big data with prediction for future health condition Analyzing healthcare big data with prediction for future health condition.  IEEE Access49786–9799.
  * Schijve () url2020pedestriansSchijve, L.  .  Pedestrian Dynamics. Pedestrian Dynamics.  https://www.incontrolsim.com/software/pedestrian-dynamics/.  [Online; accessed 25-November-2020]
  * Sonmez . (2018) sonmez2018edgecloudsimSonmez, C., Ozgovde, A. Ersoy, C.  2018\.  Edgecloudsim: An environment for performance evaluation of edge computing systems Edgecloudsim: An environment for performance evaluation of edge computing systems.  Transactions on Emerging Telecommunications Technologies2911e3493.
  * Stovner Andree (2010) stovner2010prevalenceStovner, LJ. Andree, C.  2010\.  Prevalence of headache in Europe: a review for the Eurolight project Prevalence of headache in europe: a review for the eurolight project.  The journal of headache and pain114289.
  * Tokody . (2017) tokody2017overviewTokody, D., Mezei, IJ. Schuster, G.  2017\.  An overview of autonomous intelligent vehicle systems An overview of autonomous intelligent vehicle systems.  Vehicle and Automotive Engineering Vehicle and automotive engineering ( 287–307).  Springer.
  * Tripathi . (2019) Tripathi2019Tripathi, G., Singh, K. Vishwakarma, DK.  2019\.  Convolutional neural networks for crowd behaviour analysis: a survey Convolutional neural networks for crowd behaviour analysis: a survey.  The Visual Computer355753–776.
  * Tuli . (2020) tuli2020healthfogTuli, S., Basumatary, N., Gill, SS., Kahani, M., Arya, RC., Wander, GS. Buyya, R.  2020\.  Healthfog: An ensemble deep learning based smart healthcare system for automatic diagnosis of heart diseases in integrated iot and fog computing environments Healthfog: An ensemble deep learning based smart healthcare system for automatic diagnosis of heart diseases in integrated iot and fog computing environments.  Future Generation Computer Systems104187–200.
  * Voorsluys . (2011) voorsluys2011introductionVoorsluys, W., Broberg, J. Buyya, R.  2011\.  Introduction to cloud computing Introduction to cloud computing.  Cloud computing: Principles and paradigms1–44.
  * F. Wang . (2019) wang2019demystifyingWang, F., Wang, F., Ma, X. Liu, J.  2019\.  Demystifying the crowd intelligence in last mile parcel delivery for smart cities Demystifying the crowd intelligence in last mile parcel delivery for smart cities.  IEEE Network33223–29.
  * M. Wang . (2013) url2020osmWang, M., Li, Q., Hu, Q. Zhou, M.  2013\.  Quality analysis of open street map data Quality analysis of open street map data.  International archives of the photogrammetry, remote sensing and spatial information sciences2.
  * Zeng . (2017) zeng2017iotsimZeng, X., Garg, SK., Strazdins, P., Jayaraman, PP., Georgakopoulos, D. Ranjan, R.  2017\.  IOTSim: A simulator for analysing IoT applications Iotsim: A simulator for analysing iot applications.  Journal of Systems Architecture7293–107.
  * B. Zhou . (2012) Zhou2012Zhou, B., Wang, X. Tang, X.  2012\.  Understanding collective crowd behaviors: Learning a mixture model of dynamic pedestrian-agents Understanding collective crowd behaviors: Learning a mixture model of dynamic pedestrian-agents.  2012 IEEE Conference on Computer Vision and Pattern Recognition 2012 ieee conference on computer vision and pattern recognition ( 2871–2878).
  * S. Zhou . (2010) zhou2010crowdZhou, S., Chen, D., Cai, W., Luo, L., Low, MYH., Tian, F.Hamilton, BD.  2010\.  Crowd modeling and simulation technologies Crowd modeling and simulation technologies.  ACM Transactions on Modeling and Computer Simulation (TOMACS)2041–35.

## Funding

This work was partially supported by the Spanish Ministry of Science and
Innovation under 2PIC4BioMed (PID2019-110866RB-I00) project grant, by the the
Madrid Regional Government under CABAHLA-CM (S2018/TCS-4423) project grant,
and by the Google Cloud Research Credits program with the award GCP19980904.


# Radiation damage on SiPMs for Space Applications

Anna Rita Altamura  Fabio Acerbi  Benedetto Di Ruzza  Enrico Verroi  Stefano
Merzi  Alberto Mazzi  Alberto Gola  Fondazione Bruno Kessler (FBK), Sensors &
Devices (SD), via Sommarive 18, I-38123, Trento, Italy  University of Udine,
via Palladio 8, 33100 Udine, Italy  TIFPA - Trento Institute for Fundamental
Physics and Applications, via Sommarive 14, I-38123, Trento, Italy  University
of Trento, via Sommarive 14, I-38123, Trento, Italy

###### Abstract

Silicon Photo-multipliers (SiPMs) are very sensitive photo-detectors that
experienced a big development in the last years in several applications, like
LIDAR, astrophysics, medical imaging and high energy physics (HEP)
experiments. In HEP experiments, in particular, they are often exposed to
significant radiation doses. The main purpose of this manuscript is the
characterization of several FBK SiPM technologies when exposed to 74 $MeV$
protons with a total fluence comparable to the one that they would experience
in space along circular Low Earth Orbits (LEO), Polar, during a five years
mission.

In this work, we estimated the expected proton fluences along the selected
orbit, by means of the SPENVIS software. Several fluence steps were chosen to
consider dense fluence intervals and have a more accurate sight on the whole
damage process. We estimated a maximum fluence achieved during the tests of
$6.4\times 10^{11}$ $n_{eq}/cm^{2}$. Based on such simulations, we irradiated
several SiPM technologies. We developed a custom experimental setup, which was
used to perform online reverse voltage-current, right after each irradiation
step, to minimize the effect of the annealing on the measurement.

The results are then displayed, in particular the currents, the noise and the
Photon Detection Efficiency. Also a 30-days study on the annealing of the
devices was performed.

Lastly, the conclusions are drawn on the basis of the Signal-to-Noise Ratio
(SNR), taking into account the standard parameters of famous satellites using
similar orbits as the ones considered into this work.

###### keywords:

Silicon photomultipliers , SiPM , Radiation damage , protons , noise

††journal: Nuclear Instrument and Methods in Physics

##  1 Introduction

Silicon Photo-multipliers (SiPMs) are solid-state detectors working in Geiger
mode based on an array of single-photons avalanche diodes (SPADs), thus with
an internal gain which enables them to have a good single-photon resolution.
They have experienced a large performance improvement over the last few years
and they are used in several fields, from nuclear medicine[1], high energy
physics[2], automotive, LIDAR[3]. Space is one of the applications where they
have been less explored so far[4], although several experiments are starting
to employ them[5]. One of the crucial limits of using SiPMs in HEP and space
applications is their radiation tolerance. Indeed, when exposed to a huge
amount of radiation, SiPMs could suffer a significant damage which, depending
on the type of radiation, might happen either into the bulk or on the surface
of the device[6]. In this sense, this work aims to evaluate the performance of
several FBK technologies after being exposed to proton fluences comparable
with the ones that are expected in space environment for satellites along some
of the most used orbits in space applications. In particular, we irradiated
the SiPMs at the Proton Therapy Center in Trento with 74 $MeV$ protons, at
fluences between $7.4\times 10^{6}$ $n_{eq}/cm^{2}$ and $6.4\times 10^{11}$
$n_{eq}/cm^{2}$.

Several FBK technologies were tested: VUV-HD 2019[7], NUV-HD-RH, NUV-HD-
cryo[8], RGB-HD and RGB-HD ”EnhancedBorder” [9]. The tested structures
included several $1\times 1$ $mm^{2}$ SiPMs manufactured with the same
technology, different cell pitches and structure. The different structures
were assembled on a PCB, which will be described in detail later, and placed
along a ring to obtain an irradiation as uniform as possible on each SiPM.

In the next sections, we will introduce the fluence values estimated from
simulations. Subsequently, we will show the experimental setup during the
irradiation tests, the results of the characterization measurements with a
focus on the comparison among the technologies under test and then the
conclusion will be derived in the last section.

##  2 Fluence Range Calculation

In our study, a typical satellite orbit was considered: 550 $km$ \- 97∘
inclination (circular Low Earth Orbits, Polar)[10]. The expected proton
fluence along this orbit was estimated through the use of SPENVIS
software[11], for a five years mission starting on July 1st 2022 with maximum
solar activity. SPENVIS provided as an output the spectra of total mission
average flux $[cm^{-2}s^{-1}MeV^{-1}]$ and total mission fluence
$[cm^{-2}MeV^{-1}]$ related to several contributions: protons and electrons
trapped into the terrestrial magnetic field, solar protons and galactic
protons. In Fig.1 the total mission particle fluences as a function of the
protons energy for trapped, solar, galactic protons and trapped electrons are
visible along the selected circular polar LEO orbit. The fluence spectrum of
each contribution (even from different particle types) was then converted into
an equivalent spectrum of 1 MeV neutron damage in Silicon, enabling us to
compare them. To do that, the NIEL scaling hypothesis was considered. For each
energy value of the spectrum $E$, the total displacement damage $DD(E)$
corresponding to the proton fluence $\phi(E)$ was calculated as:

$DD(E)=D(E)\cdot\phi(E)$ (1)

where $D(E)$ is the displacement damage cross section according to the NIEL
scaling hypothesis, which converts the effective particle damage into a 1 MeV
neutron equivalent damage. A plot of this spectrum is shown in Fig.2, where we
can clearly observe that the solar and trapped protons have the highest impact
on the total damage in Silicon, whereas the galactic protons and the trapped
electrons appear to affect it less. For this reason we chose to neglect these
two latter contributions, taking into account only the most relevant two.
Assuming a 80$\mu$m-thick Aluminum shielding of the whole detector, this
produces a cut-off of the protons with energy lower than 2.5 MeV and reduces
the total amount of damage on the sensors. SRIM software [12] was used for the
Aluminium shielding simulations. The resulting 2.5 MeV H ions energy loss in
80 $\mu$m Al target is visible in Fig.4, where a total absorption of the 2.5
MeV protons is visible at about 65 $\mu$m, thus a 80 $\mu$m Al shielding
results to be a reasonable choice to cut all the protons with energy lower
than 2.5 MeV.

The resulting equivalent damage spectrum was then integrated over the whole
protons energy range $[2.5MeV,E_{max}]$:

$D_{tot}=\int_{2.5MeV}^{E_{max}}DD(E)dE$ (2)

Figure 1:  Output plot from SPENVIS software showing the five years total
mission particle fluence as a function of the energy for trapped, solar,
galactic protons and trapped electrons, along a ”HERMES-like”” circular Low
Earth Orbit, Polar (550 $km$ \- 97∘).

The final step was to extract the proton fluence at 74 MeV required to create
such a total damage which means to normalize the total damage to the damage of
74 MeV protons, according to the equation:

$\phi(74MeV)=\frac{\int D(E)\phi(E)dE}{D(74MeV)}=\frac{D_{tot}}{D(74MeV)}$ (3)

where $D(E)$ is the 1 MeV neutron equivalent damage of the protons and
$D(74MeV)$ is the 1 MeV neutron equivalent damage of 74 MeV protons. In Fig.3,
the 74 MeV equivalent protons fluence from SPENVIS software related to
trapped, solar and total protons is plotted as a function of the Al shield
radius. Here, we can observe that at 80$\mu$m, we are at $4\times 10^{11}$ 74
MeV total protons, which means $6\times 10^{11}$ $n_{eq}/cm^{2}$.

Figure 2: Total damage of trapped, solar and galactic protons and trapped
electrons over the whole mission lifetime. The line marked at 2.5 MeV
represents the energy cut which is assumed to be applied to data. Figure 3:
Equivalent 74 $MeV$ protons fluence as a function of the radius of an Aluminum
spherical shielding along a Low Polar Orbit (550 $km$ \- 97∘) for trapped and
solar protons.

The fluence range considered in this work is $7.4\times 10^{4}\div 6.4\times
10^{11}$ $n_{eq}/cm^{2}$.

Figure 4: SRIM simulation of 2.5 MeV H ions energy loss in 80 $\mu$m Al target

The final proton irradiation fluences on the sensors were estimated
integrating the beam profiles on each SiPM, taking into account its specific
position within the beam spot.

A set of SiPMs was irradiated at $10^{11}$ $n_{eq}/cm^{2}$. This enabled us to
have a set of devices for further measurements apart from the parameters
estimated through the online current measurements. These SiPMs were kept in a
refrigerate environment ($-10^{\circ}C$), to avoid as much as possible the
annealing process.

##  3 Experimental setup

At the Trento Proton Therapy Center there are two possible irradiation
systems[13] with two different beam settings: a dual-ring double scattering
system[14] which ensures a large field irradiation at 148 MeV with a $98\%$
uniformity on a circular area with an approximately 5.9 cm diameter and a
gaussian-profiled beam [15] with a 7 mm $\sigma$ and a large energy range. In
particular, we decided to use the dual-ring system which provided an higher
uniformity. In this case, the standard 148 MeV energy was lowered to 74 MeV
through a Water-equivalent polystyrene material (RW3) resulting in a beam with
an equivalent uniform circular area with 7.8 cm radius.

The experimental setup included the beam output, an ionization chamber to
measure the total number of protons passing through it and a dark box, which
contained the devices during the irradiation and the measurements. In Fig.5
the setup for the dual-ring system is shown, where the dual ring and the RW3
blocks to lower the ions energy are visible.

The devices were irradiated in several steps and reverse current-voltage (I-V)
measurements were performed after each irradiation step to reduce the impact
of the annealing and to isolate the results of the damage effects. The devices
were placed on a custom Printed Circuit Board (PCB), where the SiPMs were
mounted for the measurement, as shown in Fig.6. The PCB included eight spots.
SiPMs on each structure were connected to the PCB with wire bonding and their
outputs connected through 2 m long cables to the measurement system. During
the whole irradiation time, two PCB where irradiated, each with a different
final fluence: $1.4\times 10^{8}$ $n_{eq}/mm^{2}$ and $6.4\times 10^{9}$
$n_{eq}/mm^{2}$. PCBs were inserted inside a custom dark box realized by 3D
plastic printing, with a motor-controlled light shutter. This was open during
the irradiation stage. After each irradiation step, the shutter was closed and
all the SiPMs were measured in dark conditions.

After irradiating and measuring the last PCB, which was the one with the
highest final fluence, the devices was kept in the same dark box and left to
anneal at room temperature ($20\div 25^{\circ}C$) for approximately 30 days.
During this time reverse I-V measurements were performed twice a day.

A set of SiPMs was irradiated as naked chips at a fluence of $10^{11}$
$n_{eq}/cm^{2}$ and then mounted on a PCB. This enabled us to have a set of
SiPMs irradiated at a lower fluence available for measurements of
characterization.

Figure 5: Irradiation setup with the double-ring scattering system at the
Proton Therapy Center in Trento, where the different setup components are
clearly visible: the ionization chamber (a), the PW3 used for the energy
lowering of the beam (b), the dark box including the irradiated sensors (c)
and the I-V measurement setup (d). Figure 6: Layout image of the PCB used for
the irradiation test

##  4 Results

###  4.1 Reverse Current and Breakdown voltage

The reverse current is one of the key parameters for the characterization of a
SiPM. In fact, when irradiated, the induced defects result in an increase of
both the ”leakage current” (not multiplied), from the surface and the ”dark
current” (multiplied), from the bulk of the SiPM [16]. The increase of these
parameters provide an indication of the quantity of damage that the sensor has
experienced. In this test, the measurement of the reverse current as a
function of the applied voltage was taken shortly after each irradiation step.
Each measurement was performed both in dark and light conditions through a
current-controlled LED inside the dark box. In Fig.7 the increase of the
reverse current with fluence can be observed for the NUV-HD-RH SiPM with 20
$\mu$m cell and the VUV-HD 2019 with 35 $\mu$m cell in dark conditions.

The increase of the current $\Delta I=I_{\phi}-I_{\phi_{0}}$ in dark
conditions was used to estimate the damage parameter $\alpha$ [6]. To do this,
the current increase was normalized to the current gain [16], the product of
the micro-cell gain and the excess charge factor (ECF), which represents the
ratio between the average charge produced by one primary event (including all
correlated secondary pulses generated by cross-talk and afterpulsing) and the
charge of just one primary event [17]. This parameter gives an idea of the
effective damage inside the SiPM and provide a first comparison among
different technologies. In Fig.8, the damage parameter $\alpha$ is plotted as
a function of the fluence at 3V excess bias, showing a fairly uniform trend of
all the tested technologies, apart form the RGB-HD SiPM with Enhanced border
20 $\mu m$ cell, which appears more affected by the radiation than the other
SiPMs, showing an increase of $\alpha$ starting from $10^{11}$
$n_{eq}/cm^{2}$.

Figure 7: Reverse current as a function of the bias voltage at the several
fluences for the NUV-HD-RH SiPM with 20$\mu$m cell (left) and the VUV-HD 2019
with 35$\mu$m cell (right) in dark conditions. Figure 8: Damage parameter of
the tested technologies as a function of the fluence at 3V excess bias.

From the reverse current measurement in light conditions, the breakdown
voltage was estimated from the second logarithmic derivative of the reverse
current, as visible in Fig.9. The breakdown voltage appears to remain constant
for almost all the technologies apart from the RGB-HD SiPM with Enhanced
border 20 $\mu m$ cell, where it starts to increase at $10^{11}$
$n_{eq}/cm^{2}$. This could mean that, at least in this latter technology, the
implant doping concentration of the single SPADs were damaged by the
radiation, while this does not hold true for the other technologies.

Figure 9: Breakdown voltage of the tested technologies as a function of the
fluence.

###  4.2 PDE

Photon Detection Efficiency (PDE) of the SiPMs is crucial to investigate their
ability in photo-detection. In highly-irradiated SiPMs, a direct measurement
of the PDE is not possible due to the increase of the dark current which
becomes comparable to the light current when the SiPMs are illuminated, thus
the measurement is no longer accurate. For this reason, the results on the PDE
were derived by a Responsivity measurement. That was performed on the samples
irradiated at $10^{11}$ $n_{eq}/cm^{2}$, which were illuminated by a 420 nm
LED. The results are visible in Fig.10, where the responsivity is plotted as a
function of the excess bias, for irradiated (straight line) and not-irradiated
(dashed line) samples. One sample for each technology was considered, and no
significant changes were observed in any of them. This means that the PDE
remains constant with fluence, at least up to $10^{11}$ $n_{eq}/cm^{2}$ in all
the technologies under test.

Figure 10: Responsivity as a function of the excess bias for RGB-HD, NUV-HD-
RH, VUV-HD 2019 and NUV-HD-cryo technologies, irradiated at $10^{11}$
$n_{eq}/cm^{2}$ (straight line) and not-irradiated (dashed line).

###  4.3 DCR

Dark Count Rate is one of the most important parameters in a SiPM. A direct
measurement of the DCR in highly-irradiated devices can be very difficult,
because of the high count rate which prevents the use of the common DCR
characterization methods based on a waveform analysis. For this reason, we
chose to extract the DCR from the dark current and the so-called Current Gain
$G_{c}$, which was previously introduced. The DCR can be estimated as:

$DCR=\frac{1}{q}\frac{I_{dark}}{G_{c}}$ (4)

In irradiated SiPMs, Eq.4 holds under the assumption that the current gain
measured before irradiation remains constant with the fluence[16]. To verify
this assumption, we performed a direct $G_{c}$ measurement on the samples
irradiated at $10^{11}$ $n_{eq}/cm^{2}$ at -40∘C, due to the high noise rate
which did not make a direct measurement at lower temperature possible. The
results are visible in Fig.11, where the current gain is plotted as a function
of the excess bias. The plot does not shown any relevant changes, suggesting
that the current gain is not affected by the radiation, at least up to
$10^{11}$ $n_{eq}/cm^{2}$.

Figure 11: Gain current as a function of the excess bias mneasured at -40∘C
for the RGB-HD, NUV-HD-RH, VUV-HD 2019 and NUV-HD-cryo technologies,
irradiated at $10^{11}$ $n_{eq}/cm^{2}$.

After verifying the current gain does not change with fluence, the DCR was
extracted at all the fluences for all the technologies under test, as shown in
Fig.12, where the curves of the several devices look more similar as the
fluence increases, not showing any significant trend. The RGB-HD SiPM with
Enhanced border 20 $\mu m$ cell appears more damaged than the others, starting
from $10^{11}$ $n_{eq}/cm^{2}$, as already observed in the case of the damage
parameter $\alpha$ and the breakdown voltage.

Figure 12: Dark Count Rate of the tested SiPMs as a function of the
irradiation fluence.

Although not obvious from Fig.12, some SiPMs, mainly the larger-sized ones,
exhibit some saturation effects at high fluences. SiPM saturation (due to the
high cell occupancy) means that the DCR values are comparable to the maximum
possible DCR of a SiPM with all cells constantly triggering[16] and it appears
as a anomalous deflection.

In Fig.13 a plot of the SiPM saturation (or cell occupancy) as a function of
the fluence at 3V excess bias is visible, where we observe a more clear
deflection of the curve for large-sized cell SiPMs, whereas the curve of
small-sized cell SiPMs, remains almost constant.

Figure 13: Saturation of SiPM cells as a function of the fluence at 3V excess
bias, with visible effects on the devices with larger cell pitches.

###  4.4 SNR and energy resolution

The results shown above are typical functional parameters of SiPMs. In order
to compare the technologies under test and select the most promising ones for
space applications, other parameters should be considered to extract figure of
merits more related to experiments operating in space.

An example of figure in merit useful to compare the SiPMs performance before
and after irradiation is the detector Signal-to-Noise (SNR) ratio which, for
pulse counting detectors, is defined as:

$SNR=\frac{N_{t}^{tot}-N_{t}^{noise}}{\sqrt{ENF\cdot N_{t}^{tot}}}$ (5)

where ENF is the Excess Noise Factor, i.e. a measure of the ”inaccuracy” of
the sensor and:

$\displaystyle N_{t}^{tot}$
$\displaystyle=N_{cell}\frac{t_{int}}{\tau}\Big{(}1-e^{-\frac{1+CT}{N_{cell}}(N_{ph/s}\cdot
t_{int}\cdot PDE+DCR\cdot t_{int})\frac{\tau}{t_{int}}}\Big{)}$ (6)
$\displaystyle N_{t}^{noise}$
$\displaystyle=N_{cell}\frac{t_{int}}{\tau}\Big{(}1-e^{-\frac{1+CT}{N_{cell}}(DCR\cdot
t_{int})\frac{\tau}{t_{int}}}\Big{)}$ (7)

where $N_{t}^{tot}$ is the number of SPADs triggered by the total light and
noise events in a certain integration time $t_{int}$, $N_{t}^{noise}$ is the
number of SPADs triggered by noise events in the same integration time,
$N_{ph/s}$ is the number of incident photons per second and $\tau$ is the
recharge time of the cells.

Fig.14 shows the trend of the SNR as a function of the number of incident
photons, at fixed integration time ($2\times 10^{-6}$) and fluence ($6.4\times
10^{11}$ $n_{eq}/cm^{2}$). Here we observe a better performance of the large-
sized SiPMs due to their high PDE up to $10^{10}$ ph/s, where they start to
suffer a saturation process, whereas the small-sized SiPMs appear less
affected by the saturation.

In order to extract quantitative values, thus compare the SiPM technologies,
we needed to target a specific case. For example, the number of incident 450nm
photons per second was fixed at $10^{10}$ $ph/s$ to get an high SNR without
considering saturation effects, whereas the integration time was set to
$2\times 10^{-6}s$ as in the ALPIDE Monolithic Active Pixel Silicon
sensor[18], developed at CERN for the ALICE ITS Upgrade and also considered
for future satellite particle trackers.

In Fig.15 the SNR curves are plotted as a function of the fluence at fixed
number of incident photons per second ($10^{10}$ $ph/s$) and integration time
window ($2\times 10^{-6}s$). Here we can see a general worsening of the
performance of the sensors at around $10^{11}$ $n_{eq}/cm^{2}$. Among the
SiPMs with 20 $\mu$m cell pitch, we observe a low SNR (around 200) for the
NIR-HD, NUV-HD cryo and RGB-HD technologies, whereas the NUV-HD-RH and NUV-HD-
cryo with 20 $\mu$m cell pitch technologies show a better performance. In
particular, the most significant result comes from the NUV-HD-RH SiPM with 15
$\mu$m cell which show a high SNR at at 440. Among the large-sized SiPMs,
which are supposed to have an higher SNR due to their high FF and PDE but also
an earlier saturation, the NUV-HD-cryo technology with 40 $\mu$m cell shows
the highest SNR.

Figure 14: Signal-to-noise ratio as a function of the fluence at fixed excess
bias (3V), integration time ($6.8\times 10^{-8}$ s) and fluence ($6.4\times
10^{11}$ $n_{eq}/cm^{2}$). Figure 15: Signal-to-noise ratio as a function of
the fluence at fixed excess bias (3V), integration time ($6.8\times 10^{-8}$
s) and number of photons ($2\times 10^{10}$).

In X or $\gamma$-ray spectroscopy, the energy resolution should be taken into
account:

$\sigma_{E}\propto\sqrt{\frac{ENF^{*}}{N_{ph/s}\cdot t_{int}\cdot
PDE}+\frac{ENF^{*}\cdot DCR\cdot t_{int}}{(N_{ph/s}\cdot t_{int}\cdot
PDE)^{2}}}$ (8)

where ENF∗ is the Excess Noise Factor, in this case estimated as product of
several contributions, as in [19].

The results of the energy resolution estimation are visible in Fig.16, where a
clear distinction between the SiPMs with small (15-20 $\mu$m) and large (35-40
$\mu$m) cell pitches can be noted. In particular, the large-sized ones have an
higher $\sigma_{E}$, at around 1$\%$. A trend with the cell size can be
observed, apart form the RGB-HD SiPM with Enhanced border 20 $\mu m$ cell
which has an higher $\sigma_{E}$, nearly 1$\%$, at the same level of the 35
$\mu$m cell SiPMs. Furthermore, it starts to have a dramatic increase at
$10^{11}$ $n_{eq}/cm^{2}$ until reaching a 2$\%$ value.

Figure 16: Energy resolution as a function of the fluence at fixed excess bias
(3V), integration time ($6.8\times 10^{-8}$ s) and number of photons ($2\times
10^{10}$).

Overall, NUV-HD-RH and NUV-HD-cryo technologies show the best performance
under the conditions described above after irradiation, with an higher SNR and
a lower energy resolution. It has to be pointed out that this result is merely
related to the application that we took into account, in this particular case
the X and $\gamma$-ray spectroscopy.

###  4.5 Annealing

A 30-days annealing test on all the irradiated SiPMs was performed at room
temperature ($20\div 25^{\circ}C$), with reverse current measurements twice
per day. The normalized dark currents are plotted in Fig.17 at 3V of excess
bias. Here, the RGB-HD SiPMs appear to have the fastest recovery. Indeed, the
RGB-HD SiPM with Enhanced border $20\mu m$ cell reduces its current of almost
a factor 0.4 after the whole annealing time, while the RGB-HD SiPM with
standard 20 $\mu m$ cell, after a fast initial recovery starts to slow after
100 minutes. The other technologies show an initial constant decrease until
$10^{3}$ minutes, where they present a knee and then a faster recovery. As
visible in the plot, after the whole 30-days annealing time, the RGB-HD
Enhanced Border, NUV-HD-RH and NUV-HD-cryo 20$\mu$m technologies are the ones
recovering the most, while the VUV-HD and NUV-HD-cryo 40$\mu$m appear to
recover the less.

Figure 17: $I(t)/I(t_{0})$ ratio as a function of the annealing time after
irradiation at room temperature ($20\div 25^{\circ}C$).

##  5 Conclusions

An irradiation test was performed at the Trento Proton Therapy Center in
February 2021. Several FBK technologies with different cell piches and shapes
were irradiated with 74 MeV protons at gradual fluence steps, up to $6.4\times
10^{11}$ $n_{eq}/cm^{2}$.

After a reverse current measurement, the damage factor $\alpha$ was extracted.
This did not show any clear trend for all the technologies apart from the RGB-
HD SiPM with Enhanced border 20 $\mu m$ cell, which appears to be more damaged
starting from $10^{11}$ $n_{eq}/cm^{2}$. The same behaviour can be observed
for the breakdown voltage, where the RGB-HD SiPM with Enhanced border 20 $\mu
m$ cell is the only technology to suffer from the irradiation. The DCR in all
the different technologies starts at different levels but slowly catch each
other up at increasing fluences, until reaching similar values. The RGB-HD
SiPM with Enhanced border 20 $\mu m$ cell shows a deflection at $10^{11}$
$n_{eq}/cm^{2}$, which is a result consistent with what observed for the
damage parameter and the breakdown voltage. At high fluences some of the SiPMs
under test present a saturation effect, which is particularly prominent for
large-sized SiPMs and, nonetheless, it is a parameter to take into account
during the whole characterization process.

An effective comparison among the technologies can be achieved through the
estimation of the energy resolution and the signal-to-noise ratio, which
provide information about the response of the sensor when exposed to a photon
flux. These two parameters were estimated taking into account the saturation
of the cell and the total excess noise factor from different noise sources.
The SNR is used in this work as a figure of merit. It shows a trend with the
cell size at fixed fluence. In particular, at $6.4\times 10^{11}$
$n_{eq}/cm^{2}$ the larged-sized SiPMs exhibit an higher SNR with a clear
saturation starting at at around $10^{10}$ ph/s. At fixed number of incident
photons per second, the NUV-HD-RH 20 $\mu$m and NUV-HD-cryo technologies show
the highest SNR values, in case of small cells.

After irradiation, a 30-days annealing at room temperature was performed on
the SiPMs. The RGB-HD SiPM with Enhanced border 20 $\mu m$ cell showed a
faster recovery, whereas the other technologies exhibited a very similar
behaviour with a constant decrease followed by a knee and a faster decrease,
up to around 50$\%$ recovery factor.

This work had as a main purpose a very preliminary characterization of the FBK
SiPMs for space applications. To do that, typical fluences experienced by
satellites in space were considered. All the SiPMs survived to the irradiation
tests and only the RGB-HD SiPM with Enhanced border 20 $\mu m$ cell was
observed to deviate significantly from the behaviour of the other technologies
at fluences above $10^{11}$ $n_{eq}/cm^{2}$. Overall, due to the higher SNR
and the fast recovery during the annealing time, the NUV-HD-RH and the NUV-HD-
cryo 20 $\mu$m technologies showed the best performance. For the NUV-HD-RH,
two cell sizes were tested and the 15 $\mu$m SiPM displayed a better behaviour
than the 20 $\mu$m SiPM.

## References

  * Llosa et al. [2007] G. Llosa, N. Belcari, M. G. Bisogni, G. Collazuol, A. D. Guerra, S. Marcatili, S. Moehrs, C. Piemonte,  Silicon photomultipliers and sipm matrices as photodetectors in nuclear medicine,  in: 2007 IEEE Nuclear Science Symposium Conference Record, volume 5, pp. 3220–3223.
  * Musienko et al. [2016] Y. Musienko, A. Heering, M. Wayne, R. Ruchti, A. Karneyeu, V. Postoev,  Studies of SiPMs for the CMS HCAL Upgrade,  PoS PhotoDet2015 (2016) 038.
  * Son and Lee [2010] K. T. Son, C. C. Lee,  Multiple-target laser rangefinding receiver using a silicon photomultiplier array,  IEEE Transactions on Instrumentation and Measurement 59 (2010) 3005–3011.
  * Anisimova et al. [2017] E. Anisimova, B. Higgins, J.-P. Bourgoin, M. Cranmer, E. Choi, D. Hudson, L. Piche, A. Scott, V. Makarov, T. Jennewein,  Mitigating radiation damage of single photon detectors for space applications,  EPJ Quantum Technology 4 (2017).
  * Gargano et al. [2019] F. Gargano, A. D. Benedittis, P. d. Luque, F. Palma, M. D. Santo, L. D. Venere, P. Fusco, F. Gargano, F. Giordano, F. Loparco, S. Loporchio, M. N. Mazziotta, M. Mongelli, D. Serini, G. T. Elipe, Z. M. Wang,  Characterization of plastic scintillator tiles equipped with SiPMs for the High Energy cosmic-Radiation Detection (HERD) experiment,  PoS ICRC2019 (2019) 069.
  * Moll [1999] M. Moll, Radiation damage in silicon particle detectors: Microscopic defects and macroscopic properties, Ph.D. thesis, Hamburg U., 1999.
  * Capasso et al. [2020] M. Capasso, F. Acerbi, G. Borghi, A. Ficorella, N. Furlan, A. Mazzi, S. Merzi, V. Mozharov, V. Regazzoni, N. Zorzi, G. Paternoster, A. Gola,  FBK VUV-sensitive Silicon Photomultipliers for cryogenic temperatures,  Nuclear Instruments and Methods in Physics Research A 982 (2020) 164478.
  * Gola et al. [2019] A. Gola, F. Acerbi, M. Capasso, M. Marcante, A. Mazzi, G. Paternoster, C. Piemonte, V. Regazzoni, N. Zorzi,  Nuv-sensitive silicon photomultiplier technologies developed at fondazione bruno kessler,  Sensors 19 (2019).
  * Acerbi et al. [2018] F. Acerbi, G. Paternoster, A. Gola, V. Regazzoni, N. Zorzi, C. Piemonte,  High-density silicon photomultipliers: Performance and linearity evaluation for high efficiency and dynamic-range applications,  IEEE Journal of Quantum Electronics 54 (2018) 1–7.
  * Fiore et al. [2021] F. Fiore, et al.,  The HERMES-Technologic and Scientific Pathfinder,  in: SPIE Astronomical Telescopes + Instrumentation 2020.
  * Kruglanski et al. [2010] M. Kruglanski, E. de Donder, N. Messios, L. Hetey, S. Calders, H. Evans, E. Daly,  Space Environment Information System (SPENVIS),  in: 38th COSPAR Scientific Assembly, volume 38, p. 8.
  * Ziegler et al. [2010] J. F. Ziegler, M. D. Ziegler, J. P. Biersack,  SRIM - The stopping and range of ions in matter (2010),  Nuclear Instruments and Methods in Physics Research B 268 (2010) 1818–1823.
  * Di Ruzza [2021] B. Di Ruzza,  Proton and x-ray irradiation of silicon devices at the TIFPA-INFN facilities in Trento,  in: Proceedings of 40th International Conference on High Energy physics — PoS(ICHEP2020), volume 390, p. 685.
  * Tommasino et al. [2019] F. Tommasino, M. Rovituso, E. Bortoli, C. La Tessa, G. Petringa, S. Lorentini, E. Verroi, Y. Simeonov, U. Weber, P. Cirrone, M. Schwarz, M. Durante, E. Scifoni,  A new facility for proton radiobiology at the trento proton therapy centre: Design and implementation,  Physica Medica 58 (2019) 99–106.
  * Tommasino et al. [2017] F. Tommasino, M. Rovituso, S. Fabiano, S. Piffer, C. Manea, S. Lorentini, S. Lanzone, Z. Wang, M. Pasini, W. Burger, C. La Tessa, E. Scifoni, M. Schwarz, M. Durante,  Proton beam characterization in the experimental room of the trento proton therapy facility,  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 869 (2017) 15–20.
  * Altamura et al. [2021] A. R. Altamura, F. Acerbi, C. Nociforo, V. Regazzoni, A. Mazzi, A. Gola, Characterization of silicon photomultipliers after proton irradiation up to $10^{12}n_{eq}/mm^{2}$, 2021.
  * Piemonte and Gola [2019] C. Piemonte, A. Gola,  Overview on the main parameters and technology of modern silicon photomultipliers,  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 926 (2019) 2–15. Silicon Photomultipliers: Technology, Characterisation and Applications.
  * Aglieri Rinella [2017] G. Aglieri Rinella,  The ALPIDE pixel sensor chip for the upgrade of the ALICE Inner Tracking System,  Nucl. Instrum. Methods Phys. Res., A 845 (2017) 583–587.
  * Vinogradov [2016] S. Vinogradov,  Perfomance of silicon photomultipliers in photon number and time resolution,  p. 002.