# Structural Health Monitoring system with Narrowband IoT and MEMS sensors

Flavio Di Nuzzo, Davide Brunelli, , Tommaso Polonelli, ,
and Luca Benini This work was supported by Sacertis S.r.l. and by the Italian
Ministry for Education, University and Research (MIUR) under the program
“Dipartimenti di Eccellenza (2018-2022)”. Moreover this research is funded by
ECSEL, the Electronic Components and Systems for European Leadership Joint
Undertaking under grant agreement No 826452 (Arrowhead Tools), supported by
the European Union Horizon 2020 research and innovation programme and by the
member states. F. Di Nuzzo, T. Polonelli and L. Benini are with the Department
of Electrical, Electronic and Information Engineering, University of Bologna,
40136 Bologna, Italy.are with University of Bologna, Bologna, Italy (e-mail:
{name.surname}@unibo.it). D. Brunelli is with the Department of Industrial
engineering, University of Trento, 38123 Trento, Italy (e-mail:
davide.brunelli@unitn.it).L. Benini is also with the Department of Information
Technology and Electrical Engineering at the ETH Zurich, 8092 Zurich,
Switzerland. (e-mail: lbenini@iis.ee.ethz.ch). This article has been accepted
for publication in a future issue of this journal, but has not been fully
edited. Content may change prior to final publication. Citation information:
DOI 10.1109/JSEN.2021.3075093, IEEESensors JournalPersonal use is permitted,
but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for
more information.

###### Abstract

Monitoring of civil infrastructures is critically needed to track aging,
damages and ultimately to prevent severe failures which can endanger many
lives. The ability to monitor in a continuous and fine-grained fashion the
integrity of a wide variety of buildings, referred to as structural health
monitoring, with low-cost, long-term and continuous measurements is essential
from both an economic and a life-safety standpoint. To address these needs, we
propose a low-cost wireless sensor node specifically designed to support modal
analysis over extended periods of time with long-range connectivity at low
power consumption. Our design uses very cost-effective MEMS accelerometers and
exploits the Narrowband IoT protocol (NB-IoT) to establish long-distance
connection with 4G infrastructure networks. Long-range wireless connectivity,
cabling-free installation and multi-year lifetime are a unique combination of
features, not available, to the best of our knowledge, in any commercial or
research device. We discuss in detail the hardware architecture and power
management of the node. Experimental tests demonstrate a lifetime of more than
ten years with a 17000 mAh battery or completely energy-neutral operation with
a small solar panel (60 mm x 120 mm). Further, we validate measurement
accuracy and confirm the feasibility of modal analysis with the MEMS sensors:
compared with a high-precision instrument based on a piezoelectric transducer,
our sensor node achieves a maximum difference of 0.08% at a small fraction of
the cost and power consumption.

###### Index Terms:

Internet of Things, IoT, Narrowband IoT, MEMS, Structural Health Monitoring,
SHM, Smart Sensor Systems, Sensor Communications

##  I Introduction

In our cities, buildings and civil infrastructures are increasing every year
[1]. New materials and techniques applied to complex structures create the
need for continuous and autonomous monitoring of the structural integrity of
the buildings over many years [1, 2]. Furthermore, old structures and
facilities such as bridges, viaducts, roads, and apartments have to be
continuously checked to guarantee the safety of thousands of people: in fact,
old civil and mechanical engineering structures continue to be used daily
despite aging, deterioration, and exceedance of their operational lifespan.
For instance, between 2013 and 2018, one out of nine of the United States
bridges were structurally deficient, while in Italy, 13 bridges collapsed, in
addition to a 210 meters highway bridge breakdown, which killed 43 people [2].
According to these facts, the ability to monitor a wide variety of buildings
with low-cost and real-time measurements is essential from both an economic
and a life-safety standpoint. The process of continuously monitoring the
integrity and the response of a structure is referred to as Structural Health
Monitoring (SHM) [3].

The primary function of an SHM system is to provide a continuous assessment of
structural integrity and promptly identify, or even predict, damages and
abnormal structure behaviors while generating periodic reports about the
structure state [3]. In particular, civil engineers and inspectors need
periodic data to assess the mechanical performance over long time intervals,
typically many years, and to analyze the structure’s long-term response to
environmental and daily stresses (i.e., weather, wind, weight load, corrosion,
or act of vandalism). Both static and dynamic measurements are required to
achieve this goal: corrosion and crack monitoring are typical static
measurements, while modal analysis [4] needs dynamic (velocity, acceleration)
measurements. A SHM system comprises four blocks: i) sensors and transducers,
ii) remote communication, iii) data storage, and iv) feature extraction and
data processing. Depending upon the SHM installation complexity and operation
context, the system could work in small scale, for example, on a single
structure or exploiting many thousands of sensors with national coverage. In
this paper, we focus our effort on the second case, providing support for
sparse installation of heterogeneous IoT devices for national or
internationals SHM networks.

Historically, SHM devices were produced using wired networks; however, the low
installation and maintenance costs and high reliability of state-of-the-art
LPWAN (Low-Power Wireless Area Network) and Internet of Things (IoT) protocols
have made them a compelling alternative solution for industrial deployments
[5, 6]. Indeed, due to their high installation costs, wired systems are
generally available only for long-term monitoring of important key structures.
SHM wireless sensors enable significant cost reductions [7, 8], allowing a
massive utilization in public and private infrastructures. Moreover, such
sensors are also suitable for short-term structural monitoring applications,
exploiting single clusters of devices for multiple buildings. Previous works
have pointed out that, concerning power consumption, coverage, and security,
LoRaWAN [9] and NB-IoT [10, 11] are the most suited LPWAN protocols in
distributed IoT applications. In addition to the installation, the second
limiting factor comes from transducers [12]. Indeed, it is not obvious which
kind of behavior better describes the structure health, and the sensor cost is
itself a major limiting factor. Ordinarily, to read the small vibrations and
oscillation frequency components of a building with sufficient precision
($10^{-2}m\cdot s^{-2}$) [12], piezoelectric monoaxial sensors are used.
Furthermore, to characterize a wall crack through time and space, LVDT (Linear
Variable Differential Transformers), optical-fiber based sensors [13], and
laser sensors are often exploited, which a sensitivity below one micrometer.
They are usually noted for their high cost, characterized by the transducer
and the complex analog front-end. Moreover, the harness’s deployment to
connect several devices in a large and external environment complicates the
installation and raises the costs. Due to these facts, SHM systems are not
commonplace and widespread as they should [14].

To overcome these limitations, previous works proposed the integration of
Micro-Electro-Mechanical Systems (MEMS) in SHM devices for making low-cost
monitoring systems [12, 15]. Currently, structural instrumentation using MEMS
provides low-cost installation, moderate invasive effects, and equivalent
performance compared to their macro-scale counterparts [12]. Off-the-shelf
MEMS integrated System on Chips (SoCs) are appealing for SHM application as
they measure vibrations from $10^{1}m\cdot s^{-2}$ (severe shaking) to
$10^{-2}m\cdot s^{-2}$ (micro-vibration) on large-scale structures having
natural oscillation frequencies in the range of $10^{-1}$ to $10^{1}$Hz [16].
Moreover, it has been demonstrated that data from MEMS inclinometers can also
provide additional information about the structural health of the system with
respect to an increasing traffic load [17].

This work presents a low-cost wireless SHM monitoring sensor board that uses
low-cost MEMS sensors and data communication using the NB-IoT protocol. It is
specifically designed to support building modal analysis through low-cost
(price between 1.83€ and 2.17€ at 1kunit) commercial MEMS sensors to
facilitate wide-scale SHM systems’ deployments. Using NB-IoT communication,
the sensor board guarantees extremely low power consumption ($125.4~{}mW$ run,
$145.2~{}\mu W$ sleep) and capable of global (nation-scale) connectivity
through the NB-IoT infrastructure network.

A combination of onboard processing and data aggregation decreases the
communication energy, $50\mu J$ per bit on average, by exploiting the
floating-point unit (FPU) performance of the STML4 MCU series Indeed, applying
the edge-computing methodology in our sensor board enables a data compression
rate of 256. For high-accuracy and short term deployment using a non-
rechargeable thionyl chloride 17000 mAh battery, our device reaches 214 days
of operativity. Moreover, for long-term monitoring using the same battery, the
10 years threshold is achieved, considering 6 samples per day, in which 3 MB
of vibration values compose each sample. The node has also been tested with
onboard solar energy harvester support self-sustainability using a
$60~{}mm\times 120~{}mm$ solar panel and a Li-On battery, used as an energy
buffer, of $5400~{}mAh$.

The measurement precision of our device is comparable with piezoelectric
sensors. To achieve this result, we performed a comprehensive MEMS assessment
to select the best commercial product in comparison with high precision
piezoelectric sensor. Our device measurements perfectly match results from
expensive and power-hungry piezoelectric sensors, providing an error lower
than 0.08% in frequency peak detection of modal vibration tones.

The rest of the paper is organized as follows. Related work is reviewed in
Section II. Section III presents a design exploration of NB-IoT, providing
consumption analysis and discussion of the limits. Performance comparison of
three commercial modules has been discussed to justify the more suitable
communication module for the designed SHM node. We present the hardware and
software design of our SHM board in Section IV. An energy consumption model is
elaborated in Section V, and it is also validated with laboratory
measurements. Section VI describes all laboratory simulations and results to
test and evaluate the board’s performance compared to state-of-art sensors.
Section VII concludes the paper.

##  II Related Work

Structural Health Monitoring (SHM) has been studied for some decades already;
as we can see in [18] studying the modal vibration of a structure, it is
possible to detect damages in advance. Furthermore, more recent studies
developed approaches of damage detection based on autoregressive (AR) time
series and damage sensitive features (DSFs) defined as a function of the AR
coefficients [19]. Another example is given by [20] where the AR coefficients
are calculated with autoregressive moving average (ARMA) processes and then
given to a Gaussian Mixture Model (GMM) to model the feature vector. These two
algorithms permit to detect damage, but also to locate it in the structure
and, most importantly, they use vibration data coming from inertial sensors.
Considering the improvements in microcontrollers’ computational performance
and considering the last generation of MEMS with higher precision and newer
features of machine learning, it’s possible to perform SHM in an embedded
system.

A major trend in SHM is to replace expensive piezoelectric sensors with more
affordable MEMS sensors. In [21], a comparison between a piezoelectric sensor
and a MEMS sensor for vibration-based SHM is presented, demonstrating that
MEMS is a viable technology for SHM with a remarkable cost reduction. Another
example is given by [15],which has been validated in production and serves as
the initial reference for this paper. The main limitations of this design are:
(i) it uses short-range wireless communication for data (Wi-Fi) that is also
power-hungry (0.5 W); (ii)it is not designed for battery operation, and
requires power cabling.

In [22], a sensor node that measures and monitors cracks in concrete for SHM
purposes is proposed. The sensor node uses the LoRaWAN protocol to communicate
data, and it guarantees a battery lifetime of more than 10 years. Highly
accurate static measurements of crack’s amplitude are demonstrated along with
Long-range connectivity. However, the bandwidth and sample rate of crack
metering is orders of magnitude lower than what is needed for vibration-based
SHM. Concerning static measurements [13, 23, 22], for example, cracks width or
corrosion, it is possible to detect damages in structures using strain gauges.
These measurements have low bandwidth and can be performed with very low power
sensor and electronics, which can survive in the building for all its
lifetime. Our work focuses on much higher bandwidth vibration-based analysis,
which is as essential for SHM as static analysis since it provides
complementary information [15, 21, 24]. Ideally, dynamic analysis should be
performed in a continuous, long term fashion, in parallel with static
analysis, but it generally is not, because of power and cost concerns.

The designs proposed so far in the literature and in the industrial practice
are not suitable for vibration-based (dynamic), battery-powered SHM with long
lifetime. In this work, we improve over previous designs in several
directions. First, we use the NB-IoT protocol for our communication link and
leverage its advantages over LoRaWAN [22]: NB-IoT uses a licensed band, so
there is no limitation on band occupation; moreover, there is no need of
gateways because the node connects directly with mobile operator’s base
stations. Note that, LoRaWAN protocol is generally less energy consuming than
NB-IoT [10] at low datarates, but it is not adequate for high datarate SHM
applications that need thousand of KBytes of data transferred. The NB-IoT
payload is 3 times bigger than LoRaWAN (1500 Bytes against 300 Bytes in
specific condition); furthermore, NB-IoT datarate is $200~{}kbps$ against
$50~{}kbps$ of LoRaWAN [25]. Considering a long term application as in this
work, 12kB per inertial measurement session must be sent, for a total of 6
sessions. 72kB per day is unfeasible using LoRaWAN protocol because of the low
data rate and, more importantly, for the regulated band occupation that only
permits very low duty-cycle. NB-IoT has no limitation on data transmission,
and the energy per bit (EPB) with big payload ($>$5kB) is affordable, as it is
lower than $50\mu J$ (as shown in Table I in Section IV).

After evaluating the performance of MEMS analog sensor presented in [12], we
decided to use in our work the LIS344ALH by ST Microelectronics because of its
leading edge characteristics among MEMS sensors. In fact, LIS344ALH has a
noise density of $50~{}\mu g/\sqrt{Hz}$ (the lowest value in this class of
devices), and its performance is confirmed in [15]. Moreover, thanks to the
efficiency of the microcontroller STM32 series L4 from ST Microelectronics, we
implemented an acquisition chain designed to oversample and filter the data
coming from the sensor, achieving 16 bit ADC resolution.

To the best of our knowledge, there are no other devices that exploit the NB-
IoT protocol to enable vibration-based SHM. In literature,[26, 27] provides
several examples and evaluations on wireless communication applied to SHM, but
none of them use NB-IoT. Also, in [24] are presented some examples of SHM
systems based on MEMS sensors, but in all proposed solutions, the usage of NB-
IoT is not considered.

##  III Background

###  III-A Narrowband IoT

The NB-IoT is a Low Power extension of the LTE (4G Long Term Evolution)
developed for long battery lifetime and low cost application. Its main
features are reduced power consumption, extended coverage extension, user
equipment cost reduction, and backward compatibility [28]. The NB-IoT’s power
consumption depends on multiple environment-related factors, such as the
country and network operator settings, which can drastically change the end-
device performance. NB-IoT is standardized by 3GPP for LPWANs thanks to its
capability to work virtually everywhere (e.g., TIM guarantees a country level
coverage in Italy for NB-IoT) [11].

The maximum payload for each message is 1500 B, and the full data transmission
rate is limited to $20~{}kbps$ uplink and $200~{}kbps$ downlink. The minimum
bandwidth is 180 kHz, corresponding to the size of the smallest LTE Physical
Resource Block (PRB). As presented in [29], this protocol is meant for
extended battery life applications of more than ten years when transmitting
200 bytes per day, features that fit into classic SHM requirements. Low power
consumption is achieved by NB-IoT using the LTE energy-saving mechanisms,
extending the inactivity periods to minimize energy consumption. As reported
in [30], the RCC state model (LTE Radio Resource Control) protocol has only
two states: RCC Connected and RCC Idle (Fig. 1 \- RCC), the cell handover and
redirection is not supported in NB-IoT release 13. NB-IoT features a
Discontinuous Reception (DRX) mechanism, where the module alternates active
listening for Paging Occasions and sleep periods. DRX settings are defined in
multiple subframes of 1 ms, ranging from 256 ms to 9216 ms against 10 ms to
2560 ms of LTE to reduce power consumption. NB-IoT defines it enhanced DRX in
RRC Connected mode (C-eDRX) (Fig. 1 \- C-eDRX Cycles). In RCC Idle, the eDRX
in Idle state (I-eDRX) (Fig. 1 \- I-eDRX Cycles) is equal to the one in LTE,
but also, in this case, the timings can be larger It can return in RCC
Connected state at any time, using the same context saved during the
initialization (first connection to the mobile operator’s cell). The
reconnection process from PSM state to RCC connected state decreases latency
and energy consumption compared to establishing a new connection in the
initialization phase. T3412 and T3324 set the duration of PSM and I-eDRX,
respectively (Extended Timer and the Active Timer (Fig. 1 \- T3412 and T3324).
The Active Timer (T3324) determines the duration of the I-eDRX after entering
RCC Idle state, and the Extended Timer (T3412) sets the period of the Tracking
Area Update (TAU). The TAU is the same as that of LTE, but NB-IoT can also
configure a longer period of up to 413 days (spent in PSM).

Figure 1: Behaviour of a NB-IoT module with description

###  III-B NB-IoT module: a comparative study

We compared three different modules available on the market: the Quectel BG96,
the Quectel BC95-G and, the U-Blox Sara N211. The Sara N211 module is mounted
on the development board C030-N211 produced by U-Blox; the modules from
Quectel are mounted on development boards build by MIKROE, called MIKROE-3144
and MIKROE-3294, respectively for the BG96 and the BC95-G. We mainly focused
on each module’s energy consumption in PSM and the energy per packet required
for each uplink in different coverage conditions.

The U-Blox dev-board also mounts a current measuring circuit, consisting of a
shunt resistor placed on the supply voltage and a differential amplifier based
on the TSZ121 operational amplifier. For a fair comparison, we built the same
current measurement setup on the MIKROE dev-boards. We sent the same packet
using UDP protocol and measured the energy, latency, and reliability.
Moreover, for each transmission, we assessed the RSSI (Received Signal
Strength Indicator) to classify the level of coverage. Using a -20 dB coaxial
passive attenuator, we simulate different connection conditions. Results in
Fig. 2 gather all the acquired data. The vertical axis represents the power
consumption expressed in Joule, and the horizontal one shows the RSSI in dBm.

Figure 2: Power Consumption in transmission: Quectel BC95, Quectel BG96 and
U-Blox Sara N211

Taking as a reference Fig. 2, the following considerations are validated. The
Quectel BG96 has higher energy consumption; indeed, it consumes 71% more on
average compared to the Quectel BC95-G and the U-Blox Sara N211; this fact is
given by the extra functionalities of this module. The Quectel BG96 can also
operate as an LTE-M module or GPRS (General Packet Radio Service) module;
furthermore, it has additional interfaces and even a GNSS (Global Navigation
Satellite System) module. The U-Blox Sara N211 and the Quectel BC95-G are
comparable; in good coverage, the Quectel BC95-G consumes 15% less energy on
average than the U-Blox Sara N211; in medium coverage, the energy consumption
is almost the same.

We selected the Quectel BC95-G instead of the BG96 because of the lower
average energy consumption and for the NB-IoT focused firmware. In fact, the
latter provides a set of proprietary AT commands with extended functionalities
not needed in our application scenario, such as commands to configure and use
the MQTT (Message Queue Telemetry Transport) protocol or to enable LTE and
GNSS functionalities.

###  III-C Quectel BC95-G characterization and energy considerations

We measured the energy consumption of the BC95-G in different conditions.
Moreover, aiming to design a reliable and plug & play device, we assessed NB-
IoT technology boundaries in SHM like environments. We perform more than 600
measurements using different passive and active RF attenuator to simulate
different coverage situations. Measurements are reported in Fig. 3. We grouped
the measurements in 3 areas defined by the RSSI:

Good Coverage: | $-95~{}dBm<$ | RSSI; |
---|---|---|---
Medium Coverage: | $-110~{}dBm<$ | RSSI |  $<-95~{}dBm$;
Bad Coverage: |  | RSSI |  $<-110~{}dBm$;

In the following tests, the RSSI spans from a maximum of $-75~{}dBm$ to a
minimum of $-120~{}dBm$, a typical received power in most real application
scenarios. In Fig. 3, the energy consumption is lower than $2~{}J$ (with one
outlier) for transmission with RSSI greater than $-96~{}dBm$, the mean energy
consumption is lower than $1.1~{}J$ with RSSI greater than $-96~{}dBm$. In
case of an RSSI lower than $-110~{}dBm$ the energy consumption is $4.071~{}J$
in average because of the bad coverage level, and ECL (the Enhanced Coverage
Level) is equal to 2. Hence the module performs multiple ($2^{i}$ with
$i=1...7$ based on ECL [30]) re-transmission to send the packet correctly. In
Bad connectivity conditions, each packet requires $3.8\times$ more energy than
in Good or $2.8\times$ more than in Medium conditions; this is a significant
difference which must be kept into consideration while estimating the battery
life.

Figure 3: Quectel BC95-G: average energy consumption in different coverage
levels

In comparison with LoRaWAN, or other LPWAN protocols, the NB-IoT requires more
energy per packet in general. However, due to its larger payload and its
national licensed connectivity, the energy per bit from the sensor to the
cloud is the best in class for the LPWAN category [11].

NB-IoT protocol has no restriction on band utilization because it uses
licensed LTE bands, so there is no limitation on the number of bytes sent in a
single connection to the cell. In SHM applications, NB-IoT is ideal because
gathered data can be buffered, and latency requirements are not so severe.
Several energy consumption measurements with different payloads size have been
done. Table I reports the mean energy consumed for each payload size and the
energy per bit (EPB). Notice that the EPB with $500~{}Bytes$ of data
transmitted is $235.1~{}\mu J$, thus 5.53 times more than the EPB with
$10.8~{}KBytes$ of data. An SHM node can generate big chunks of data (hundreds
of kBytes), that NB-IoT protocol can send with no restriction keeping a low
power profile and with low EPB when data is buffered and sent in a single
connection.

TABLE I: Mean energy consumption and EPB with different payload sizes, the $\Delta$ difference is compared to 500 Bytes of Payload Payload | Mean E. | $\Delta$ | EPB | EPB$\Delta$
---|---|---|---|---
[B] | Consumption [J] |  | [$\mu J$] |
10 | 0.7130 | $\times 1.32$ | 8912 | $\times 0.02$
200 | 0.8123 | $\times 1.15$ | 507.7 | $\times 0.46$
500 | 0.9405 | —— | 235.1 | ——
1300 | 1.0326 | $\times 0.91$ | 99.29 | $\times 2.36$
5400 | 2.1199 | $\times 0.44$ | 49.07 | $\times 4.79$
10800 | 3.6702 | $\times 0.25$ | 42.48 | $\times 5.53$

##  IV Sensor node design

###  IV-A Hardware design

After the exploration of the most suitable NB-IoT module and configuration,
the hardware design of the complete SHM node is discussed in this section.
Recent works in the literature have already investigated the sensor front-end
[15], and the wireless connectivity [11], providing fundamental hardware
specifications. In fact, as the major problems of commercial SHM systems are
the high sensor and installation cost, we combined MEMS sensor micro-vibration
requirements [15] with an LPWAN transceiver [11]. An SHM board with wireless
connectivity and battery supply drastically reduces the installation costs.

The main components of the designed prototype are:

  * •

STM32L476 microcontroller by ST Microelectronics: L4 Series microcontroller
with ARM Cortex-M4 and Floating Point Unit (FPU). It feature low power
consumption (39 uA/MHz) and a direct SD card interface;

  * •

LIS344ALH MEMS inertial sensor: high performance 3-axis $\pm$2 g/$\pm$6 g
ultra-compact linear accelerometer;

  * •

IIS2ICLX High-accuracy, high-resolution, low-power, 2-axis digital
inclinometer; it embeds a machine learning core for pattern recognition and
automatic trigger generations. Moreover, it could provide inclinometric
measures with an accuracy of 0.015 $mg/LSB$;

  * •

Quectel BC95-G LTE-CatNB1 NB-IoT module, characterized in Section III-C

Furthermore, the board is equipped with two temperature sensors for onboard
and external temperatures. A linear voltage regulator has been added to
provide a stable supply voltage starting from the battery output. A USB
compatible battery charger system with an integrated power switch for Li-
Ion/Li-Polymer is mounted in case rechargeable batteries or an energy
harvester are used. Two current measurement circuits are implemented in the
SHM sensor node to sample the current used by the circuit in real-time.
Measures are split into two power domains: the Quectel BC95-G and the rest of
the components. Analyzing the entire board’s power consumption permits the
detection of abnormal current drains and, more importantly, the prediction of
the residual battery energy. Indeed, as reported in the previous section, the
physical positioning and the wireless connectivity could vary the power
consumption of the communication link up to $3.8\times$.

The inclinometer IIS2CLX is used for two main tasks: gather clinometric data
and provide an interrupt in case of a peak detection. To reduce power
consumption between two measurement sessions, the board is halted in a ”sleep”
condition, but in case of an extraordinary event (e.g., accelerometric peak
caused by an earthquake), the IIS2CLX will wake-up to transmit emergency
measurements. The final 4-layer Printed Circuit Board (PCB) is $120~{}mm\times
60~{}mm$ and is shown in Figure 7.

###  IV-B Firmware design and data acquisition

The board Firmware is based on FreeRTOS to ensure code efficiency and
scalability. Three main tasks are defined:

  * •

BC95-G managment Task: it manages all the communications and network
operations between the NB-IoT module and the microcontroller. The STM32
controls initial settings and the first connection procedure, than it enables
both data transmission and reception;

  * •

SD card Task: in this task, the MCU manages the data logging on the SD card.
Measurements coming from different sensors and sessions are organized and
saved in text files, providing a ready to use format for end-users;

  * •

Sensor acquisition Task: in this task, all procedures that concern the sensors
data acquisition are considered; in particular, the MCU handles acquisition
and filtering of inertial sensor data.

In BC95-G management task, the microcontroller managed the NB-IoT
communication providing suitable AT commands through a UART serial port.
Furthermore, the state machine also includes a full set of recovery
capabilities in case of errors or unexpected situations. For example, when the
connection to the network operator stales or takes more than 1 minute, the
software automatically resets the module and tries a new connection. During
the transmission phase, the task manages the packed sending, supporting data
fragmentation and replication for large volume uplinks. Afterward, it manages
the PSM state and the RTC alarm for the next measuring sessions.

Concernign the sensor data acquisition, a filtering task is implemented. SHM
typically measures vibrations in the $020$Hz bandwidth, while principal modes
of the structure are in the range of $010$Hz. On extraordinary stressful
events for the structure (like an earthquake) the typical frequencies can
reach a higher range, typically up to 45Hz [15]. For this reason, the SHM node
provides 16 bit acceleration values on the three axes at 100Hz data rate. We
applied oversampling on the 12-bit ADC of the microcontroller to achieve
16-bit as ENOB (Effective Number Of Bits) necessary for achieving sensitivity
and precision requirements.

A further advantage associated with the oversampling technique concerns the
post processing, which eases the implementation of digital filters. In fact,
the digital FIR (Finite Impulse Response) filter is directly optimized for ARM
Cortex-M4 to decimate data while applying a low pass filtration. The
acquisition chain samples the data at $f_{OS}=25.6~{}kHz$ from the sensor, the
gathered data is then processed by a FIR filter with a ratio decimation of
256, generating filtered data with 16 bit resolution at the final data rate of
$f_{S}=100~{}Hz$. The filter is a FIR filter composed by 6 different stages
with a resulting decimation factor of 256 and a cutoff frequency of $50~{}Hz$.
The minimum attenuation in stop band is $60~{}dB$ and the ripple in pass band
is $0.1~{}dB$ peak-to-peak, maximum. The overall number of coefficients is
776, with a measured computation time of about $510~{}\mu s$, way lower than
$10~{}ms$ ($f_{S}=100~{}Hz$).

The SD card task manages the session data logging. While the sensor
acquisition task is running, the packets with filtered data are saved on an SD
card in a text file format. A double buffering technique is used between this
task and the sensor task.

##  V Energy Consumption model

###  V-A Battery Life Estimation

Different current measurements in multiple operating conditions have been
carried out to characterize the behavior of the SHM sensor. To estimate the
battery lifetime, we have analyzed the power profile of a complete
acquisition, composed by $5.2~{}kB$ of data. As shown in Fig. 4, the energy
consumption is partitioned into two major blocks: the energy used by data
acquisition ($E_{acq}$) and the one consumed for data transmission ($E_{TX}$).
These two blocks contribute as reported in Fig. 4:

  1. 1.

$E_{acq1s}$: one second of acquisition at 100 Hz;

  2. 2.

$E_{SD-WR}$: write data on the SD Card;

  3. 3.

$E_{c-1TX}$: connect and transmit the first packet;

  4. 4.

$E_{pkt-TX}$: transmit one packet (general);

  5. 5.

$E_{cdrx-disc}$: energy consumed by Connected_DRX and disconnect phases.

Figure 4: Different energy contributions aligned on a measured transmission

For each acquisition and transmission phase, the NB-IoT module sends 1300 B,
each packet consists of 650 filtered samples (16-bit depth per sample) saved
on the SD card and thereafter transmitted. In Fig. 4, the energy consumption
varies with the number of packets ($N_{pkt-TX}$) given by Eq. 1:

$\footnotesize N_{pkt-TX}=\frac{T_{acq}*f_{s}}{Payload}.$ (1)

where $T_{acq}$ is the time duration in seconds of the sampling time, and it’s
set by the user according to the application. The energy consumption of the
$E_{acq}$ is given by the Eq. (2), the $E_{TX}$ is given by Eq. (3) and the
total energy consumption for a session is given by Eq. (4):

$\footnotesize E_{acq}=E_{acq1s}(6.5*N_{pkt-TX})+N_{pkt-TX}*E_{SD-WR}$ (2)
$\footnotesize E_{TX}=E_{c-1TX}+E_{cdrx-disc}+(N_{pkt-TX}-1)E_{pkt-TX}$ (3)
$\footnotesize E_{tot}=E_{TX}+E_{acq}$ (4)

Table II reports the energy consumption of every sub-block, highlighted in
Fig. 4, calculated using a MatLab script.

TABLE II: Energy Consumption Contributions Contribution | Energy [mJ] | Contribution | Energy [mJ]
---|---|---|---
$E_{acq1s}$ | 52.596 | $E_{SD-WR}$ | 2.1816
$E_{c-1TX}$ | 659.72 | $E_{pkt-TX}$ | 450.83
$E_{cdrx-disc}$ | 616.97 | - | -

The current used by the node in sleep mode was measured using a precise power
source meter the Keithley 2470. When the STM32 enters in Stop Mode 2, all
sensors are consequently configured in low power mode, as well as the Quectel
BC95-G, which is in PSM. In this configuration, the current absorbed by the
board ($I_{sleep}$) is $34~{}\mu A$. The total energy consumed in one day
($E_{day}$) is given by the Eq. 5, where $V_{s}$ is the supply voltage of 3.3
V, $N_{session}$ is the number of acquisition and transmission performed per
day and $T_{sleep}$ is the number of seconds in sleep condition.

$\footnotesize E_{day}=E_{tot}*N_{session}+T_{sleep}*V_{s}*I_{sleep}$ (5)

The battery selected for our application is produced by SAFT, named
LS336000111https://www.saftbatteries.com/products-solutions/products/ls-lsh-
lsp. It is a Lithium - Thionyl chloride ($Li-SOCl_{2}$) primary cell, it’s
ideally suited for long-term applications with its 17Ah of capacity in a
D-size bobbin cell format. The total energy avaible by this cell is 226440 J.
The battery capacity is temperature dependent. The nominal value is verified
at $25^{\circ}$C, but it could be halved at $-40^{\circ}$C, thereby modifying
results in Fig. 5.

Figure 5: Battery life estimation varying the sensors acquisition time and the
number of sessions per day

Fig. 5 shows the battery lifetime (in years) as a function of the acquisition
time for each session, and the number of sessions per day. The acquisition
time plotted in Fig. 5 is from 4 minutes to 20 minutes but it can be lower or
higher depending on the application. We show four different curves according
to the different number of sessions per day: 1, 2, 4, 6. Notice that a
configuration with no more than 7 minutes acquisition time and one session per
day, is necessary for achieving ten years lifetime, with 84 kB transmitted
daily. On the other hand, very frequent acquisitions can discharge the battery
quite quickly. For example, 6 sessions with 20 minutes of measurement drain
the battery charge in 214 days, because 1.44 MB are transmitted every day.

###  V-B Energy Harvesting

In addtion to aggressive power management, to achieve long term monitoring an
energy harvester circuit is designed. Indeed, considering a typical scenario
for SHM, where sensors are deployed out in the open on building or bridges, a
solar harvesting is a valuable solution. In this scenario, a different battery
cell is needed because the LS336000 is not rechargeable: a Li-Ion alternative
is the VL34570_xlr by SAFT. This cell has the same size of the LS336000
(D-size bobbin cell) and the same nominal voltage, $3.7~{}V$, but the capacity
is lower: 5.4 Ah. If we focus on a long term application, we can estimate the
energy consumption per day with the aforementioned power model. According to
[20], to perform an analysis of structural damage detection using a time
series based algorithm, SHM operators need at least 6000 samples at 100 Hz. To
collect this amount of data, our SHM sensor needs 60 seconds of acquisition
(from each installed sensor) 6 times per day to check the integrity of the
structure. With this type of battery, using the equations (1), (2), (3), (4),
we obtain a duration of 3 years. All results are reported in Table III.

TABLE III: Specs and Data for SHM long term application using solar energy harvesting Parameter | Value | Parameter | Value | Parameter | Value
---|---|---|---|---|---
$N_{sessions}$ | 6 | $t_{acq}$ | 60 s | $t_{active}$ | 546 s
$t_{sleep}$ | 85854 s | $E_{TX}$ | 5.334 J | $E_{acq}$ | 3.441 J
$E_{day}$ | 61.998 J | $E_{cell}$ | 71928 J | $BatteryLife$ | 3.18 Years

The daily energy per day is equal to 61.998 J, with a corresponding average of
17.22 mWh.

A solar panel in good environment conditions has a power density from 15 to
100 $mW/cm^{2}$ [31]. Thus, a solar panel with the same surface as the board
(120 mm $\times$ 60 mm) is considered. Supposing a worst case scenario with 4
hours of complete solar light and 25% loss from recharge and storage circuitry
we obtain an average generated power of 3.24 Wh.

The energy intake from the solar panel each day is 2 orders of magnitude
bigger than the power consumed by the node.

###  V-C Model validation

To check the stability of the software and to validate the power consumption
model we extensively tested the sensor node in a real SHM environment. The
software is configured to acquire 60 seconds of data and transmit it through
NB-IoT protocol to the server, this session is repeated 6 time in a day, one
session every 4 hours.

Figure 6: Power consumption log of 1000 seconds

In Fig. 6 we show a log of 1000 seconds, this specific window includes one
complete session, in which the total energy consumption is 8.535 J. In the
following, a comparison with Equations (6), (7), (8), (9), (10) is proposed in
validation of our energy model.

$\footnotesize E_{TX}=E_{c-1TX}+E_{cdrx-disc}+(N_{pkt-TX}-1)E_{pkt-TX}$ (6)
$\footnotesize E_{acq}=E_{acq1s}(6*N_{pkt-TX})+N_{pkt-TX}*E_{SD-WR}$ (7)
$\footnotesize E_{tot}=E_{TX}+E_{acq}$ (8) $\footnotesize
E_{sleep-1000s}=V_{s}*(1000-T_{sleep})*I_{sleep}$ (9) $\footnotesize E_{tot-
model}=E_{tot}+E_{sleep_{1}000s}$ (10)

From Eq. (10), the energy consumed with this window, calculated using our
model, is $E_{tot-model}=8.613~{}J$ that is slightly more than the measured
one (0.9% more). Hence we can conclude that the model overestimates by a small
factor the energy consumption and the battery life assumptions are verified.
Furthermore the firmware never stopped in 3 days test doing a total of 18
session and transmitting a total of 324 KB to the server. Lastly, the energy
consumed in 1 day, measured with our set up, is 62.1 J and considering the
SAFT LS336000 cell, we obtain a battery life of 10.1 years.

##  VI In-field validation

Figure 7:  Overview of the test set up: in orange the direction of the oscillations imposed to the structure by the oscillating plate, in violet the axes orientation of our sensor node, and in green the axis orientation of the piezoelectric sensor.  Figure 8: FFT analisys of Dytran 3191A1 (bottom graph) and LIS344ALH (top graph) data; the values near tones are the precise frequency. TABLE IV: Tone Frequency Comparison LIS344ALH vs Dytran 3191A1 Modal Vibration | LIS344ALH | DYTRAN 3191A1 | $\Delta\%$
---|---|---|---
I Mode | 2.805 Hz | 2.807 Hz | -0.07%
II Mode | 8.383 Hz | 8.379 Hz | +0.05%
III Mode | 13.133 Hz | 13.125 Hz | +0.06%
IV Mode | 16.066 Hz | 16.052 Hz | +0.08%
Figure 9: FFT analisys of LIS344ALH in 3 test: undamaged structure
(NO_DAMAGE), slightly damaged structure (DAMAGE 1) and damaged structure
(DAMAGE 2) Figure 10: FFT analisys of IIS2CLX in 3 test: undamaged structure
(NO_DAMAGE), slightly damaged structure (DAMAGE 1) and damaged structure
(DAMAGE 2)

To verify the correctness of acquisitions and to check the measurement
accuracy of our SHM sensor node, we assessed the system in a civil engineering
test laboratory test structure and environment (Fig. 7). The steel test
structure is mounted on an oscillating plate that can apply programmed
vibration to simulate typical buildings behaviors. The sensor used as
reference is a high precision piezoelectric transducer, the Dytran
3191A1222https://www.dytran.com/Model-3191A-Industrial-Accelerometer-P1625/.
It is specifically designed for SHM applications, and used in high-end
instrumentation, as reported in several studies starting from 2010 [32].

A vibration has been imposed on the oscillating plate to see the structure’s
first four vibration modes on 2.8 Hz, 8.4 Hz, 13.1 Hz, and 16.2 Hz. The data
coming from the Dytran 3191A1 and the LIS344ALH has been recorded in multiple
measurement sessions of 180 s. Fig. 8 shows the two resulting FFTs. Notice
that the data from LIS344ALH permit to detect the expected vibration modes, as
well as the Dytran 3191A1. Moreover there are no other unexpected frequency
components. The exact frequency values calculated from the two sensors are
reported in Table IV and shown in Fig. 8. The difference is maximum 0.08%;
confirming that Low-cost analog MEMS accelerometers can be used for the
assessment of buildings in SHM.

After the comparison tests, a damage detection simulation has been performed.
The steel structure used in the tests (Fig. 7) can be modified to simulate
specific damage, and 2 different situations have been simulated, replacing
columns of the steel structure with thinner ones. Fig.9 shows the FFT of 3
measurements; the first modal tone is detailed to highlight the frequency
shift. Notice that the first modal tone with undamaged structure (Fig.9 \-
NO_DAMAGE) is at $2.807~{}Hz$, but when a column of the structure is replaced
with a thinner one simulating light damage (Fig.9 \- DAMAGE 1), the frequency
of the first modal tone shifts of $0.089~{}Hz$ ($2.718~{}Hz$). In the last
test 4 columns of the structure are substituted with thinner ones to simulate
moderate damage (Fig.9 \- DAMAGE 2), and this can be clearly measured in the
resulting frequency of the first modal tone $2.284~{}Hz$ ($0.523~{}Hz$ less
than “undamaged” structure). The same analysis has been executed with data
gathered from the IIS2CLX inclinometer sensor. Results show similar modal
frequencies, a difference less than 0.3%. In Fig.10 shows the result of the
test, with the frequency shift in the 3 cases. IIS2CLX is a digital sensor
with built-in programmable features like a machine learning core processor and
a programmable low pass filter. It is more expensive, with 14.31 € at 1kunit,
it is almost $7\times$ the cost of the LIS344ALH. If the specific SHM
application requires machine learning core features, the IIS2CLX could be a
viable solution. If the main focus is on cost reduction or the application
needs a scale-up to hundreds or thousands of measurement points, the LIS344ALH
is the most suitable solution.

##  VII Conclusions

Nowadays, the ability to monitor the integrity of a wide variety of buildings
with low-cost and real-time measurements is essential from both an economic
and a life-safety standpoint. In this work, we proposed a completely
untethered wireless sensor node specifically designed to support long-term
modal analysis with long-range connectivity at low power consumption.

We analyzed and tuned the main parameters of the NB-IoT communication
protocol, chosen because it provides wireless connectivity to the 4G (and
future 5G) global infrastructure network at low power consumption, suitable
for continuous monitoring. We provided a comparison of the most recent NB-IoT
modules on the market and selected the most appropriate for a continuous SHM
scenario. We presented the design, both hardware and software, of a SHM node
that can operate unattended for more 10 years, or even indefinitely with the
support of a small size solar panel of 72 $cm^{2}$.

We validated the measurement accuracy of the system by comparing our low-cost
devices with a state-of-the-art piezoelectric transducer used in high-end
commercial instrumentation for temporary cabled installations.

Results show a difference lower than 0.08% in the accuracy of estimation of
the modal vibration frequencies, with a cost reduction of around $10\times$.
Moreover, our solution enables ease of deployment due to the total absence of
cables and long operation lifetime. Lastly, we present three structural damage
laboratory tests that confirm the usability of our solution in SHM
applications.

## References

  * [1] B. Romano, F. Zullo, L. Fiorini, A. Marucci, and S. Ciabò, “Land transformation of italy due to half a century of urbanization,” _Land use policy_ , vol. 67, pp. 387–400, 2017.
  * [2] G. M. Calvi, M. Moratti, G. J. O’Reilly, N. Scattarreggia, R. Monteiro, D. Malomo, P. M. Calvi, and R. Pinho, “Once upon a time in italy: The tale of the morandi bridge,” _Structural Engineering International_ , vol. 29, no. 2, pp. 198–217, 2019.
  * [3] C. A. Tokognon, B. Gao, G. Y. Tian, and Y. Yan, “Structural health monitoring framework based on internet of things: A survey,” _IEEE Internet of Things Journal_ , vol. 4, no. 3, pp. 619–635, 2017.
  * [4] F. Zonzini, A. Girolami, L. De Marchi, A. Marzani, and D. Brunelli, “Cluster-based vibration analysis of structures with gsp,” _IEEE Transactions on Industrial Electronics_ , vol. 68, no. 4, pp. 3465–3474, 2021\.
  * [5] M. R. Dhage and S. Vemuru, “Structural health monitoring of railway tracks using wsn,” in _2017 International Conference on Computing, Communication, Control and Automation (ICCUBEA)_. IEEE, 2017, pp. 1–5.
  * [6] S. Valenti, M. Conti, P. Pierleoni, L. Zappelli, A. Belli, F. Gara, S. Carbonari, and M. Regni, “A low cost wireless sensor node for building monitoring,” in _2018 IEEE Workshop on Environmental, Energy, and Structural Monitoring Systems (EESMS)_. IEEE, 2018, pp. 1–6.
  * [7] D. Catenazzo, B. O’Flynn, and M. Walsh, “On the use of wireless sensor networks in preventative maintenance for industry 4.0,” in _2018 12th International Conference on Sensing Technology (ICST)_. IEEE, 2018, pp. 256–262.
  * [8] A. Burrello, A. Marchioni, D. Brunelli, S. Benatti, M. Mangia, and L. Benini, “Embedded streaming principal components analysis for network load reduction in structural health monitoring,” _IEEE Internet of Things Journal_ , vol. 8, no. 6, pp. 4433–4447, 2021.
  * [9] J. Haxhibeqiri, E. De Poorter, I. Moerman, and J. Hoebeke, “A survey of lorawan for iot: From technology to application,” _Sensors_ , vol. 18, no. 11, p. 3995, 2018.
  * [10] M. Ballerini, T. Polonelli, D. Brunelli, M. Magno, and L. Benini, “Experimental evaluation on NB-IoT and lorawan for industrial and iot applications,” in _2019 IEEE 17th International Conference on Industrial Informatics (INDIN)_ , vol. 1. IEEE, 2019, pp. 1729–1732.
  * [11] M. Ballerini, T. Polonelli, D. Brunelli, M. Magno, and L. Benini, “NB-IoT vs. lorawan: An experimental evaluation for industrial applications,” _IEEE Transactions on Industrial Informatics_ , 2020.
  * [12] A. Sabato, C. Niezrecki, and G. Fortino, “Wireless mems-based accelerometer sensor boards for structural vibration monitoring: a review,” _IEEE Sensors Journal_ , vol. 17, no. 2, pp. 226–235, 2016.
  * [13] H. Wang, P. Xiang, and L. Jiang, “Strain transfer theory of industrialized optical fiber-based sensors in civil engineering: A review on measurement accuracy, design and calibration,” _Sensors and Actuators A: Physical_ , vol. 285, pp. 414–426, 2019.
  * [14] E. G. Popkova, “Preconditions of formation and development of industry 4.0 in the conditions of knowledge economy,” in _Industry 4.0: Industrial Revolution of the 21st Century_. Springer, 2019, pp. 65–72.
  * [15] A. Girolami, D. Brunelli, and L. Benini, “Low-cost and distributed health monitoring system for critical buildings,” in _2017 IEEE Workshop on Environmental, Energy, and Structural Monitoring Systems (EESMS)_ , 2017, pp. 1–6.
  * [16] F. Naeim, “Dynamics of structures—theory and applications to earthquake engineering,” _Earthquake Spectra_ , vol. 23, no. 2, pp. 491–492, 2007.
  * [17] A. Burrello, D. Brunelli, M. Malavisi, and L. Benini, “Enhancing structural health monitoring with vehicle identification and tracking,” in _2020 IEEE International Instrumentation and Measurement Technology Conference (I2MTC)_. IEEE, 2020, pp. 1–6.
  * [18] C. R. Farrar, H. Sohn, and G. H. Park, “A statistical pattern recognition paradigm for structural health monitoring,” 01 2004.
  * [19] Y. Liao, A. Kiremidjian, R. Rajagopal, and C.-H. Loh, “Structural damage detection and localization with unknown post-damage feature distribution using sequential change-point detection method,” _Journal of Aerospace Engineering Vol. 32, Issue 2_ , 11 2018.
  * [20] K. Krishnan Nair and A. S. Kiremidjian, “Time Series Based Structural Damage Detection Algorithm Using Gaussian Mixtures Modeling,” _Journal of Dynamic Systems, Measurement, and Control_ , vol. 129, no. 3, pp. 285–293, 08 2006\.
  * [21] F. Zonzini, M. M. Malatesta, D. Bogomolov, N. Testoni, A. Marzani, and L. De Marchi, “Vibration-based shm with upscalable and low-cost sensor networks,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 69, no. 10, pp. 7990–7998, 2020.
  * [22] T. Polonelli, D. Brunelli, M. Guermandi, and L. Benini, “An accurate low-cost crackmeter with lorawan communication and energy harvesting capability,” in _2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA)_ , vol. 1, 2018, pp. 671–676.
  * [23] H. Wang, L. Jiang, and P. Xiang, “Improving the durability of the optical fiber sensor based on strain transfer analysis,” _Optical Fiber Technology_ , vol. 42, pp. 97–104, 2018.
  * [24] P. Ragam and N. Devidas Sahebraoji, “Application of mems-based accelerometer wireless sensor systems for monitoring of blast-induced ground vibration and structural health: a review,” _IET Wireless Sensor Systems_ , vol. 9, no. 3, pp. 103–109, 2019.
  * [25] K. Mekki, E. Bajic, F. Chaxel, and F. Meyer, “A comparative study of lpwan technologies for large-scale iot deployment,” _ICT Express_ , vol. 5, pp. 1–7, 03 2019.
  * [26] L. Alonso, J. Barbarán, J. Chen, M. Díaz, L. Llopis, and B. Rubio, “Middleware and communication technologies for structural health monitoring of critical infrastructures: A survey,” _Computer Standards & Interfaces_, vol. 56, 09 2018.
  * [27] C. Arcadius Tokognon, B. Gao, G. Y. Tian, and Y. Yan, “Structural health monitoring framework based on internet of things: A survey,” _IEEE Internet of Things Journal_ , vol. 4, no. 3, pp. 619–635, 2017.
  * [28] Y. . E. Wang, X. Lin, A. Adhikary, A. Grovlen, Y. Sui, Y. Blankenship, J. Bergman, and H. S. Razaghi, “A primer on 3gpp narrowband internet of things,” _IEEE Communications Magazine_ , vol. 55, no. 3, pp. 117–123, 2017.
  * [29] A. Adhikary, X. Lin, and Y. . E. Wang, “Performance evaluation of nb-iot coverage,” in _2016 IEEE 84th Vehicular Technology Conference (VTC-Fall)_ , 2016, pp. 1–5.
  * [30] B. Martinez, F. Adelantado, A. Bartoli, and X. Vilajosana, “Exploring the performance boundaries of nb-iot,” _IEEE Internet of Things Journal_ , vol. 6, no. 3, pp. 5702–5712, 2019.
  * [31] N. Garg and R. Garg, “Energy harvesting in iot devices: A survey,” in _2017 International Conference on Intelligent Sustainable Systems (ICISS)_ , 2017, pp. 127–131.
  * [32] F. G. Baptista and J. V. Filho, “Optimal frequency range selection for pzt transducers in impedance-based shm systems,” _IEEE Sensors Journal_ , vol. 10, no. 8, pp. 1297–1303, 2010.

|  Flavio Di Nuzzo received the bachelor’s degree in electronic engineering
and the master’s degree in electronic engineering from the University of
Bologna in 2016 and 2019, respectively. He has done a year as a research
fellow at the University of Bologna studying the Narrowband-IoT technology
applied to Civil Structural Health Monitoring. He currently works in Ducati
Motor Holding as an Electronic Engineer in the Vehicle Testing department.
---|---
|  Davide Brunelli (Senior Member, IEEE) received the M.S. (cum laude) and
Ph.D. degrees in electrical engineering from the University of Bologna,
Bologna, Italy, in 2002 and 2007, respectively. He is currently an associate
professor at the University of Trento, Italy. His research interests include
IoT and distributed lightweight unmanned aerial vehicles, the development of
new techniques of energy scavenging for low-power embedded systems and energy-
neutral wearable devices, Drones, UAVs and Machine Learning. He was leading
industrial cooperation activities with Telecom Italia, ENI, and
STMicroelectronics. He has published more than 200 papers in international
journals or proceedings of international conferences. He is an ACM member.
---|---
|  Tommaso Polonelli (Student Member, IEEE) received a Ph.D. and a master’s
degree in electronics and telecommunications engineering from the University
of Bologna, Bologna, Italy, in 2020 and 2017, respectively. He is currently a
postdoctoral fellow in the center of Project Based Learning, ETH Zurich,
Switzerland. His research interests lie in the area of wireless sensor
networks, wearable devices, structural health monitoring, autonomous unmanned
vehicles, power management techniques, and the design of batteries-operating
devices and embedded video surveillance. He has collaborated with several
universities and research centers, such as the University College Cork, Cork,
Ireland, Imperial College London, London, U.K., and the ETH, Zurich,
Switzerland. He has authored over 20 papers in international journals and
conferences.
---|---
|  Luca Benini (Fellow, IEEE) holds the chair of digital circuits and systems
at ETHZ and is Full Professor at the Università di Bologna. He received a PhD
from Stanford University. He served as chief architect in STMicroelectronics
France. Dr. Benini’s research interests are in energy-efficient parallel
computing systems, smart sensing micro-systems and machine learning hardware.
He has published more than 1000 peer-reviewed papers and five books. He is a
fellow of the ACM and a member of the Academia Europaea.
---|---


# Feel the Tension: Manipulation of Deformable Linear Objects in Environments
with Fixtures using Force Information

Finn Süberkrüb1, Rita Laezza2, and Yiannis Karayiannidis2,3 1TC Plattling,
Deggendorf Institute of Technology, Germany finn.sueberkrueb@th-
deg.de2Department of Electrical Engineering, Chalmers University of
Technology, Sweden {laezza<EMAIL_ADDRESS>of Automatic
Control, Lund University, Sweden<EMAIL_ADDRESS>work was partially
supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP)
funded by the Knut and Alice Wallenberg Foundation and the ELLIIT Strategic
Area for ICT research, supported by the Swedish Government.

###### Abstract

Humans are able to manipulate Deformable Linear Objects (DLOs) such as cables
and wires, with little or no visual information, relying mostly on force
sensing. In this work, we propose a reduced DLO model which enables such blind
manipulation by keeping the object under tension. Further, an online model
estimation procedure is also proposed. A set of elementary sliding and
clipping manipulation primitives are defined based on our model. The
combination of these primitives allows for more complex motions such as
winding of a DLO. The model estimation and manipulation primitives are tested
individually but also together in a real-world cable harness production task,
using a dual-arm YuMi, thus demonstrating that force-based perception can be
sufficient even for such a complex scenario.

##  I Introduction

Force Torque (FT) sensing has been extensively used in robotic contact and
interaction tasks both for feedback control schemes and for estimating
important properties of the objects being manipulated [1, 2, 3, 4]. However,
most research has been limited to rigid objects, whose state can be summarized
by their pose. Deformable objects constitute a still under-explored class of
objects that often require more complex state representations, since such
objects may also change in shape. Research on FT-based deformable object
manipulation has been limited due to challenges in sensing, since FT
measurements require the object to provide sufficient resistance to an applied
force.

This work focuses on Deformable Linear Objects (DLOs), which are characterized
by being much larger along one dimension than the other two dimensions [5].
Objects like cables, wires and hoses are present in numerous industrial
applications, and pose an interesting robotics manipulation challenge. When a
DLO with low compression strength such as a rope is manipulated, it does not
offer resistance to deformation in any direction, unless it is under tension.
Therefore, in the absence of visual information, keeping the DLO taut provides
the most amount of information about its state. This property is exploited to
formulate our simplified graph representation and nine manipulation
primitives.

Fig. 1 shows an instance of the proposed primitives, namely clipping a DLO
into a fixture. To be successful, the DLO must be kept under tension so that
it does not bend as it is pushed through the narrow slot. This is an important
task for example in wire harness production, where individual cables must be
routed through fixtures. Note that due to the challenges of DLO manipulation,
manual assembly is still the most common wire harness manufacturing method
[6].

Although visual and tactile sensing are also important for DLO manipulation,
in this work we focus on FT information alone. Sanchez et al. [7] proposed a
blind deformable object manipulation approach, but with a more complex model
and for a problem without environmental contacts. Here we focus on tasks where
the environment provides contacts which constrain the Degrees of Freedom
(DOFs) of the DLO. To validate our work, we first evaluate the model
estimation in simulation. Secondly, the manipulation primitives are evaluated
individually in real-world experiments. Finally, a wire harness problem is
solved using a sequence of manipulation primitives.

Figure 1: Illustration of DLO model with five feature points $v_{1}$ to
$v_{5}$, four edges $e_{1}$ to $e_{4}$, with forces $f_{2}$, $f_{3}$ and
distances $d_{2}$, $d_{3}$ marked on edges incident on $v_{3}$. The feature
points $v_{2}$ and $v_{3}$ are constrained by the YuMi SmartGrippers positions
$\bm{p}_{g,right}$ and $\bm{p}_{g,left}$.

##  II Related Work

There has been a growing interest in robotic manipulation of deformable
objects, with comprehensive reviews being published on the subject by Sanchez
et al. [5], Arriola-Rios et al. [8], and Yin et al. [9]. When addressing
deformable object manipulation problems, there are many sub-problems which
must be solved. Arriola-Rios et al. divide these into: (i) modeling of shape,
(ii) modeling of deformation dynamics, (iii) learning and estimation of model
parameters, (iv) perception and prediction, as well as (v) planning and
control. Our work addresses elements of all sub-problems.

There are three model types commonly used to represent deformable objects,
namely mass-spring systems [10], position-based dynamics, or finite element
methods [9, 11]. If the object’s parameters are known, such approaches can
produce realistic models and are already widely used in computer graphics
[12]. An alternative to physics-based models are purely geometric approaches
in which the object is reduced to elementary shapes [13]. Estimation of model
parameters based on sensor data is often necessary.

Visual sensing is the most commonly used modality for perception of deformable
objects, since it provides information about overall shape [14, 15]. However,
vision may fail in case of occlusions by the robot or by the DLO itself and
cannot provide information regarding relatively small deformation and tension.
There have been a few works exploring DLO manipulation with environmental
contacts based on vision alone [16, 17]. Zhu et al. [18] proposed a motion
planning framework, using two motion primitives designed for a robot to shape
a DLO through circular contacts. While they use a vision-based contact
detector, we attempt to estimate contacts through FT information and make no
assumptions on the shape of the contact surface.

Conversely, FT sensing enables the possibility to estimate material properties
of an object, such as Young’s modulus, but cannot easily be used to identify
its global shape. Selective probing can be used in order to combine position
information from joint encoders with force information. Estimation of material
properties is particularly useful to adapt controller parameters. Sanchez et
al. [19] presented an approach to estimate the deformation of a foam cube.
However, purely force-based position or shape estimators have not yet been
used for DLO manipulation problems. FT measurements have also been combined
with visual information to estimate material parameters [20, 21].

Regardless of the sensing modality and the DLO model, there are many control
strategies in the literature which involve estimating a Jacobian of the object
[22]. For example, Berenson [23] introduced the concept of diminishing
rigidity to define a Jacobian where points on the DLO closer to the grasped
point are assumed to act more rigidly. Other approaches have been proposed
where a Jacobian is learned using neural networks [24] or estimated through
weighted least-squares [25]. In this work, we instead estimate a reduced graph
model. Note that planning of a manipulation sequence is beyond the scope of
this paper and other works address that problem [26, 27].

##  III Problem Statement

Figure 2: Robotic setup of a YuMi robot equipped with specialized parallel
grippers and one fixture attached to a pegboard. Relevant robot frames are
indicated as: $\\{h_{FT}\\}$, the FT sensing frame, $\\{h_{l}\\}$ the frame
associated to the tool tip at the left gripper, and $\\{h_{r}\\}$ at the right
gripper. A fixture frame is denoted as $\\{h_{fix}\\}$ and the world frame as
$\\{h_{world}\\}$. Red arrows indicate $x$-axes, green arrows $y$-axes and
blue arrows $z$-axes.

We address force-based manipulation of DLOs without complementary visual
information, therefore the robot is assumed to be equipped with at least one
wrist FT sensor capable of measuring the tension of the DLO. Furthermore, the
problem is formulated assuming that once a DLO is gripped, its local pose is
known. Consequently, parallel grippers must prevent involuntary slipping of
the DLO along the $x$ (red) and $z$ (blue) axes, while also allowing a
controlled sliding motion along $x$ when desired. This can be achieved
mechanically through specialized fingertips, shown in Fig. 2, which guide the
DLO to a known location on the fingers and stop any further displacement along
$z$.

We assume that the workspace compensates for the blindness of the robot by
providing sufficient contact points in the form of fixtures or pivoting
points, which restrict the DOFs of the DLO, thus enabling tensioning. This is
imperative, since when a DLO is fully stretched between two points, the shape
between them can be assumed to be a straight line. Note that gravity itself
can be used to restrict the DOFs of the DLO. Once an end-effector equipped
with a FT sensor is gripping the object by holding it limp in the air, the
shape of the DLO segment below the gripper can be inferred to be vertical,
assuming quasi-static motion.

Based on these assumptions, we derive a reduced model of the DLO, presented in
Section IV, which can be estimated online, as described in Section V. This
estimated model is then sufficient to define nine elementary manipulation
primitives, introduced in Section VI.

##  IV Model Representation

We propose a reduced model $\\{\mathcal{G},\rho\\}$ to describe the state of a
uniformly weighted DLO based on force-torque information. The state of the DLO
is modeled as a path graph $\mathcal{G}=(V,E)$ where: $V$ is a set of feature
points on the DLO which can be listed in the order
$v_{1},v_{2},\ldots,v_{|V|}$; and $E$ is a set of edges connecting adjacent
feature points, $(v_{i},v_{i+1})$ for $i=1,2,\ldots,|V|-1$. This model
provides the minimal information necessary to enable a set of elementary
manipulation primitives. The model also requires knowledge of the DLO’s weight
per unit length, $\rho$. If the length and/or weight of the DLO are not known
a priori, they can be estimated using a sequence of the aforementioned
manipulation primitives. Note that the cardinality of $V$, denoted by $|V|$,
is not constant and will change as the manipulator interacts with the DLO and
indirectly with the environment.

###  IV-A Feature Points, $V$

The $i$-th feature point describing the state of the DLO is defined as a tuple
$v_{i}=\left(\mathbf{p}_{i},\mathbf{Q}_{i},m_{i}\right)$, where
$\mathbf{p}_{i}\in\mathbb{R}^{3}$ is the position,
$\mathbf{Q}_{i}\in\mathbb{R}^{4}$ is the orientation and $m_{i}\in\mathbb{R}$
is the mass.

If the feature point is the result of a perceived tension, it may be due to an
unknown contact with the environment or another gripper. While in the first
case $\mathbf{p}_{i}$ must be estimated, in the latter it can be obtained
directly through forward kinematics. Furthermore, if an additional feature
point $v_{k}$ is introduced to define a goal, it can be determined by linear
interpolation between two neighboring points, $v_{k-1}$ and $v_{k+1}$. The
index $k$ will depend on the location of the added point relative to the
existing points in the model. The position is then calculated by interpolation
as:

$\mathbf{p}_{k}=\mathbf{p}_{k-1}+\delta(\mathbf{p}_{k+1}-\mathbf{p}_{k-1})\text{,
with }\delta\in[0,1]$ (1)

Since the orientation cannot be accurately estimated, it is only inferred from
the pose of the gripper. For estimated points, the orientation is set so that
the $x$-axis is aligned with the length of the DLO, without twist i.e. the
orientation of two adjacent points is the same. Similarly to the position,
when a point $v_{k}$ is added to describe a goal, a linear interpolation
between two quaternion orientations is computed by Slerp (spherical linear
interpolation):

$\mathbf{Q}_{k}=\mathbf{Q}_{k-1}(\mathbf{Q}_{k-1}^{-1}\mathbf{Q}_{k+1})^{\delta}\text{,
with }\delta\in[0,1]$ (2)

Finally, the mass of the DLO is equally distributed between the feature
points, leading to $m_{i}$. Depending on the relative location of a feature
point with respect to other points, the mass is computed as:

$m_{i}=\begin{cases}\frac{{d_{i}}}{2}\rho&\text{$i$ is an end-point of the
DLO}\\\ \frac{d_{i-1}+d_{i}}{2}\rho&\text{$i$ is between feature
points}\end{cases}$ (3)

where ${d_{i}}$ is a property of the edges to be described next.

###  IV-B Edges, $E$

The $i$-th edge, connecting two adjacent feature points $(v_{i},v_{i+1})$ is
defined as a tuple $e_{i}=\left({d_{i}},s_{i},f_{i},k_{i}\right)$, where
${d_{i}}\in\mathbb{R}$ describes the length, $s_{i}\in\mathbb{R}$ is the sag,
$f_{i}\in\mathbb{R}$ the tension and $k_{i}\in\mathbb{R}$ the spring constant
of the DLO between feature point $v_{i}$ and an adjacent feature point
$v_{i+1}$.

The sag $s_{i}$ between two feature points, illustrated in Fig. 3, is a
measure used to determine whether the tensioning of the DLO is sufficient for
the assumption of a linear connection. If two feature points are not on the
same horizontal line and the sag-to-span ratio ($s/a\ll 1$) is small enough,
no part of the DLO is below the lower feature point. In such cases, the sag
can be approximated by equation (4), defined in [28]:

$s_{i}\approx+\frac{1}{4}\sqrt{6\frac{(a^{2}+h^{2})^{2}}{a^{2}}\left[\frac{d_{i}}{\sqrt{(a^{2}+h^{2})}}-1\right]}$
(4)

where $a$ and $h$, shown in Fig. 4, can be computed from the positions of the
edge endpoints, namely $\mathbf{p}_{i}$ and $\mathbf{p}_{i+1}$. The expression
is not defined for $a=0$, which occurs when one of the feature points is the
end-point of the DLO, constrained by gravity alone. In such cases, the sag is
set to $0$.

The weight compensated DLO tension ${f}_{i}$ between an end-effector at
feature point $v_{i}$ equipped with a FT sensor and a second feature point
$v_{i+1}$ can be calculated from the force measurements $\mathbf{f}_{sens}$
and the preceding edge’s tension ${f}_{i-1}$ according to:

$\hat{\mathbf{p}}_{i}{f}_{i}=\mathbf{f}_{sens}-\mathbf{g}m_{i}+\hat{\mathbf{p}}_{i-1}{f}_{i-1}$
(5)

where $\mathbf{g}=[0,0,g]^{T}$ denotes the gravitational acceleration, and the
unit vectors $\hat{\mathbf{p}}_{i-1}$ and $\hat{\mathbf{p}}_{i}$ denote the
normalized force directions:

$\hat{\mathbf{p}}_{i}=\frac{(\mathbf{p}_{i+1}-\mathbf{p}_{i})}{||\mathbf{p}_{i+1}-\mathbf{p}_{i}||}$
(6)

With only one FT sensor, the force ${f}_{i-1}$ can only be taken into account
under the assumption that it results from the mass of the rope (a free hanging
feature point). In order to sense a second tension e.g. if the DLO is
tensioned on both sides of the gripper, another FT sensor would be necessary.
For the two terminating feature points only one edge exists and the last part
of equation (5) is omitted.

Figure 3: The force measured by an FT sensor in rigid contact with feature
point $v_{i}$ is shown as $\mathbf{f}_{sens}$. The position $\mathbf{p}_{i-1}$
and mass $m_{i-1}$ can be determined using knowledge of $\mathbf{p}_{i}$ and
the measured force $\mathbf{f}_{sens}$. Force ${f}_{i}$ is a s result of the
tension between $\mathbf{p}_{i}$ and $\mathbf{p}_{i+1}$, while the force
${f}_{i-1}$ is caused by the mass $m_{i-1}$. Note that the model assumes
${f}_{i}$ is directed along the straight line. Both forces act on the feature
point $v_{i}$. Furthermore, the parameters $a$ and $h$ for a catenary curve of
a uniformly weighted flexible DLO under tension showing the maximal sag
$s_{i}$.

##  V Model Estimation

The model estimation procedure consists of three main parts: (i) estimating
the positions of feature points resulting from contact with the environment,
(ii) estimating the spring constants of individual edges, (iii) merging
estimated features with the known model and interpolate non-estimable feature
points between known contact points.

###  V-A Feature Point Estimation

Let us consider the case illustrated in Fig. 4, where one gripper is holding
the DLO at a position $\mathbf{p}_{g}$, while the other end is fixed to a
single contact, at an unknown position $\mathbf{p}_{c}$. Both $\mathbf{p}_{g}$
and $\mathbf{p}_{c}$ are lying along the line with direction of the tension
$\mathbf{f}_{g}$. This leads to the following observation model, for a given
measurement $j$:

$\mathbf{G}_{j}{\mathbf{p}_{g_{j}}}=\mathbf{G}_{j}{\mathbf{p}_{c}}$ (7)

where $\mathbf{G}_{j}$ is the unnormalized projection matrix, which projects
the position vectors along the orthogonal complement of the force
${\mathbf{f}_{g}}_{j}$, defined as:

$\mathbf{G}_{j}={\mathbf{f}^{T}_{g_{j}}}{\mathbf{f}_{g_{j}}}\mathbf{I}_{3}-{\mathbf{f}_{g_{j}}}{\mathbf{f}^{T}_{g_{j}}}$
(8)

where $\mathbf{I}_{n}$ denotes an identity matrix of dimensions $n\times n$.
The contact point position can then be estimated as the intersection of
several measurements obtained by changing the gripper position. Equation (7)
can be seen as standard observation model [29] of the form:

$\mathbf{y}=\mathbf{A}\bm{\phi}+\mathbf{e}$ (9)

with measurement $\mathbf{y}:=\mathbf{G}\mathbf{p}_{g}$, linear mapping
$\mathbf{A}:=\mathbf{G}$, parameter $\bm{\phi}:=\mathbf{p}_{c}$ and additive
noise $\mathbf{e}$. The same observation model from equation (7) can be used
in a Kalman filter to estimate the contact point $\tilde{\mathbf{p}}_{c}$. The
Kalman update step with a new measurement is given by:

$\tilde{\mathbf{p}}_{c}:={\tilde{\mathbf{p}}_{c}}^{-}+\mathbf{K}(\mathbf{G}_{j}\mathbf{p}_{g_{j}}-\mathbf{G}_{j}{\tilde{\mathbf{p}}_{c}}^{-})$
(10)

where $\mathbf{K}$ is the Kalman gain and ${\tilde{\mathbf{p}}_{c}}^{-}$, is
the parameter estimated forward in time. The Kalman filter adds the
possibility to monitor uncertainties. Certain motion primitives can be
executed in order to obtain measurements that improve the estimates, when
uncertainty is high.

Figure 4: A DLO which is tensioned between the unknown point
${\mathbf{p}}_{c}$ and the known gripper position ${\mathbf{p}}_{g}$ with the
weight compensated force ${\mathbf{f}}_{g}$.

###  V-B Edge Elasticity Estimation

An important parameter for force control is the elasticity of the edge. More
specifically, it can be used to appropriately tune the tension force
controller presented in Section VI. Based on Hooke’s law, we define:

$\text{f}_{g}=k\Delta l$ (11)

where $\Delta l$ is the displacement of the DLO from its relaxed state and
${\text{f}}_{g}$ the tension which is non-negative. To avoid negative values
due to noise, the absolute value of the force is used for the estimation.
Further, to avoid dependency on the resting position corresponding to zero
tension, we obtain estimates of $k$ by using the backward Euler approximation
of the time derivative of equation (11), that is:

$\text{f}_{g}(t)-\text{f}_{g}(t-1)=k\big{[}\Delta l(t)-\Delta l(t-1)\big{]}$
(12)

Compared to the standard observation model from equation (9),
$\text{f}_{g}(t)-\text{f}_{g}(t-1)$ corresponds to the measurement, $k$ to the
parameter and $\Delta l(t)-\Delta l(t-1)$ to a regressor.

###  V-C Model Update by Estimates

While a DLO is being manipulated, constraints are dynamically introduced by
end-effectors, fixtures, contact points or known distances, and it becomes
necessary to update the model accordingly. This is done by mapping the
constraints enforced on a feature point to the rest of the model. To this end,
we define the function $F:\mathcal{C}\times\mathcal{P}\rightarrow\mathcal{P}$,
which given current feature point positions in the model
$\mathbf{p}_{i}\in\mathcal{P}$, maps the constrains $\mathcal{C}$ to a new set
of positions $\mathcal{P}$.

Taking the example from Fig. 1, feature points $v_{2}$ and $v_{4}$ are fully
constrained by a rigid connection to the grippers, while feature points
$v_{1}$ and $v_{5}$ are constrained by the edge length $d_{1}$ and $d_{4}$ and
gravity. Furthermore, the feature point $v_{3}$ was added to the model in
order to define a clipping goal. At this stage the position $\textbf{p}_{3}$
can only be defined by the distances $d_{2}$ or $d_{3}$ and linearly
interpolated between $v_{2}$ and $v_{4}$. In this state, the mapping of the
measurements to the model parameters is as follows:

$\begin{bmatrix}\mathbf{p}_{2}\\\ \mathbf{p}_{3}\\\
\mathbf{p}_{4}\end{bmatrix}=\left(\begin{bmatrix}1&0\\\
\frac{d_{3}}{d_{2}+d_{3}}&\frac{d_{2}}{d_{2}+d_{3}}\\\
0&1\end{bmatrix}\otimes\mathbf{I}_{3}\right)\begin{bmatrix}\mathbf{p}_{{g,right}}\\\
\mathbf{p}_{{g,left}}\end{bmatrix}$ (13)

where $\otimes$ denotes the Kronecker product.

As soon as $v_{3}$ makes contact with the fixture, it is constrained by the
estimated contact point. First, the constraint is mapped to the neighbouring
points $v_{2}$ and $v_{4}$. Then, based on this update, the other points in
the model $v_{1}$ and $v_{5}$ must also be updated. Since the FT sensor
provides only local information about the DLO, knowledge of the previous
configuration must be taken into account to determine the global state of the
object.

##  VI Manipulation Primitives and Control

In this section we present a set of elementary manipulation primitives for
DLOs, which can be executed solely based on FT sensing, and minimal knowledge
about the environment. These primitives consist of elementary motions such as
clipping and sliding, which are typically encountered in wire harness
manufacturing. They are illustrated in Fig. 5 together with their description.
More complex tasks, such as winding a DLO, are achieved by combining a
sequence of primitives.

Figure 5: For Coordinated Control the DLO is grasped by two end-effectors,
while for Individual Control, the DLO may be grasped by only one end-effector
and constrained by a second fixed point e.g. a fixture, another static end-
effector, etc. Description of manipulation primitives: A \- The DLO is
tensioned. B \- The DLO is let slide through one of the two grippers in a
controlled manner. C \- The DLO is clamped into a fixture with both grippers
in parallel. The fixture exerts a force on the DLO which must be overcome to
fasten the DLO properly. D \- The DLO is tensioned between a gripper and a
fixed point. E \- The DLO is let slide through the gripper in a controlled
manner. F \- The DLO is clamped into a fixture. G \- The gripper is aligned
along the DLO to achieve the most stretched DLO orientation. Any bending will
degrade the linear approximation. H \- The DLO is grasped by another end-
effector along its edge. I \- The DLO is rotated around a feature point. This
is important to improve estimation or to change the orientation of the DLO
under tension.

Each primitive includes a desired force $\text{f}_{d}\in\mathbb{R}$, position
$\mathbf{x}_{d}\in\mathbb{R}^{3}$, quaternion orientation
$\mathbf{Q}_{d}=[q_{d_{0}}\ \mathbf{q}_{d}]\in\mathbb{R}^{4}$ and a projection
matrix $\mathbf{P}\in\mathbb{R}^{12\times 12}$ that specifies the dimensions
along which a force control is executed. The desired end-effector velocities
$\bm{\zeta}_{t}$ in task space are provided by a hybrid motion-force
controller [30] given by:

$\bm{\zeta}_{t}=(\mathbf{I}_{12}-\mathbf{P})\bm{\zeta}_{m}+\mathbf{P}\bm{\zeta}_{f}$
(14)

where the motion control for both arms is given by
$\bm{\zeta}_{m}\in\mathbb{R}^{12}$ and the force control by
$\bm{\zeta}_{f}\in\mathbb{R}^{12}$. The motion control signal $\bm{\zeta}_{m}$
is composed of linear and angular velocity proportional controllers:

$\displaystyle\dot{\mathbf{x}}_{m,i}$
$\displaystyle=k_{p}(\mathbf{x}_{d,i}-\mathbf{x}_{c,i})$ (15)
$\displaystyle\bm{\omega}_{m,i}$
$\displaystyle=k_{o}(q_{c_{0},i}\mathbf{q}_{d,i}-q_{d_{0},i}\mathbf{q}_{c,i}-\mathbf{q}_{d,i}\times\mathbf{q}_{c,i})$
(16)

where $k_{p}$ and $k_{o}$ are positive control gains, $\mathbf{x}_{c,i}$ and
$\mathbf{Q}_{c,i}=[q_{c_{0},i}\ \mathbf{q}_{c,i}]$ describe the current
position and orientation, and the index $i$ takes values in the set
$\\{left,right,abs,rel\\}$. Note that the primitives are divided into
individual control, where the two end-effectors are controlled separately,
with the control signal being defined as:

$\displaystyle\bm{\zeta}_{m}$
$\displaystyle=[\dot{\mathbf{x}}_{m,left}^{T}\quad\bm{\omega}_{m,left}^{T}\quad\dot{\mathbf{x}}_{m,right}^{T}\quad\bm{\omega}_{m,right}^{T}]^{T}$
(17)

and coordinated control, where the absolute and relative end-effector position
are controlled [26] using:

$\displaystyle\bm{\zeta}_{m}$
$\displaystyle=[\dot{\mathbf{x}}_{m,abs}^{T}\quad\bm{\omega}_{m,abs}^{T}\quad\dot{\mathbf{x}}_{m,rel}^{T}\quad\bm{\omega}_{m,rel}^{T}]^{T}$
(18)

The linear velocity $\dot{x}_{f,i}$ used for force control is obtained by a
proportional controller with adaptive damping factor $k_{f}$:

$\dot{x}_{f,i}=k_{f}(\text{f}_{d,i}-\text{f}_{c,i})$ (19)

where $\text{f}_{d,i}$ and $\text{f}_{c,i}$ denote the current and the desired
force respectively. By setting $k_{f}=\hat{k}^{-1}$, where $\hat{k}$
corresponds to the online elasticity estimate, the inverse damping controller
in equation (19) becomes adaptive.

As a representative example of the tensioning primitives, we present D \-
Single arm DLO Tensioning in more detail. The projection matrix is defined
based on the direction between the end-effector and the fixture, that is
$\mathbf{f}_{g}=\mathbf{p}_{c}-\mathbf{p}_{g}$ (see Fig. 4), as follows:

$\displaystyle\mathbf{P}$
$\displaystyle=\text{diag}(\mathbf{f}_{g},\mathbf{0}_{3},\mathbf{0}_{3},\mathbf{0}_{3})$
(20)

where $\mathbf{0}_{n}$ denotes a zero vector of dimension $n$ and
$\text{diag}(\cdot)$ is a diagonal matrix. Setting $\dot{x}_{f,left}$ as the
first three elements of $\bm{\zeta}_{f}$, force control is applied uniformly
to all three dimensions of the left end-effector. The projection matrix
subsequently reduces the force control to the direction along the DLO. As a
representative example of the sliding primitives, we present C \- Parallel DLO
Sliding, where an independent gripping force controller is used:

$p_{finger}=k_{f}(\text{f}_{slid}-\text{f}_{c})+k_{i}\int(\text{f}_{slid}-\text{f}_{c})\,dt$
(21)

Here the projection $\mathbf{P}$ is a zero matrix since force control is
achieved by adjusting the finger distances $p_{finger}$ based on the
frictional tension error $\text{f}_{slid}-\text{f}_{c}$. The integral
component, with parameter $k_{i}$, compensates for offset caused by different
DLO thicknesses. Furthermore, the motion control uses a constant relative
velocity $\dot{x}_{m,rel}$ to slide the end-effector along the DLO while
setting $\dot{\mathbf{x}}_{m,abs}:=\mathbf{0}_{3}$ and
$\bm{\omega}_{m,abs}=\bm{\omega}_{m,rel}:=\mathbf{0}_{3}$.

##  VII Experimental Results

A dual-arm YuMi robot was used for the real-world experiments, where one of
the arms was equipped with an ATI Mini40 6-axis force-torque sensor. The
environment consisted of a pegboard with DLO mounting possibilities which
provided ground truth information about the fixtures.

(a)

(b)

(c)

Figure 6: Experimental Results: (a) The adaptive damping controller is tested
is simulation with three different cables of different elasticities. The plot
shows the error as the spring constant is change from $10$ Nm-1, to $100$ Nm-1
and finally to $1000$ Nm-1. (b) Plot of the uncertainty in the position and
velocity along the $x$-axis. The velocity is mirrored about the horizontal
axis and a positive shift was applied along the vertical axis for visual
comparison of the two measurements. (c) Plots of the estimation error (top)
and uncertainty (bottom) of the contact point along each dimension. The
vertical axis of the top plot is cropped for better visualisation, since the
estimation error directly after the contact point change reaches $86$ mm.

###  VII-A Elasticity Parameter Estimation and DLO Tensioning

An essential pre-requisite for the proposed model is to be able to keep the
DLO under tension. To evaluate the adaptive controller from equation (19), we
tested the D -Single Arm DLO Tensioning primitive with three different DLO
elasticities, as shown in Fig 6(a). For all cases the controller was able to
maintain the DLOs under tension while keeping all oscillations below the force
limits of the robot. Notably, the estimation of the elasticity parameter
improves while the tension error increases. A possible explanation can be that
large oscillations give a wider range of measurements, thus allowing for a
more accurate approximation. On the other hand, a small error leads the system
to oscillate only marginally, resulting in very similar measurements.
Consequently, the measurement noise has a relatively larger influence, which
causes the absolute error to increase.

###  VII-B Contact Point Estimation

The accuracy of the DLO state representation relies on the identification of
contact points by a Kalman filter. In the following experiment, we evaluated
the uncertainty of this estimate for manipulation primitive I \- Rotating
Around Feature Point. The distance between the gripper and the fixture is kept
constant by a tension controller while a circular motion is executed. The
contact point was estimated as described in Section V-A. The uncertainty of
the FT sensor was taken from the sensor’s specification, while the uncertainty
of the end-effector position was disregarded. The process uncertainty
($1\times 10^{-7}$) was tuned manually in order to achieve a satisfactory
trade-off between responsiveness after a contact point change and estimation
precision. During the circular motion, the velocity oscillates along the $x$\-
and $y$-axes. The relationship between the velocity and uncertainty is shown
in Fig. 6(b) along the $x$ dimension.

To further test the robustness of the contact point estimation, while
executing manipulation primitive I \- Rotating Around Feature Point, a second
fixture was placed in the path of the DLO, thus changing the contact point.
Fig. 6(c) consists of two graphs of the same experiment. The top plot shows
the position estimation error and the bottom plot shows the uncertainty along
the three dimensions.

Initially, the first contact point was accurately estimated already after half
a circle. Note that a full rotation corresponds to approximately 125
iterations of the estimator. Once the contact point changed (at the dashed
line), the estimate of the new contact point converged again after an initial
spike in the error. For a circular motion, when the velocity along one
dimension is higher, measurements with a larger angular difference along this
dimension are included in the estimate. However, when there are small
displacements along one dimension, very similar measurements are used for the
estimation, which means that the measurement error has a relatively larger
effect.

###  VII-C Manipulation Primitive Evaluation

The error rate of the manipulation primitives was evaluated. To this aim all
primitives were executed multiple times and the placement of the fixtures, the
initial position of the end effectors, as well as the elasticity and length of
the DLOs were randomly changed. Results are summarized in Table I.

TABLE I: Repeated experiments to evaluate the reliability of the manipulation primitives, for randomized states. Motion Primitive | Repetitions | Failed trials
---|---|---
Parallel DLO Tension - adaptive control | 25 | 2
Single arm DLO Tension - adaptive control | 25 | 1
Parallel DLO Clipping | 15 | 1
Single arm DLO Clipping | 15 | 0
Parallel DLO Sliding | 15 | 0
Single Arm DLO Sliding | 15 | 0
Gripper Alignment | 15 | 0
Grasp a Tensioned DLO - actively tensioned | 15 | 0
Rotate Around Feature Point | 15 | 0
|  |

The three failures of the tensioning primitives were due to poor estimation of
the elasticity parameter. At first, the DLO was estimated to be significantly
more elastic than it actually was, which caused a movement that quickly built
up tension, reaching the limit of the robot. Further, the failure in the C \-
Parallel DLO Clipping primitive was due to an excessive gripper distance which
resulted in. The DLO angle upon contact with the fixture was too narrow to
apply sufficient force to overcome the clipping force.

###  VII-D Harness Production and Model Verification

A wire harness production scenario was used to demonstrate that manipulating
DLOs without visual features, relying on force measurements alone, is
feasible. The task required routing two cables through different fixtures,
loops, and distances. The target configuration of the cables was predefined as
well as the sequence of manipulation primitives111Video demonstration at
https://youtu.be/3Y5HFzrlxds..

In the same experiment, the accuracy of the model was validated by determining
the ground truth of the routed cable via an external camera. The deviation
between model and ground truth in the horizontal plane is on average $7.5$ mm
and maximum $14$ mm. Along the vertical $z$-axis average $16$ mm and maximum
$28$ mm. The larger deviation along the $z$-axis can be explained by the
effect of gravity. The points of the model are estimated when the DLO is under
tension. Depending on the fixture, the DLO was displaced by gravity along the
$z$-axis before the ground truth data was recorded. The importance of the sag
threshold can be understood in the graphical overlay of the segmented image
and the DLO model in Fig. 7. When segments are not tensioned (top left), the
fixed feature points are still correct, but the DLO between the edges is not
linear, as assumed in the model. In contrast, for the taut case (top right),
the model matches the ground truth data well.

Figure 7: The pixel-based DLO segmentation of the image is shown in black. The
feature points and connections of the model are shown in red. The green
crosses show the manually selected ground truth contact points.

##  VIII Conclusion

We presented a DLO manipulation approach relying only on FT sensing. This is
feasible in applications which provide environmental contacts which limit the
DLO’s DOFs and enable tensioning. By keeping the DLO under tension, it is
possible to estimate the location of these contacts, and therefore to keep a
simplified path graph representation of the object’s state. This
representation was successfully employed to solve a cable harness problem.
Future work will aim to complement FT sensing with visual information, as well
as to equip both grippers with FT sensors which would further improve
manipulation.

## References

  * [1] M. Suomalainen, Y. Karayiannidis, and V. Kyrki, “A survey of robot manipulation in contact,” Robotics and Autonomous Systems, p. 104224, 2022\.
  * [2] A. Salem and Y. Karayiannidis, “Robotic assembly of rounded parts with and without threads,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 2467–2474, 2020.
  * [3] D. Almeida and Y. Karayiannidis, “Dexterous manipulation by means of compliant grasps and external contacts,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1913–1920, 2017.
  * [4] L. Manuelli and R. Tedrake, “Localizing external contact using proprioceptive sensors: The contact particle filter,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5062–5069, 2016.
  * [5] J. Sanchez, J. A. C. Ramon, B.-C. Bouzgarrou, and Y. Mezouar, “Robotic manipulation and sensing of deformable objects in domestic and industrial applications: A survey,” The International Journal of Robotics Research, vol. 37, pp. 688 – 716, 2018.
  * [6] K. Galassi and G. Palli, “Robotic wires manipulation for switchgear cabling and wiring harness manufacturing,” in 2021 4th IEEE International Conference on Industrial Cyber-Physical Systems (ICPS), pp. 531–536, 2021.
  * [7] J. Sanchez, K. Mohy El Dine, J. A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar, “Blind manipulation of deformable objects based on force sensing and finite element modeling,” Frontiers in Robotics and AI, vol. 7, p. 73, 2020.
  * [8] V. E. Arriola-Rios, P. Guler, F. Ficuciello, D. Kragic, B. Siciliano, and J. L. Wyatt, “Modeling of deformable objects for robotic manipulation: A tutorial and review,” Frontiers in Robotics and AI, vol. 7, 2020.
  * [9] H. Yin, A. Varava, and D. Kragic, “Modeling, learning, perception, and control methods for deformable object manipulation,” Science Robotics, 6(54), 2021\.
  * [10] E. Nabil, B. Belhassen-Chedli, and G. Grigore, “Soft material modeling for robotic manipulation,” Annalen der Physik, vol. 162, pp. 184–193, 2012\.
  * [11] M. Müller, J. Dorsey, L. McMillan, R. Jagnow, and B. Cutler, “Stable real-time deformations,” ETH Zürich, 2002.
  * [12] D. McConachie, “Deformable object manipulation: Learning while doing,” University of Michigan, 2020.
  * [13] S. Miller, J. van den Berg, M. Fritz, T. Darrell, K. Goldberg, and P. Abbeel, “A geometric approach to robotic laundry folding,” The International Journal of Robotics Research, vol. 31, no. 2, pp. 249–267, 2012.
  * [14] J. Schulman, A. Lee, J. Ho, and P. Abbeel, “Tracking deformable objects with point clouds,” in 2013 IEEE International Conference on Robotics and Automation, pp. 1130–1137, 2013.
  * [15] T. Tang and M. Tomizuka, “Track deformable objects from point clouds with structure preserved registration,” The International Journal of Robotics Research, 2019.
  * [16] D. Henrich, T. Ogasawara, and H. Worn, “Manipulating deformable linear objects-contact states and point contacts,” in Proceedings of the 1999 IEEE International Symposium on Assembly and Task Planning (ISATP’99)(Cat. No. 99TH8470), pp. 198–204, IEEE, 1999.
  * [17] J. Acker and D. Henrich, “Manipulating deformable linear objects: characteristic features for vision-based detection of contact state transitions,” in Proceedings of the IEEE International Symposium on Assembly and Task Planning, 2003., pp. 204–209, IEEE, 2003.
  * [18] J. Zhu, B. Navarro, R. Passama, P. Fraisse, A. Crosnier, and A. Cherubini, “Robotic manipulation planning for shaping deformable linear objects with environmental contacts,” IEEE Robotics and Automation Letters, vol. 5, no. 1, pp. 16–23, 2019.
  * [19] J. Sanchez, C. M. Mateo, J. A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar, “Online shape estimation based on tactile sensing and deformation modeling for robot manipulation,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 504–511, 2018.
  * [20] P. Boonvisut and M. C. Çavuşoğlu, “Estimation of soft tissue mechanical parameters from robotic manipulation data,” IEEE/ASME Transactions on Mechatronics, vol. 18, no. 5, pp. 1602–1611, 2013.
  * [21] F. Barbara, S. Rüdiger, S. Cyrill, T. Matthias, and B. Wolfram, “Learning the elasticity parameters of deformable objects with a manipulation robot,” in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1877–1883, 2010.
  * [22] M. Shetab-Bushehri, M. Aranda, Y. Mezouar, and E. Özgür, “As-rigid-as-possible shape servoing,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3898–3905, 2022.
  * [23] D. Berenson, “Manipulation of deformable objects without modeling and simulating deformation,” International Conference on Intelligent Robots and Systems (IROS), 2013.
  * [24] M. Yu, H. Zhong, F. Zhong, and X. Li, “Adaptive control for robotic manipulation of deformable linear objects with offline and online learning of unknown models,” arXiv preprint arXiv:2107.00194, 2021.
  * [25] L. Romain, K. Alexandre, and M. Maud, “Automatic shape control of deformable wires based on model-free visual servoing,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 5252–5259, 2020.
  * [26] G. A. Waltersson, R. Laezza, and Y. Karayiannidis, “Planning and control for cable-routing with dual-arm robot,” IEEE International Conference on Robotics and Automation (ICRA), 2022.
  * [27] J. Guo, J. Zhang, D. Wu, Y. Gai, and K. Chen, “An algorithm based on bidirectional searching and geometric constrained sampling for automatic manipulation planning in aircraft cable assembly,” Journal of Manufacturing Systems, vol. 57, pp. 158–168, 2020.
  * [28] J. Wittenburg, H. A. Richard, and J. Zierep, Das Ingenieurwissen Technische Mechanik.  Springer-Verlag Berlin Heidelberg, 2014.
  * [29] S. Dierk, Intelligente Verfahren.  Springer Berlin Heidelberg, 2017.
  * [30] K. M. Lynch and F. C. Park, Modern Robotics.  Cambridge University Press, 2017.