# An Evaluation of Score Level Fusion Approaches for Fingerprint and Finger-
vein Biometrics

Kamer Vishi and Vasileios Mavroeidis
Department of Informatics SecurityLab
University of Oslo (UiO) Norway
{kamerv<EMAIL_ADDRESS>
###### Abstract

Biometric systems have to address many requirements, such as large population
coverage, demographic diversity, varied deployment environment, as well as
practical aspects like performance and spoofing attacks. Traditional unimodal
biometric systems do not fully meet the aforementioned requirements making
them vulnerable and susceptible to different types of attacks. In response to
that, modern biometric systems combine multiple biometric modalities at
different fusion levels. The fused score is decisive to classify an unknown
user as a genuine or impostor. In this paper, we evaluate combinations of
score normalization and fusion techniques using two modalities (fingerprint
and finger-vein) with the goal of identifying which one achieves better
improvement rate over traditional unimodal biometric systems. The individual
scores obtained from finger-veins and fingerprints are combined at score level
using three score normalization techniques (min-max, z-score, hyperbolic
tangent) and four score fusion approaches (minimum score, maximum score,
simple sum, user weighting). The experimental results proved that the
combination of hyperbolic tangent score normalization technique with the
simple sum fusion approach achieve the best improvement rate of 99.98%.

Keywords: multibiometrics, biometric fusion, fingerprint, finger-vein,
authentication systems, identity management, privacy, security

##  1 Introduction

The most important practical consideration for a biometric system is its
accuracy. The combination of multiple different samples or multiple biometric
modalities is a natural approach for improving the performance of a biometric
system. This approach depends on specific methods for data fusion during the
analytical process. Biometric fusion can be performed at different levels such
as sensor level, feature (template) level, score level and decision level.
Currently, the most popular and suitable way of biometric fusion is at score
level since all commercially available biometric sensors and feature
extractors do not provide access to their feature extraction algorithms.

Some of the limitations of the aforementioned unimodal biometric systems are
overcome by combining multiple biometric modalities. These systems are known
as multimodal biometric systems and are more secure due to the integration of
multiple independent pieces of evidence [1]. For instance, the chance of
getting a valuable biometric system increases with the number of involved
biometric traits. In addition, the integration of multiple traits increase
security since it is more difficult to spoof multiple biometric modalities of
real users [2]. Advantages of multimodal biometric systems over unimodal
systems have been discussed by Ross et al. [3].

In this paper, we present an evaluation of normalization and fusion techniques
using finger-vein and fingerprint biometrics and their potential application
as biometric identifiers. The individual scores obtained from finger-vein and
fingerprints are combined at score level using three score normalization
techniques (Min-Max, Z-Score, Hyperbolic Tangent) and four score fusion
approaches (Minimum Score, Maximum Score Simple Sum and User Weighting). The
fused-score classifies an unknown user as genuine or impostor.

The rest of the paper is organized as follows: Section 2 gives an overview of
related work, Section 3 describes the normalization and fusion techniques,
Section 4 presents the experiments conducted and the data analysis, Section 5
presents the experimental results, and lastly, Section 6 rounds off the paper
with a discussion and indication of future work.

##  2 Related Work

Different score fusion approaches have been proposed for fusing scores
received from different biometric modalities.

Raghavendra et al. [4] proposed a multimodal biometric score level fusion
scheme using Gaussian Mixture Model and Monte Carlo Method. The authors fused
face, speech and palmprint modalities and showed that their method gives
higher performance than other fusion schemes, such as Simple sum rule,
Weighted sum rule, Fishers Linear Discriminate Analysis (FLD), and Likelihood
Ratio (LR) methods.

Derawi et al. [5] fused gait and fingerprint traits for user authentication on
mobile devices. The authors used four different methods to normalize the
scores (min-max, z-score, median absolute deviation, tangent hyperbolic) and
four fusion approaches (simple sum, user-weighting, maximum score and minimum
score). The fusion results of fingerprint and gait recognition showed an
improved performance over other methods and brought as a step closer to the
advancement of user authentication on mobile devices.

Peng et al. [6] fused finger vein, fingerprint, finger shape and finger
knuckle print of a single human finger based on triangular norm (t-norm). The
results showed that score level fusion using triangular norm obtains a larger
distance between genuine and imposter score distribution, as well as lower
error rates.

Conti et al. [7] and Lau et al. [8] proposed a multi-instance fusion approach.
Their approach is based on fusing two comparison scores using fuzzy logic
rules from two different fingerprints. Experimental results showed an
improvement of 6.7% using the comparison score level fusion rather than a
unimodal authentication system.

Kisku et al. [9] proposed a multimodal biometric system using face and ear
biometrics. They used Gaussian Mixture Model (GMM) with belief fusion for the
estimated scores characterized by Gabor responses, and the proposed fusion is
accomplished by Dempster-Shafer (DS) decision theory. It has been proved that
DS provides increased accuracy and significant improvements over the existing
classical fusion rules.

##  3 Score Level Fusion

Most of the multimodal biometric systems integrate data at score level due to
the strong trade-off between the ease in combining the data and better
information content. Besides, it is a relatively straightforward way to
combine scores generated by different comparators (matchers)111Note that the
term ”matching” as a synonym for ”comparison” has been deprecated in the ISO
SC37 Harmonized Biometric Vocabulary [10].. Therefore, score level fusion is
the preferred approach for integrating biometric data. Figure 1 illustrates
this process.

Figure 1: Score Level Fusion Diagram for Fingerprint and Finger-vein.

At score level, fused scores of each modality are combined with a variety of
techniques to produce a new score that is compared with the threshold in the
decision module. There are two key approaches in use today for consolidating
comparison scores; classification and combination. In classification we can
construct a vector with individual scores that are classified into accept or
reject classes. A classification approach might use a decision tree, Support
Vector Machine (SVM), or Linear Discriminate Analysis (LDA) to classify the
vector as an impostor or genuine. In combination approach, individual scores
are combined to generate a single scalar score to render the final decision
[11]. The combination approach for consolidating scores has compiled a
superior performance record versus the other levels.

Score level fusion has two main steps. The first step is score normalization,
where calculated scores are created by a certain comparator $S_{i}$ and are
mapped onto a new score scale or domain $S_{i}^{{}^{\prime}}$. For instance,
if a comparator X produces scores on a domain $[1,100]$ and a comparator Y
generates scores on a domain $[1,2500]$ these scores need to be normalized and
mapped at a common scale. The second step of fusion at score level is fusion.
There are many score fusion techniques, however, in this paper, we have used
only some of the fusion techniques that are recommended by ISO standards -
"ISO/IEC TR 24722:2007 - Multimodal and other Multibiometric Fusion"[12] and
are described in Section 3.2.

###  3.1 Score Normalization

Score normalization processes have been researched extensively [11]. In this
section, we describe some fundamental normalization concepts used in
biometrics to facilitate easier interpretation of our paper. The score
normalization process is performed to transform the comparator’s parameters
and data types into a common domain. Commonly, score normalization techniques
are evaluated based on their robustness and efficiency.

The most used score normalization techniques that are also used and evaluated
in this paper are Min-Max (MM), Z-Score (ZS), and Hyperbolic Tangent (TanH).
These methods are discussed below.

Table 1: Symbols Used for Score Normalization Expressions. Statistical Measures | Genuine Distribution | Impostor Distribution | Both
---|---|---|---
Minimum Score | $S_{Min}^{G}$ | $S_{Min}^{I}$ | $S_{Min}^{B}$
Maximum Score | $S_{Max}^{G}$ | $S_{Max}^{I}$ | $S_{Max}^{B}$
Mean | $S_{Mean}^{G}$ | $S_{Mean}^{I}$ | $S_{Mean}^{B}$
Score Standard Deviation | $S_{SD}^{G}$ | $S_{SD}^{I}$ | $S_{SD}^{B}$

Min-Max Normalization (MM)

performs a linear transformation of the original data. This is one of the
simplest normalization techniques and is appropriate when the limits of the
produced scores are known. MM is considered efficient and provides adequate
performance; however, it may not yield completely accurate results if the data
contain outliers. MM maps raw scores to the [0,1] range and given scores
S${}_{Max}^{B}$ and S${}_{Min}^{B}$ designates the endpoints of the score
range.

$S^{\prime}=\frac{S-S_{Min}^{B}}{S_{Max}^{B}-S_{Min}^{B}}$ (1)

Z-Score Normalization (ZS)

is one of the most commonly used normalization techniques. It uses an
arithmetic mean and a standard deviation to normalize the data; therefore, a
priori knowledge regarding the average score and score variances of the
comparator is needed. ZS is considered efficient and tends to work
exceptionally well if the scores of each modality follow a Gaussian
distribution; however, this technique may not achieve similar accuracy if the
data used contain outliers (the mean and standard deviation are sensitive to
outliers). ZS normalization transforms the scores to a normal distribution
with an arithmetic mean S${}_{Mean}^{I}$ of 0 and a standard deviation
S${}_{SD}^{I}$ of 1.

$S^{\prime}=\frac{S-S_{Mean}^{I}}{S_{SD}^{I}}$ (2)

Hyperbolic Tangent Normalization (TanH)

is efficient (when the parameters are selected carefully), provides adequate
performance, and is very robust in handling outliers. TanH maps the raw scores
to the (0,1) range, where S${}_{Mean}^{G}$ and S${}_{SD}^{B}$ are the mean and
standard deviation estimation of the score distribution, respectively.

$S^{\prime}=0.5\cdot tanh\cdot\frac{0.01(S-S_{Mean}^{G})}{S_{SD}^{B}}+1$ (3)

Figure 2, illustrates distribution graphs for the aforementioned fingerprint
score normalization techniques; where x represents raw scores before
normalization and y represents normalized scores.

Figure 2: Distributions of Genuine and Impostor Comparison Scores.

###  3.2 Score Fusion Techniques

In general, score fusion techniques fall under two categories: classification
and combination. Classification techniques formulate the problem by dividing
the decision space into two classes: genuine and impostor. The reliability and
effectiveness of this method are dependent on the large amount and quality of
the input data that are used to train the classifier. In addition, the
comparison scores do not need to be homogeneous; hence the normalization step
is not required. Some of the classification methods that have been researched
are neural networks, nearest neighborhood algorithms, and tree-based
classifiers.

Combination is the most common and effective technique for combining biometric
scores. This technique combines scores from multiple comparators and generates
a single score. Combination requires score normalization before fusing the
scores. This paper analyses the following score level fusion techniques:
Maximum Score, Minimum Score, Simple Sum, and Weighted Sum.

Maximum Score:

the max rule estimates the mean of the posteriori probabilities by the maximum
value.

$max(i=1\>to\>N)\>S_{i}^{\prime}$ (4)

Minimum Score:

the min rule sets the minimum value of posteriori probabilities.

$min(i=1\>to\>N)\>S_{i}^{\prime}$ (5)

Simple Sum:

it is a weighted average of the raw scores. Comparison scores are summed
without the benefit of normalization routines. It simplistically assumes that
the raw scores supplied by the biometric methods used have a comparable scale,
distribution, and strength. It can be used in ambiguous classifications
resulting from high-level noise.

$\sum(i=1\>to\>N)\>S_{i}^{\prime}$ (6)

Weighted Sum:

this method computes the combined score as a weighted sum of the comparison
scores. The motivation behind the idea of user-specific weights for computing
a weighted sum of scores is that some biometric traits cannot be reliably
obtained from some people (e.g., individuals with faint fingerprints).
Assigning a lower weight to a fingerprint score and a higher weight to a
finger-vein score reduces the probability of a false rejection.

$\sum(i=1\>to\>N)\>W_{i}^{*}\cdot S_{i}^{\prime}$ (7)

When it is implemented correctly, score level fusion can improve accuracy,
thwart fraudsters, and increase usability. On the other hand, in case it is
implemented incorrectly, a multibiometric system might experience performance
degradation in comparison to a unimodal system. Furthermore, multimodal
biometric systems have some disadvantages. They could potentially have a
higher cost of ownership, they could increase user inconvenience, they could
decrease user acceptance, and also exacerbate privacy issues.

##  4 Experiment Setup

In this section we present details regarding the database, datasets and
modifications needed to conduct our experiments.

###  4.1 Database

Fingerprint and finger-vein experiments in this paper are conducted over four
different datasets collected during the summer of 2010 by the Machine Learning
and Applications (MLA) Group at Shandong University, in China [13].

The MLA group named this database ”SDUMLA- HMT: A Multimodal Biometric
Database”. The database includes 106 subjects; 61 males and 45 females between
17 and 31 years old [13]. The database consists of face images captured from
seven different view angles, finger vein images of six fingers, gait videos
from six view angles, finger-vein images from a finger-vein sensor and
fingerprint images captured by five different sensors. Figure 3, shows some
sample images from SDUMLA-HMT database.

Figure 3: Sample Images of Face, Finger-vein, Gait, Iris and Fingerprint from
SDUMLA-HMT Database [13].

In addition, modifications were needed over the acquired data to conduct our
experiment. In particular, we have reduced the number of participants,
fingers, and impressions as follows: we use two index fingers of both hands
out of 6 fingers provided in the dataset, five impressions for each
fingerprint out of eight that were captured, and we reduced the number of
participants from 106 to 100. These modifications were demanded because in the
provided database there were available only finger-vein images of the index
fingers. Consequently, we have
$2(fingers)x100(subjects)x5(attempts)=1000\>images$ per dataset. In other
words, we used 2000 fingerprint images from two datasets (DS2 and DS3) out of
a total of 25,440 fingerprint images. Furthermore, we decided to follow the
naming convention recommended in ISO 19794-2 [14]. To do that we requested the
original finger-code positions from the MLA Group and we renamed them
accordingly.

An illustration of finger-code positions (names) from the MLA Group and ISO
19794-2 is given in Figure 4.

Figure 4: Illustration of Finger-code Positions (names).

###  4.2 Fusion Experiments

Based on a quality assessment conducted over the database, the best datasets
are named with suffix 1, while the worst datasets are named with suffix 2.

  * •

Fingerprint Datasets:

    * –

Fingerprint, best quality dataset (SDUMLA-HMT DS2) is named as FP_DS1,

    * –

Fingerprint, worst quality dataset (SDUMLA-HMT DS3) is named as FP_DS2.

  * •

Finger-vein Datasets:

    * –

Finger-vein, best quality dataset (SDUMLA-HMT Finger-vein-Lamp) is named as
FV_DS1,

    * –

Finger-vein, worst quality dataset (SDUMLA-HMT Finger-vein) is named as
FV_DS2.

We have defined four fusion scenarios; fusion with the best and the worst
dataset as follows:

  1. 1.

Fusion of FP_DS1 and FV_DS1

  2. 2.

Fusion of FP_DS1 and FV_DS2

  3. 3.

Fusion of FP_DS2 and FV_DS1

  4. 4.

Fusion of FP_DS2 and FV_DS2

##  5 Experimental Results

We have retrieved both low and high Equal Error Rates (EERs) from unimodal
biometric systems. In table 2, we can see that TanH gives better performance
than MinMax and Z-Score. Additionally, Simple Sum gives better results than
Maximum Score, Minimum Score and User Weighted Sum. Specifically, Simple Sum
yields 99.98 % average verification accuracy, making it the most stable fusion
framework.

Table 2: Sample Comparison Results for Different Normalization and Fusion
Techniques. EER (%)
---
| Normalization Technique
Fusion Technique | MinMax(MM) | Z-Score(ZS) | TanH
Minimum Score (MinS) | 3.81836% | 7.32851% | 1.01881%
Maximum Score (MaxS) | 0.11591% | 0.11206% | 0.07837%
User Weighting (UW) | 0.08949% | 0.15935% | 0.01763%
Simple Sum (SS) | 0.08281% | 0.10955% | 0.00010%

###  5.1 Fusion Scenarios

In the following sections we summarize the results of four fusion scenarios
(fusion with the best and the worst dataset).

####  5.1.1  Scenario #1:Fusion of FP_DS1 and FV_DS1

This case shows the fusion performance of finger-vein dataset FV_DS1 and
fingerprint dataset FP_DS1 using Hyperbolic Tangent estimators (TanH)
normalization and Simple Sum (SS). The results show that the fingerprints
dataset attains an EER of 0.86%, the finger-veins dataset an EER of 0.71%, and
their fusion a low EER of 0.00010%.

####  5.1.2 Scenario #2: Fusion of FP_DS1 and FV_DS2

This scenario shows the fusion performance of the fingerprint dataset FP_DS1
and finger-vein dataset FV_DS2 using Hyperbolic Tangent (TanH) and Maximum
Score (MaxS). The results show that the fingerprints dataset attains an EER of
0.86%, the finger-veins dataset an EER of 7.35%, and their fusion a low EER of
0.0320%.

####  5.1.3 Scenario #3: Fusion of FP_DS2 and FV_DS1

This case shows the fusion performance of the fingerprint FP_DS2 and the
finger-vein datasets FV_DS1 using MinMax normalization (MM) and Maximum Score
(MaxS) fusion rule. The results show that the fingerprints dataset attains an
EER of 1.01%, the finger-veins dataset an EER of 0.71%, and their fusion a low
EER of 0.00015%.

####  5.1.4 Scenario #4: Fusion of FP_DS2 and FV_DS2

This case shows fusion performance of the fingerprint FP_DS2 and finger-vein
datasets FV_DS2 using Hyperbolic Tangent estimators (TanH) normalization and
Maximum Score fusion rule. The results show that the fingerprints dataset
attains an EER of 1.01%, the finger-veins dataset an EER of 7.35%, and their
fusion a low EER of 0.0038%.

Figure 5, presents four graphs for different normalization and fusion
techniques per fusion scenario (we get an insight of the increased performance
when TanH and SS is used, in relation to other normalization and fusion
techniques).

Figure 5: Performance of Fingerprint and Finger-vein Using TanH Score
Normalization and Simple Sum Score Fusion

To summarize, the results indicate that fusion of finger-vein and fingerprint
improves the performance of biometric systems. We can define improvement as
the percentage difference between the minimum EER of fingerprint or finger-
vein (we choose the lowest value between the two) minus the EER of their
fusion and (divided by) the minimum EER of fingerprint or finger-vein (we
choose the lowest value between the two):

$\displaystyle Improvement$ $\displaystyle=$
$\displaystyle\frac{EER_{min(print|vein)}-EER_{fusion(print+vein)}}{EER_{min(print|vein)}}\cdot
100$ (8)

Table 3 summarizes our results. The values in percentages indicate EER.

Fingerprint alone Finger-vein alone Fusion(print+vein) Improvement (method)
0.86 % 0.71 % 0.0001 % 99.98 % (TanH+SS) 0.86 % 7.35 % 3.81 % 48.10 %
(MM+MinS) 1.01 % 0.71 % 0.1872 % 79.50 % (TanH+UW) 1.01 % 7.35 % 0.0038 %
89.62 % (ZS+SS)

Table 3: Multimodal Fusion Improvements of Fingerprint and Finger-vein
Recognition.

##  6 Conclusion

Multibiometric systems are the new frontier, and this is evident from the
number of ongoing efforts in the research and commercial domains. These
systems address some of the deficiencies mentioned above of the unimodal
biometric systems. Theoretically, they present a huge potential, but research
and commercial systems have yet to reflect this promise. Furthermore, the lack
of standards indicates that more work is required before it reaches an
acceptable level of maturity.

Experimental results show that in most cases fused performance (fingerprint
and finger-vein) was significantly improved compared to unimodal biometric. It
is worth noting that the best fusion performance is achieved by the
combination of Hyperbolic Tangent (TanH) score normalization technique and
Simple Sum (SS) method for fusion, yielding an EER of 0.00010%. In other
words, a multimodal biometric system with fingerprint and finger-vein would
perform 99.98% better than a unimodal biometric system of fingerprint or
finger-vein. In addition, it has been observed that both MinMax and Z-Score
methods are sensitive to outliers. On the other hand, TanH score normalization
is both robust and efficient.

Nevertheless, our future work should investigate fusion approaches using
fingerprint and finger-vein biometrics at feature extraction level or template
level, to secure biometric templates and enhance privacy by hiding the meaning
of extracted features (points) from finger-vein and fingerprint in the stored
template.

## Acknowledgement

This research is supported by the Research Council of Norway under the Grant
No.: IKTPLUSS 248030/O70 and 247648 for SWAN and Oslo Analytics projects,
respectively. This research is also part of the SecurityLab of University of
Oslo.

## References

  * [1] L. I. Kuncheva, C. J. Whitaker, C. A. Shipp, and R. P. W. Duin.  Is Independence Good for Combining Classifiers?  In Proceedings of the 15th International Conference on Pattern Recognition, volume 2, pages 168 –171 vol.2, 2000.
  * [2] B. Bhanu and V. Govindaraju.  Multibiometrics for Human Identification.  Cambridge University Press, 2011.
  * [3] Anil K. Jain and Arun Ross.  Multibiometric Systems.  Communications of the ACM, 47(1):34, 2004.
  * [4] R. Raghavendra, A. Rao, and G. Hemantha Kumar.  A Novel Approach for Multimodal Biometric Score Fusion Using Gaussian Mixture Model and Monte Carlo Method.  International Conference on Advances in Recent Technologies in Communication and Computing. ARTCom ’09., pages 90–92, 2009.
  * [5] M. O. Derawi, D. Gafurov, R. Larsen, C. Busch, and P. Bours.  Fusion of gait and fingerprint for user authentication on mobile devices.  In 2010 2nd International Workshop on Security and Communication Networks (IWSCN), pages 1–6, May 2010.
  * [6] Jialiang Peng, Ahmed A. Abd El-Latif, Qiong Li, and Xiamu Niu.  Multimodal Biometric Authentication Based on Score Level Fusion of Finger Biometrics.  Optik - International Journal for Light and Electron Optics, 125(23):6891–6897, 2014.
  * [7] Vincenzo Conti, Giovanni Milici, Patrizia Ribino, Filippo Sorbello, and Salvatore Vitabile.  Fuzzy Fusion in Multimodal Biometric Systems, pages 108–115.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2007.
  * [8] CW Lau, Bin Ma, HML Meng, YS Moon, and Yeung Yam.  Fuzzy Logic Decision Fusion in a Multimodal Biometric System.  Interspeech, 2004.
  * [9] Dakshina Ranjan Kisku, P Gupta, H Mehrotra, and J Sing.  Multimodal Belief Fusion for Face and Ear Biometrics.  Intelligent Information Management, 01:166–171, 2009.
  * [10] ISO/IEC JTC 1/SC 37 Biometrics: SC 37 Standing Document 11 (SD 11), Part 1 Harmonization Document - Biometric Vocabulary.
  * [11] Kamer Vishi and Audun Jøsang.  A new approach for multi-biometric fusion based on subjective logic.  In Proceedings of the 1st International Conference on Internet of Things and Machine Learning, IML ’17, pages 68:1–68:9, New York, NY, USA, 2017. ACM.
  * [12] ISO/IEC TR 24722:2007, Information Technology - Biometrics - Multimodal and Other Multibiometric Fusion, 2007.
  * [13] Yilong Yin, Lili Liu, and Xiwei Sun.  SDUMLA-HMT: A Multimodal Biometric Database.  Biometric Recognition, pages 260–268, 2011.
  * [14] ISO/IEC 19794-2:2005, Information technology – Biometric Data Interchange Formats – Part 2: Finger Minutiae Data, 2005.


# Machine Learning Based Compensation for
Inconsistencies in Knitted Force Sensors

Roland Aigner
<EMAIL_ADDRESS>0000-0002-8503-4335 Andreas Stöckl
<EMAIL_ADDRESS>0000-0003-1646-0514

###### Abstract

Knitted sensors frequently suffer from inconsistencies due to innate effects
such as offset, relaxation, and drift. These properties, in combination, make
it challenging to reliably map from sensor data to physical actuation. In this
paper, we demonstrate a method for counteracting this by applying processing
using a minimal artificial neural network (ANN) in combination with
straightforward pre-processing. We apply a number of exponential smoothing
filters on a re-sampled sensor signal, to produce features that preserve
different levels of historical sensor data and, in combination, represent an
adequate state of previous sensor actuation. By training a three-layer ANN
with a total of 8 neurons, we manage to significantly improve the mapping
between sensor reading and actuation force. Our findings also show that our
technique translates to sensors of reasonably different composition in terms
of material and structure, and it can furthermore be applied to related
physical features such as strain.

##  1 Introduction

Textile based sensors are of high interest in research and industry due to
numerous beneficial properties, such as lightness, breathability, and
potential stretchability. In particular, knits are inherently elastic
textiles, due to their geometric composition of courses of interlocking loops,
as opposed to weaves, where yarn is travelling straight. This elasticity makes
them ideal for sensing stress or strain [1, 2, 3] that generally perform
according to Holm’s theory [4], which states that contact resistance $R$
depends on material resistivity $\rho$, material hardness $H$, contact point
count $n$, and pressure $P$, with

$R=\frac{\rho}{2}\sqrt{\frac{\pi H}{nP}}\,.$

Consequently, the overall sensor resistance drops when pressure at the loop
intermeshing points’ contacts is increased, e.g., by straining or pressing.
However, depending on the yarn material properties and/or structural
composition of a knitted structure, knitted sensors usually suffer from
considerable inconsistencies that have to be addressed [5, 6, 7], such as
settling effects, offset, overshooting, hysteresis, as well as long- and
short-term sensor drift, some of which we speculate are due to slight
structural re-arrangements of the yarn within the fabric. Inherently, the raw
measurement signal tends to seriously deviate from the desired output, i.e.,
the applied force. This is undesirable in several use cases since it
complicates downstream analysis of raw sensor data. Unfortunately, since these
effects are unlike common noise, traditional methods such as frequency-domain-
or Kalman-filters are insufficient to get rid of those.

Figure 1: Timeline of one of our recordings using the PES sensor (top):
Overlaying normalized trends of ground truth $\overline{F}$ and sensor
readings $\overline{G}$ clearly shows an upwards drift of the sensor readings.
Zooming into three segments of the timeline (bottom left), reveals
inconsistencies and even contradicting effects, such as overshooting vs.
underestimation in the first segment. Our method utilizes a neural network
that is trained using several smoothed sensor signals as features (bottom
right). We combine a set of those with different smoothing factors to
incorporate different degrees of historic information.

Figure 1 (top) shows an example timeline plot of a recording of applied force
and resulting sensor reading. Long-term drift is most eminent, however short-
term inconsistencies are also apparent when zooming in (cf. Figure 1 bottom
left, for a magnification of three timeline snippets), which illustrate that a
basic mapping, e.g., by multi-point calibration is impractical and not
promising. Due to the nature of a knitted fabric, effects like latency,
hysteresis, drift, offset, overshooting, etc. (and the interaction thereof)
are innate. Furthermore, as their extent depends also on the chosen knitting
structure, which makes their use challenging for scenarios where reliability
is required.

Our main hypothesis for this work is that inconsistencies that are reflected
in the sampling data are in fact ultimately deterministic, as their cause is
in the knits’ physical and geometrical composition, however too complex to
analyze or model manually from empirical observations. Hence, our approach is
to utilize an artificial neural network (ANN) to learn and model those factors
instead. A further objective is to keep the computational complexity low by
reducing the number of required features and by using small-scale networks, so
our method is viable for low-end embedded devices with highly limited
computational capabilities.

In this paper, we present an easy-to-implement method for mitigating this
error, which is based on a small-scale multilayer perceptron neural network
(MLP NN) and comes with little modeling/training effort and low computational
cost. NNs are in general frequently used for modeling complex non-linear
systems, as it has been shown that even small networks are able to well
approximate any continuous function of $n$ real variables [8, 9]. Moreover,
MLP networks are relatively easily trained using Backpropagation (BP)
algorithms [10, 11].

The main contributions of this paper are as follows:

  * •

a method for rectifying inherent inconsistencies in raw sensor data that
result from the structural nature of knitted sensors.

  * •

description of a related feature-set that we used as an input vector for an
ANN that model several levels of historic sensor data.

  * •

mitigation of short-term errors, was well as removal of long-term sensor
drift, that go beyond sensor hysteresis.

  * •

an exemplary processing pipeline including a NN of minimal complexity that is
easily trained and computationally undemanding during operation.

  * •

a demonstration of the method’s transferability to sensor knits of different
structure and to different objectives (e.g., mapping sensor data to stain
instead of stress).

While there are numerous works that used NNs for classification, e.g., for
detecting hand posture [12], others have used neural networks in scenarios
similar to ours, however mostly for compensating hysteretic behavior. Dang et
al. [13] used RBF NNs to model Preisach hysteresis of piezoceramic actuators.
Similarly, Lien et al. [14] used hysteretic recurrent NNs in the context of
piezoelectric actuators, modeling hysteresis in the neurons’ activation
function. Tong et al. [15] used BHSMs to test a NN that approximates
hysteretic non-linearities. Wu et al. [16] implemented a dynamic NN structure
based on the Hammerstein model for dynamic error compensation of infrared
thermometer sensors. More recently, Weiss et al. [17] applied Kalman filters
for preprocessing chemical sensors’ data for downstream machine learning.
Jondhale et al. [18] combined Kalman filters with GRNNs for 2D-position
tracking from RSSI signals. In the field of textile-based sensing, Atitallah
et al. [19] compared filtering methods such as moving average, moving median,
Savitzky-Golay, and Gaussian for processing data from a sensor glove that
incorporated CNT-based sensors. Vu et al. [20] implemented an adaptive fuzzy-
NN for capacitive pressure sensors, which were based on spacer-knit
structures. Finally, Liu et al. [21] utilized RBF NNs to compensate for
hysteresis disturbance in non-affine, nonlinear systems, however the test set
consists of limited, generated data that is furthermore repetitive.

While all those works are related, objective, material, and use cases differ
from ours and hence the techniques are hardly transferable. To our knowledge,
there is so far no related work applying NNs for data rectification of knitted
piezoresistive stress/strain sensors targeted at random, real-world actuation.

##  2 Sensor Implementation & Data Acquisition

Our sensors utilize a widespread knitting pattern that is often, however
inconsistently, called ”Twill” in the textile industry, due to its structural
similarity with a Twill Weave. It consists of courses with alternating knit
and float stitches, shifted by one needle every other row (cf. Figure 2(a)) .
The high number of floats results in exceptional stability along course-
direction and high elasticity along wale-direction, when compared to more
straightforward knits, such as Plain or Double Jersey [22]. In this regard it
exhibits a characteristic similar to a Cardigan, however with orthogonal
anisotropic behavior, which can be an advantage in certain use case scenario,
where omni-directional elasticity is not desired.

(a)

(b)

(c)

Figure 2: Closeups and knitting patterns of the Twill based knitting
structures (a): for the PES- (tubular structure, left), and Lycra (connected,
i.e., PES tucked to back-face Resistat, right). PES sensor patch (b), with
conductive yarn traces connecting the resistive area (black) on both upper and
lower ends. We evaluated our sensors using a custom-built tensile tester which
is equipped with a force cell (c).

Since a Twill can be knitted on a single needle bed, the opposite bed is
available for additional structures. We decided to apply our knitted sensors
on a substrate carrier structure, by knitting a Twill consisting of PES on the
front bed and another Twill using resistive yarn on the back bed, to have the
sensor part completely covered by the PES on one side for protecting it from
abrasion. Furthermore, this gave us better control in balancing for a uniform
stability across the whole fabric, when compared to integrating a sensor patch
as an Intarsia [12]. Note that this requires the two faces to be connected to
not fall apart; we did this by tucking the resistive yarn to the PES at the
outer wales of the sensor area produces a tubular structure (cf. Figure 2(a),
left).

From a previous study [23], we learned that adding Lycra to the substrate can
significantly improve elastic recoil, minimizing hysteresis. We therefore
fabricated two variations: one with pure PES-substrate, one with additional
Lycra. Since the resistive face is not complemented with Lycra, we tightly
connected front and back faces for these sensor patches, by tucking the
substrate to the sensor knit for every loop (cf. Figure 2(a), right), to
prevented interference from resistive face’s lagging behind the elastic
substrate. We decided to include this additional version in our evaluation, to
estimate how well our method translates to sensors of different design.

For the substrate carrier, we used a den 150 PES from TWD Fibres GmbH, and for
the Lycra thread we used a 140 Lycra core covered with PES den 150/20 from
Jörg Lederer GmbH. For the resistive sensing areas, we used Polyester-based,
Carbon-sheathed Resistat P6204111https://shakespeare-pf.com/product/polyester/
from Shakespeare® with den 100/24 and $\scriptstyle\mathtt{\sim}$10
M$\Omega$/m. For the connector traces, we used silver-coated PA-yarn Madeira
HC40222https://www.shieldex.de/products/madeira-hc-40/, with den 260 and <300
$\Omega$/m. All our patches were knitted on a flat-bed knitting machine of
type ADF 530-32 KI W Multi Gauge from KARL MAYER STOLL, at gauge E 7.2. For
more details regarding materials and fabrication of our sensors, we refer to
[23].

##  3 Data Acquisition

For collecting training and test data, we used a custom-made tensile tester,
which we built from a CNC milling machine (cf. Figure 2(c)). The clamps for
attaching the patches at both ends featured needles at 2 cm distance to secure
the textile against slipping. The moving actuator was equipped with a single-
point load-cell (Sauter CP P1-Ba-d-1810333https://www.kern-
sohn.com/shop/en/measuring-instruments/measuring-cells/CP-P1/) and was sampled
at $\scriptstyle\mathtt{\sim}$40 Hz by an ADS 1231 24-bit Delta-Sigma
ADC444https://www.ti.com/product/ADS1231. Sensor resistance values were
acquired by a simple voltage divider with a 606 k$\Omega$ reference resistor,
sampled by an Adafruit ADS1115 16-bit
ADC555https://www.adafruit.com/product/1085 at $\scriptstyle\mathtt{\sim}$128
Hz. We buffered samples and averaged values in windows of
$\scriptstyle\mathtt{\sim}$25 ms, again resulting in a rate of
$\scriptstyle\mathtt{\sim}$40 Hz for our final samples. Measurements for force
and resistance, as well as actuator displacement and timestamps were captured
into CSV files by the MCU firmware. An ESP32 on an Adafruit HUZZAH32 Feather
board666https://www.adafruit.com/product/3405 was used for sampling and
recording to SD card.

Figure 3: Strain-histograms of our three trajectories show approximate
Gaussian distribution (top). Due to different knitting structures and material
compositions, the two sensors operate in different force ranges when actuated
using identical trajectories (bottom).

The tensile tester was controlled by Art-Soft Mach4 CNC Control Software
(v4.2.0), running on a Windows 10 PC. To simulate pseudo-natural motion for
reasonably representative actuation of the sensors and collecting
corresponding sensor data, we generated non-repetitive G-code trajectories
based on Perlin Noise [24, 25], to control the actuator along the x-axis. We
created three of those trajectories, to get one data set for training our ML
models and two for testing their performance. Based on pre-evaluation of the
sensors [23], we chose the amplitudes of our trajectories to move in an
approximate range of 0% to 30% extension (cf. Figure 3, top). Maximum, mean,
and SD of velocities and accelerations can be found in Table 1. We collected
data for both of our sensors, i.e., PES-only (”PES”) and Lycra-enhanced
(”Lycra”) variants. Note that this resulted in different force ranges, due to
the difference in firmness (cf. Figures 3, bottom). Our objective was to test
our method on sensors of slightly different structure and material, to get an
estimation of its portability between sensor designs. The total duration of
our actuation process and the timespan we recorded was
$\scriptstyle\mathtt{\sim}$23 minutes each.

Table 1: Statistics of our three generated actuator trajectories. Velocities were around 1 mm/s, with a maximum of 4.930 mm/s. | v [mm/s] | a [mm/s²]
---|---|---
traj. | max | mean | SD | max | mean | SD
#1 | 4.142 | 0.943 | 0.593 | 19.05 | 1.397 | 1.303
#2 | 4.930 | 0.979 | 0.617 | 18.61 | 1.497 | 1.343
#3 | 4.039 | 1.030 | 0.644 | 19.25 | 1.546 | 1.414

##  4 Data Processing

We noticed our sampling frequency was not perfectly constant (µ=41.5 Hz,
=14.2), as a consequence of our multi-component setup and the serial
communication with Mach4, which sometimes caused lags. We therefore re-sampled
our recorded force and resistance data to 20 Hz by linear interpolation in
between the sample points, to simplify handling of the data in our scripts.

The main objective of our work is to infer force-data from the raw
measurements that were taken from the sensor, meaning both trends should be as
identical as possible, which is not the case with raw measurement data (cf.
Figure 1). Hence, we argue the coefficient of determination $r^{2}$ is a
reasonable metric for quantifying this property and hence to judge about the
performance of our approach777Note that we use the less-common lower-case
notation $r^{2}$ to avoid confusion with the sensors’ electrical resistance
$R$.. However, since force $F$ and sensor resistance $R$ are inversely
proportional, we utilize the sensor conductivity $G=1/R$ instead of the
resistance. Furthermore, since both $F$ and $G$ cover largely different
ranges, we normalized both to identical ranges to $\overline{F}$ and
$\overline{G}$, by removing mean and scaling to unit variance using the
StandardScaler from the scikit learn Python package888https://scikit-
learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html.
This pre-processing step is also beneficial (and in fact recommended [26]) for
better performance of machine learning estimators later on. Hence, we
calculate the coefficient of determination with

$r^{2}(X,Y)=1-\frac{\sum_{i}\left(x_{i}-y_{i}\right)^{2}}{\sum_{i}\left(x_{i}-\texttt{mean}(Y)\right)^{2}}\,$

substituting $X$ with normalized force measurement $\overline{F}$, and $Y$
with pre-processed and normalized conductivity measurement $\overline{G}$. To
quantify the performance of our machine learning model, we then use the
prediction $p$ for $Y$ instead.

After pre-processing our raw data, we calculated our initial $r^{2}$ scores as
baseline values with 0.423, 0.471, and 0.526 for the recordings with our PES
version and as 0.703, 0.667, and 0.734 for those with the Lycra version (cf.
Table 3). It is already eminent that the performance of our Lycra patch is
superior, which is in line with previous findings [23]. However, we first
focus on the most basic implementation, without additional Lycra, which we
expect to benefit most from improvement by computational means.

As mentioned, our hypothesis is that an ANN is able to model the knit’s state
and infer the actuation from an (seemingly randomly) inconsistent sensor
reading. For example, it seems reasonable that sensor offset and settling
speed is affected by recent elongation. From that follows, that there needs to
be historic data available for the current prediction; this can either be
implemented by a feedback mechanism, or otherwise by choosing input features
that include temporal information. For sake of simplicity, we decided for the
latter. Using a number of $n$ previous samples within a certain time-frame as
features is not promising since this would result in a very high number of
features and would therefore rapidly increase complexity of computation and
network topology, increasing the risk of overfitting. Furthermore, this
approach would highly depend on the sample-rate. Instead, we decided for
providing historic data in the form of several smoothed signals, with varying
degrees of responsiveness. We utilized exponential smoothing [27] with

$y(t)=\alpha x_{t}+(1-\alpha)y(t-1)\,,$

where $\alpha$ is a smoothing factor in range [0 1]; i.e., the lower the value
for $\alpha$, the higher the drag. We noticed initialization with $y(0)=x_{0}$
introduced too much bias for the signals with high drag, therefore we
initialized with the mean of samples values within a window of M samples,
$y(0)=\texttt{mean}(x_{0}\ldots x_{M-1})$. For calculating an adequate window-
size that is depending on drag and sample rate $f$, we empirically found
$M=1/\texttt{ceil}(f\alpha)$ delivers reasonable initialization values. By
filtering the sensor conductivity $\overline{G}$ with a set of smoothing
factors $\alpha_{i}$, we gain a set of $N$ filtered sensor signals $G_{i}$,
reflecting different degrees of temporal data (cf. Figure 1, bottom right),
which represent the elements of our feature vector. Note that our exponential
smoothing implementation does not take timing into account, which is a further
reason we re-sampled our data to a constant rate of 20 Hz.

Multi-layer Perceptrons (MLP) trained by back-propagation (BP) algorithms are
commonly used for function approximation, we therefore used the
MLPRegressor999https://scikit-
learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html of
scikit learn, with relu activation function and maximum iterations of 10.000.
Note that MLPs are particularly sensitive to feature scaling [28], which makes
our previously described pre-processing mandatory.

We started with experimental $\alpha$ values of 0.5, 0.1, 0.025, and 0.0025,
which gained promising results. From there, we tried different sets of
smoothing factor vectors $A=(\alpha_{1}\ldots\alpha_{N})$, with
$\alpha_{i}=1/a^{i}$, modifying $a$ and $N$ to get several sets of different
sizes and granularity. Since we found that data smoothed with $\alpha$ values
below 10-4 held too little information, we mostly refrained from going beyond
those. Apart from feature vectors, we varied the ANN’s hidden layer sizes
(i.e., neuron counts) and topologies. We did not go beyond neuron counts of 32
and beyond 4 hidden layers, since we started to notice frequent overfitting at
these values. We then empirically tested all permutations101010912 variations,
after removing duplicates and those including layers with less than 2 neurons,
both due to flooring the products of those parameters to find the optimal
configuration (cf. Table 2 for a complete listing).

Table 2: Listing of all our used parameters. Smoothing factor count N and base value a were varied for determining factors $\alpha_{i}$, e.g., a = 10 and N = 3 result in A = (10-1, 10-2, 10-3). Number of hidden layers’ (HL) neurons are determined by building the products of base sizes with topology vectors and flooring the results, e.g., 6 $\times$ (½, 1, ¼) would give three layers with 3, 6, and 1 neurons, respectively. When removing all permutations containing sizes <2, 114 unique variations remain. params | values |
---|---|---
(a, N) | (2.5, 4) | (0.4, 0.16, 0.064, 0.0256)
| (2.5, 7) | (0.4, 0.16, …0.0016384)
| (2.5, 10) | (0.4, 0.16, …0.000104858)
| (5, 4) | (0.2, 0.04, 0.008, 0.0016)
| (5, 7) | (0.2, 0.04, …0.0000128)
| (10, 3) | (0.1, 0.01, 0.001)
| (10, 4) | (0.1, 0.01, 0.001, 0.0001)
| N/A (baseline) | (0.5, 0.1, 0.025, 0.0025)
HL base | 2, 3, 4, 6, 8, 12, 16, 32
topology |  ( 1, 1 ), ( 1, ½), ( 1, ¼),
( 1, 1, 1 ), ( 1, 1, ½), ( 1, ½, ½), ( 1, ½, ¼),
( ½, 1, 1 ), ( ½, 1, ½), ( ½, 1, ¼),
( 1, 1, 1, 1 ), ( 1, 1, 1, ½), ( 1, 1, ½, ½),
( 1, ½, ½, ½), ( 1, ½, ½, ¼), ( 1, ½, ¼, ¼),
( 1, ¼, ¼, ¼), ( ½, 1, 1, 1 ), ( ½, 1, 1, ½),
( ½, 1, ½, ½), ( 1, ½, ½, ½)
Table 3: $r^{2}$ of our initial (pre-processed) and predicted data. We included results of our two test sets (A, B), as well as the training sets (T) for sake of completeness. PES show highest gain from our approach; however, the Lycra patches ultimately yield higher scores. |  r²(F,G) |  r²(F,p) |  gain
---|---|---|---
PESt |  0.423 |  (0.781) |  (0.358)
PESA |  0.471 |  0.791 |  0.320
PESB |  0.526 |  0.767 |  0.241
Lycrat |  0.703 |  (0.841) |  (0.138)
LycraA |  0.667 |  0.830 |  0.163
LycraB |  0.734 |  0.828 |  0.094

##  5 Evaluation and Discussion

###  5.1 Results

Our tests showed that a low number of hidden layers worked well in most cases
and increasing them did not considerably improve scores. Occasionally, adding
a fourth hidden layer did even degrade results, but this seems also to be
subject to network topology. Overall, hidden layer base sizes of at least 4
were required, otherwise the resulting models would be unusable. In terms of
features, we saw that smoothing factors resulting from $a$ = 10 (with both $N$
= 3 and $N$ = 4) did not perform well. We speculate this is because the
smoothing factors are too far apart and the resulting low number of $G_{i}$
features include too little historical information. Overall, our variations
with $a$ = 5 performed better, however the best scores were achieved with $a$
= 2.5 and $N$ <10\. A spreadsheet including all the scores of our network
variations can be found in the supplementary material.

Figure 4: We feed pre-processed (re-sampled and normalized) data
$\overline{G}$ into a number of exponential smoothing filters using different
smoothing factors and use the results as feature vector for our neural
network. In our tests, the combination of a = 2.5 and N = 7 with hidden layer
sizes (4, 2, 2) resulted in the best $r^{2}$ score.

Our systematic tests resulted in the best $r^{2}$ score for the parameter-
combinations of $a$ = 2.5, $N$ = 7 (i.e., $A$ = (0.4, 0.16, 0.064, 0.0256,
0.01024, 0.004096, 0.0016384), HL base size = 4, and topology = (1, ½, ½),
which gives neuron counts of (4, 2, 2) (cf. Figure 4). Predictions of force
values from our training sets resulted in $r^{2}$ values of 0.791 and 0.767
(PES), as well as 0.830 and 0.828 (Lycra), producing highest gains for the PES
variants. Figure 5 shows the result of test set #1, with $p$ overlaid on
$\overline{G}$ and $\overline{F}$. It is striking that the long-term drift was
removed. Inspection of section snippets (cf. Figure 5, left) reveal
considerable improvement over the pre-processed input signal that goes well
beyond what could be achieved with more basic transformation techniques. Using
identical parameters, the network was also trained for the Lycra test-set, and
as can be seen in Figure 5 (right), there is similar improvement. In general,
however, we can observe occasional underestimation of peaks, while the model
seems to perform very well at low-force areas. Figure 6 shows a comparison
between deviations before and after applying the correction, i.e., pre-
processed input signal $\overline{G}$ vs. prediction $p$. We calculated the
difference between each of the signals and the ground-truth $F$ values, binned
them by $F$ and calculated relative squared error for each bin. We can see
that correction is most effective in lower regions, in particular below 7 N
for PES and below 15 N for Lycra.

Figure 5: Our results show that prediction values $p$ rectify sensor
inconsistencies $G$ exceptionally well. Most striking is the removal of long-
term drift (top). When looking more closely at segments (left), we see that
$p$ aligns well with ground truth data $F$ in most cases. The same method also
translates well to our Lycra-variants using different yarn material and
knitting composition. Figure 6: Comparisons of binned relative squared error
(RSE) of pre-processed data ($\overline{G}$) and prediction ($p$) indicates
that most of the signal correction is accomplished at lower-force areas.

###  5.2 Further Experiments

Since the majority of related work utilizes knitted sensors not primarily as
force sensors but rather as strain sensors, we ran our recordings of actuator
offset through the same pipeline to see if our technique also translates to
this objective. As mentioned above, $d$ also drifts over time which may be the
initial cause of this long-term drifting effect, since $G$ does not
considerably drift relative to $d$, however both trends still differ
significantly, with initial $r^{2}$-scores between 0.260 and 0.319 (PES), as
well as 0.337 and 0.490 (Lycra). Using our unmodified processing pipeline as
presented in Figure 4, we were able to boost those scores up to 0.699 (PES),
and 0.669 (Lycra) for the test sets (further details can be found in the
supplement). This implies that our method works exceptionally not only for
force as a main metric.

In terms of feature choice would like to note that we tried several
variations, e.g., including additional information: first, we experimented
with slope values, i.e., first derivatives of each of the smoothed signals
$G_{i}$, as well as a set of smoothed first derivatives of $G$, however we
found those did not improve prediction quality significantly as they seem
redundant. Second, we briefly experimented with features taken from the
frequency domain by calculating windowed FFTs, however the signal turned out
to contain little information beyond very low frequencies and we did not go
into great lengths to exploring this direction any further. In terms of
alternative machine learning methods, we experimented with
linear111111https://scikit-
learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html,
polynomial121212https://scikit-
learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html
(3rd and 4th order), and random forest regressors131313https://scikit-
learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html,
but were not able to produce results of comparable quality.

Furthermore, we briefly tried re-sampling to rates other than 20 Hz, namely
half and twice the frequency. We got slightly worse results with 10 Hz and
similar results with 40 Hz, so we kept our initial value of 20 Hz for data
pre-processing.

###  5.3 Discussion & Limitations

Our results show that our approach performs remarkably well in improving the
mapping from sensor reading to physical actuation. It also proves successful
when applied to a sensor with considerable differences in structure and
elastic behavior. Furthermore, the network can be trained to successfully
predict not only force but also strain values. The fact that our processing
pipeline is exceptionally small and requires little computation effort, makes
it viable for applications in highly limited environments and platforms. We
would like to note that due to the nature of our method that is based on
features from smoothed sensor readings, as opposed to resorting to a learned
hysteresis-model, it is reasonable to assume that our method adapts well to
permanent effects, such as chemically and mechanically induced material
degradation.

We do acknowledge a few limitations of this work. First, we did not go into
great length in investigating entirely different ANN topologies, such as Deep
Belief Networks [29], Extreme Learning Machines [30], Echo State Networks
[31], etc. We do not expect serious performance gains by changing the topology
type, however it will be an interesting direction to explore, and we leave
this for future work.

Second, we do not have data that goes beyond our roughly 23 minutes
recordings. We did observe in prior evaluations, that drift decreases in a
logarithmic manner, therefore we expect the most challenging effect of drift
is to be addressed at the beginning. Furthermore, our method presents a multi-
purpose and adaptive way of handling the issue in that it provides a means of
modeling long-term drift in the form of highly smoothed sensor signals (cf.
$G_{6}$ and $G_{7}$ in Figure 1, bottom right) and use this as highly compact
and low-complexity features.

Third, we present a method of predicting scaled data. Mapping the range to
meaningful physical values will require some initial calibration step. Note
that this calibration would also be required without our pipeline, since,
e.g., readings in Ohms or Siemens need to be translated to Newtons either way.

Fourth, we mentioned we re-sampled our data to a constant sample-rate, since
our exponential smoothing filters are not considering timing data and are
therefore sensitive to varying $\Delta t$. We believe this is not that much of
an issue in real-world applications, since our issue arose from our testing
apparatus, syncing data from multiple sources. As mentioned, the actual ADC
and firmware would be able to sample at a much higher rate (128 Hz in our
case) and the solution comes down to implement a solid down-sampling routine,
that outputs at a reasonably constant rate.

Fifth, for this investigation, we initialized our exponential smoothing
filters with the mean of the first $M$ samples to avoid biased starting values
(cf. Section 4). Applying this in a real-world application would require a
short duration for initialization for collecting those samples. However, we
did not see this as a major limitation and not as the core of this work. We
trust it is not a serious challenge to find alternative initialization methods
to set $y(0)$ as there are numerous ways to do so [32], or to substitute the
entire smoothing filter, as we do not believe our method relies on this exact
method for smoothing.

##  6 Conclusion

We demonstrated a method of utilizing an ANN for correcting inconsistent
sampling data read from a knitted resistive force sensor, by pre-filtering the
raw input signal and thus providing multiple levels of temporal information as
a feature vector to the NN. Once trained, the pipeline can be used as a real-
time filter for translating sensor readings to physical actuation data. We
demonstrated our method using a MLP NN that in its best-performing
configuration requires only three hidden layers and a total of 8 neurons to
achieve considerable improvement over the input data, which signifies
exceptionally low computational requirements and therefore facilitates
applications in a wide variety of scenarios. Although we applied the technique
to translate raw sensor data to the trend of physical force/stress, the method
can be easily translated to related metrics such as strain. Furthermore, we
successfully applied our technique to a knitted sensor of different structure
and behavior, which implies it translates well to slightly different
conditions, possibly even to entirely different use cases beyond knitted
sensors that suffer from similar issues, such as considerable hysteresis and
drift. No doubt, the very best solution to specific sensors may vary in
detail, slight modifications of the network and features (e.g., smoothing
factors and number of features) may be beneficial for fine-tuning to the
scenario at hand. However, we noticed during our experiments that many of
those slight adjustments only result in minor improvements that may not be
significant or representative and could be subject to the particular training
data.

## Acknowledgment

This research is part of the COMET project TextileUX (No. 865791, which is
funded within the framework of COMET – Competence Centers for Excellent
Technologies by BMVIT, BMDW, and the State of Upper Austria. The COMET program
is handled by the FFG.

## References

  * [1] D. De Rossi, A. Della Santa, and A. Mazzoldi, “Dressware: wearable hardware,” Materials Science and Engineering: C, vol. 7, no. 1, pp. 31–35, 1999.
  * [2] E. P. Scilingo, F. Lorussi, A. Mazzoldi, and D. De Rossi, “Strain-sensing fabrics for wearable kinaesthetic-like systems,” IEEE Sensors Journal, vol. 3, no. 4, pp. 460–467, 2003.
  * [3] X. Liang, H. Cong, Z. Dong, and G. Jiang, “Size prediction and electrical performance of knitted strain sensors,” Polymers, vol. 14, no. 12, 2022\.
  * [4] R. Holm, ”Electric Contacts: Theory and Application”.  ”Heidelberg”: ”Springer Berlin”, 4 ed., 1967.
  * [5] H. Zhang, X. Tao, T. Yu, and S. Wang, “Conductive knitted fabric as large-strain gauge under high temperature,” Sensors and Actuators A: Physical, vol. 126, no. 1, pp. 129–140, 2006.
  * [6] A. Grassi, F. Cecchi, M. Maselli, M. Röling, C. Laschi, and M. Cianchetti, “Warp-knitted textile as a strain sensor: Characterization procedure and application in a comfortable wearable goniometer,” IEEE Sensors Journal, vol. 17, no. 18, pp. 5927–5936, 2017.
  * [7] B. Bozali, J. J. F. van Dam, L. Plaude, and K. M. B. Jansen, “Development of hysteresis-free and linear knitted strain sensors for smart textile applications,” in 2021 IEEE Sensors, pp. 1–4, 2021.
  * [8] G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of Control, Signals and Systems, vol. 2, no. 4, pp. 303–314, 1989\.
  * [9] J. Park and I. W. Sandberg, “Universal Approximation Using Radial-Basis-Function Networks,” Neural Computation, vol. 3, pp. 246–257, 06 1991.
  * [10] S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach.  Prentice Hall, 3 ed., 2010.
  * [11] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, pp. 536–536, 1986.
  * [12] S. Lee, Y. Choi, M. Sung, J. Bae, and Y. Choi, “A knitted sensing glove for human hand postures pattern recognition,” Sensors, vol. 21, no. 4, 2021\.
  * [13] X. Dang and Y. Tan, “An inner product-based dynamic neural network hysteresis model for piezoceramic actuators,” Sensors and Actuators A: Physical, vol. 121, no. 2, pp. 535–542, 2005.
  * [14] J. Lien, A. York, T. Fang, and G. D. Buckner, “Modeling piezoelectric actuators with hysteretic recurrent neural networks,” Sensors and Actuators A: Physical, vol. 163, no. 2, pp. 516–525, 2010.
  * [15] Z. Tong, Y. Tan, and X. Zeng, “Modeling hysteresis using hybrid method of continuous transformation and neural networks,” Sensors and Actuators A: Physical, vol. 119, no. 1, pp. 254–262, 2005.
  * [16] D. Wu, S. Huang, W. Zhao, and J. Xin, “Infrared thermometer sensor dynamic error compensation using hammerstein neural network,” Sensors and Actuators A: Physical, vol. 149, no. 1, pp. 152–158, 2009.
  * [17] M. Weiss, M. S. Wiederoder, R. C. Paffenroth, E. C. Nallon, C. J. Bright, V. P. Schnee, S. McGraw, M. Polcha, and J. R. Uzarski, “Applications of the kalman filter to chemical sensors for downstream machine learning,” IEEE Sensors Journal, vol. 18, no. 13, pp. 5455–5463, 2018.
  * [18] S. R. Jondhale and R. S. Deshpande, “Kalman filtering framework-based real time target tracking in wireless sensor networks using generalized regression neural networks,” IEEE Sensors Journal, vol. 19, no. 1, pp. 224–233, 2019\.
  * [19] B. Ben Atitallah, J. R. Bautista-Quijano, H. Ayari, A. Y. Kallel, D. Bouchaala, N. Derbel, and O. Kanoun, “Comparative study of digital filters for a smart glove functionalized with nanocomposite strain sensor,” in 2021 18th International Multi-Conference on Systems, Signals & Devices (SSD), pp. 1366–1371, 2021.
  * [20] C. C. Vu and J. Kim, “Highly elastic capacitive pressure sensor based on smart textiles for full-range human motion monitoring,” Sensors and Actuators A: Physical, vol. 314, p. 112029, 2020.
  * [21] W. Liu and T. Zhao, “An active disturbance rejection control for hysteresis compensation based on neural networks adaptive control,” ISA Transactions, vol. 109, pp. 81–88, 2021.
  * [22] D. J. Spencer, Knitting Technology: A Comprehensive Handbook and Practical Guide.  Woodhead Publishing, 3 ed., 2001.
  * [23] R. Aigner and F. Hepper, “An evaluation of multi-component weft-knitted twill structures for sensing tensile force,” 2023.
  * [24] K. Perlin, “An image synthesizer,” in Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’85, (New York, NY, USA), p. 287–296, Association for Computing Machinery, 1985.
  * [25] K. Perlin, “Improving noise,” in Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’02, (New York, NY, USA), p. 681–682, Association for Computing Machinery, 2002.
  * [26] scikit learn, “scikit-learn user guide: 6.3. preprocessing data,” 2023.  Accessed on May 30th, 2023.
  * [27] E. S. Gardner Jr, “Exponential smoothing: The state of the art,” Journal of forecasting, vol. 4, no. 1, pp. 1–28, 1985.
  * [28] scikit learn, “scikit-learn user guide: 1.17. neural network models (supervised),” 2023.  Accessed on May 30th, 2023.
  * [29] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in Proceedings of the 19th International Conference on Neural Information Processing Systems, NIPS’06, (Cambridge, MA, USA), p. 153–160, MIT Press, 2006.
  * [30] G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, “Extreme learning machine: Theory and applications,” Neurocomputing, vol. 70, no. 1, pp. 489–501, 2006.  Neural Networks.
  * [31] H. Jaeger and H. Haas, “Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication,” Science, vol. 304, no. 5667, pp. 78–80, 2004.
  * [32] S. Nahmias, Production and Operations Analysis.  McGraw-Hill Companies,Incorporated, 2008.