# Chaos in Motion: Unveiling Robustness in Remote Heart Rate Measurement
through Brain-Inspired Skin Tracking

Jie Wang, Jing Lian, Minjie Ma, Junqiang Lei, Chunbiao Li, Bin Li, Jizhao Liu
This work is supported by the Natural Science Foundation of Gansu Province
(Grants 21JR7RA510, 21JR7RA345, 22JR5RA543). (Corresponding author: Jizhao
Liu.)Jie Wang, Bin Li, Jizhao Liu are with the School of Information Science
and Engineering, Lanzhou University, Lanzhou 730000, China (e-mail:
<EMAIL_ADDRESS>liujz@lzu.edu.cn). Jing Lian is with the School of
Electronics and Information Engineering, Lanzhou Jiaotong University, Lanzhou
730070, China (e-mail: lian322scc@163.com).Minjie Ma is with the Department of
Thoracic Surgery, The First Hospital of Lanzhou University, Lanzhou 730000,
China (e-mail: maminjie24@sina.com).Junqiang Lei is with the Department of
Radiology, The First Hospital of Lanzhou University, Lanzhou 730000, China
(e-mail: leijq@lzu.edu.cn).Chunbiao Li is with the School of Artificial
Intelligence, Nanjing University of Information Science and Technology,
Nanjing 210044, China (e-mail: chunbiaolee@nuist.edu.cn).

###### Abstract

Heart rate is an important physiological indicator of human health status.
Existing remote heart rate measurement methods typically involve facial
detection followed by signal extraction from the region of interest (ROI).
These SOTA methods have three serious problems: (a) inaccuracies even failures
in detection caused by environmental influences or subject movement; (b)
failures for special patients such as infants and burn victims; (c) privacy
leakage issues resulting from collecting face video. To address these issues,
we regard the remote heart rate measurement as the process of analyzing the
spatiotemporal characteristics of the optical flow signal in the video. We
apply chaos theory to computer vision tasks for the first time, thus designing
a brain-inspired framework. Firstly, using an artificial primary visual cortex
model to extract the skin in the videos, and then calculate heart rate by
time-frequency analysis on all pixels. Our method achieves Robust Skin
Tracking for Heart Rate measurement, called HR-RST. The experimental results
show that HR-RST overcomes the difficulty of environmental influences and
effectively tracks the subject movement. Moreover, the method could extend to
other body parts. Consequently, the method can be applied to special patients
and effectively protect individual privacy, offering an innovative solution.

###### Index Terms:

Biomedical monitoring, remote heart rate measurement, rPPG, brain-inspired
neural network, ROI.

##  I Introduction

Heart rate (HR) stands as a crucial and convenient physiological indicator,
with its precise measurement proving important for diagnosing cardiovascular
diseases and providing daily medical care. The method of remote heart rate
measurement using regular cameras offers non-contact advantages in comparison
to traditional measurement techniques. This method can present innovative
avenues for comprehensive cardiovascular health assessment and diagnosis, it
operates on the fundamental principle that the pulsation of the heart leads to
pulsations within the blood flow. As blood flows through the skin, it induces
subtle, periodic color variations in the skin area. From this phenomenon, the
corresponding blood volume pulse (BVP) signal [1] can be reconstructed,
facilitating the feasibility of remote heart rate measurement. However, BVP
signals are exceptionally weak and susceptible to noise interference arising
from fluctuations in lighting conditions and subject movements. In response to
these challenges, a myriad of techniques have been proposed to acquire stable
BVP signals for heart rate measurement. These techniques can be grouped into
two categories: traditional signal processing methods and deep learning
methods.

Remote photoplethysmography (rPPG) has gained attention for its convenience
and speed. Recent advancements in deep learning methods, including meta-
learning, generative adversarial networks, and transformers, show significant
potential in rPPG measurement [2, 3, 4]. However, these methods often require
subjects to face the camera with frontal facial orientations, leading to
inaccuracies in face detection in real-world scenarios where subjects struggle
to maintain stable postures. Non-frontal views can result in imprecise
positioning of ROI pixels and undetected facial features, rendering ROI
acquisition for signal extraction unfeasible [5]. Furthermore, unique skin
conditions and limited relevant databases for special patients pose challenges
for current heart rate measurement methods, despite the crucial need for
accurate heart rate acquisition for diagnosing and treating special patients.
Additionally, utilizing face videos for heart rate measurement raises privacy
concerns, particularly when collected without individuals’ consent for
biometric measurement purposes [6]. The challenges are illustrated in Fig. 1.

Figure 1: Challenges with existing heart rate measurement methods.

To address these issues, we design a robust motion-aware remote heart rate
measurement framework, which can accurately identify skin ROI and be applied
to body parts other than the face, making it applicable for special patients
and addressing privacy protection concerns. Our framework contains three
phases: (a) ROI extraction phase; (b) signal analysis phase; and (c) HR
calculation phase.

In the ROI extraction phase, we introduced brain-inspired neural networks
(continuous coupled neural network, CCNN) for robust skin tracking. CCNN
encodes changing pixels into chaotic signals and static pixels into periodic
ones, distinguishing skin from non-skin pixels. It excels in precise ROI
extraction, overcoming head movement challenges. We have expanded this method
to enhance privacy protection and extend to clinical applications like infant
monitoring. In the signal analysis phase, we perform spatiotemporal feature
analysis of optical flow signals within ROI videos. After extracting RGB
signals, we filter the G-channel and conduct time-frequency analysis on
temporal signals for heart rate calculation. Finally, mode values of heart
rates at each pixel yield final measurements. This framework addresses
inaccurate face detection, special patient applications, and privacy concerns,
reducing dependency on extensive databases.

The main contributions of this work are summarized as follows:

  * •

We propose a novel remote heart rate measurement framework consisting of three
phases: ROI extraction, signal analysis, and HR calculation. Each phase has a
clear physical interpretation, ensuring high interpretability of the
framework.

  * •

To the best of our knowledge, the proposed ROI extraction method differs from
all other methods relying on facial detection. In comparison, we introduce
chaos theory to computer vision tasks, applying the video processing mechanism
of the primary visual cortex for remote heart rate measurement.

  * •

We show that this framework effectively extracts skin ROI, demonstrating
robustness against motion and enabling reliable ROI extraction for subjects
with head movements. It extends applicability to other body parts, ensuring
applications for special patients and protecting individual privacy.

##  II RELATED WORK

Relevant prior work includes studies of heart rate measurement, traditional
signal processing methods based on rPPG, and deep learning methods.

###  II-A Heart Rate Measurement

HR measurement methods, encompassing contact and non-contact approaches, play
vital roles in clinical and daily settings [7]. While electrocardiography
(ECG) is prevalent in clinical contexts, photoplethysmography (PPG) through
wearable devices is common in daily life. However, both methods have
limitations, such as discomfort and the need for skilled personnel and
electrode application [8, 9, 10]. In contrast, non-contact methods like remote
photoplethysmography (rPPG), utilizing regular cameras, offer a solution by
enabling contactless monitoring. Despite the advantages of rPPG under
consistent ambient lighting, changes in lighting conditions and subject head
movements pose challenges [11]. These challenges, including dynamic lighting
changes and subject movement complexities, have prompted the transition from
traditional signal processing to deep learning methods [12].

###  II-B Traditional Signal Processing Methods Based on rPPG

In the early stages of rPPG technology research, researchers emphasized blind
source separation (BSS) techniques, like principal component analysis (PCA)
[13] and independent component analysis (ICA) [14], to extract clean BVP
signals from mixed signals. However, this required extensive trial and
advanced estimation techniques [15]. Distinguishing signal distortions from
pulse signals and periodic motion was challenging, hindering broader
application. To address BSS limitations, De Haan [16] introduced a
standardized skin color vector and skin light reflection models across various
color channels. Leveraging signal complementarity in different color spaces,
he proposed the chrominance-based rPPG (CHROM) algorithm, mitigating subject
head movement effects on pulse signal extraction. Wang [17] expanded this
method with the plane orthogonal to the skin (POS) algorithm, facilitating
pulse signal extraction. Li [18] addressed background illumination variation
noise through illumination rectification, eliminated non-rigid motion caused
by facial expressions like smiling, and improved HR measurement accuracy.
Tulyakov [19] introduced a self-adaptive matrix completion optimization
framework, robustly estimating heart rate and dynamically selecting useful
facial regions for measurement, overcoming noise from subject movements and
facial expressions.

Traditional methods often require complex signal processing and rely on rPPG
signal characteristics, skin reflection models, and prior assumptions. This
makes them susceptible to variations in environmental illumination and
individual differences, resulting in signal quality instability. While these
methods have pioneered rPPG technology, the research focus has shifted towards
deep learning methods due to these challenges [12].

###  II-C Deep Learning Methods

Hsu [20] introduced a deep learning framework for real-time pulse estimation
using an RGB camera, achieving superior performance on public databases.
Following this, research in rPPG technology shifted towards deep learning
methods due to their strong modeling capabilities and in-depth exploration of
algorithms. Lee [2] argues that end-to-end learning methods use a global model
during inference, which may not adapt well to unforeseen changes in real-world
applications, affecting estimation performance. Therefore, he proposed a
transductive meta-learner that adjusts weights using unlabeled samples during
deployment, achieving fast adaptation to distribution changes. Niu [21]
constructed multi-scale spatiotemporal maps with face information and proposed
a feature disentangling strategy to achieve robust physiological measurements.
Lu [3] points out that existing methods primarily focus on weak BVP signal
enhancement without explicitly modeling noise, so he used Dual-GAN to jointly
model BVP predictor and noise distribution, achieving robust remote
physiological measurement. Yu [4] suggests recent deep learning methods focus
on mining subtle rPPG clues using CNNs with limited spatiotemporal receptive
fields. He used temporal difference transformers to build a framework for
long-range spatiotemporal perception, showing superior performance in intra-
and cross-database experiments. Deep learning methods in rPPG technology
leverage powerful modeling to mine the complex relationship between skin color
changes and pulse signals, significantly improving measurement accuracy [12].

While deep learning methods have shown high accuracy, their lack of
interpretability poses challenges in understanding how they learn or extract
heart rate-related signals, especially when face detection is hindered by
severe head movements [22]. Despite their success on existing public
databases, replicating such results in other databases or real-world
applications is not guaranteed. Moreover, deep learning methods typically
require large amounts of training data and long training times, demanding
significant computing resources. Our method addresses these limitations by
overcoming head movement tracking challenges, robustly extracting motion-
unaffected skin areas, generating ROI synthetic videos, and performing time-
frequency analysis on each pixel’s time signal for accurate heart rate
extraction. It offers good interpretability, strong generalization, and low
computing resource requirements.

Figure 2: The framework of HR-RST.

##  III PROPOSED METHOD

###  III-A Problem Statement and Challenges

Extracting ROI is crucial in heart rate measurement tasks, often accomplished
using established face detection algorithms like CLNF OpenFace2.2.0 [23],
SeetaFace111https://github.com/seetaface/SeetaFaceEngine, MTCNN face detector
[24], etc. These algorithms offer high real-time performance by providing
facial landmark points, which can guide ROI extraction either through the face
frame or the coordinates of these points, as depicted in Fig. 1A. However,
significant head movements can introduce inaccuracies in landmark positioning,
leading to errors in recognizing facial edge pixels and potentially including
environmental areas in the extracted ROI. Privacy concerns surrounding the use
of face videos in remote heart rate measurement research have also become
prominent, as they contain individually identifiable visual information.
Additionally, recording face videos for database construction is costly and
time-consuming, further complicating research efforts. Moreover, heart rate
monitoring for special patients like infants and burn victims, characterized
by complex movements and skin constraints, faces challenges due to limited
relevant databases. Thus, exploring alternative body parts for heart rate
measurement is imperative to address these existing issues [25].

###  III-B HR-RST

To mitigate the impact of head movements and facilitate robust heart rate
signal extraction, we conceptualize remote heart rate measurement as analyzing
the spatiotemporal characteristics of optical flow signals in the video. The
proposed framework, named HR-RST, is depicted in the flow chart shown in Fig.
2. It comprises three phases: ROI extraction, signal analysis, and HR
calculation. The first phase aims to extract reliable skin ROI, the second
phase involves signal analysis based on each point within the ROI synthetic
video, and the third phase calculates the heart rate based on the signal
analysis results from the second phase.

###  III-C ROI Extraction

To address the issues outlined in Section III-A, this work utilizes CCNN for
ROI extraction to tackle problems of face detection failure and to accurately
extract ROI, thereby facilitating subsequent signal extraction tasks. The
proposal of CCNN stems from recent advancements in neurodynamics. [26]
demonstrates that all pulse-coupled neural network (PCNN) models fail to
explain the chaotic behavior of biological neurons stimulated by periodic
signals. The analysis centers on the refractory period induced by neuron
hyperpolarization, resulting in fluctuations in the dynamic threshold of the
PCNN model. These fluctuations lead to differences in the dynamic
characteristics of PCNN neurons compared to real neurons. The firing of action
potentials in neuron clusters is assumed to follow a Gaussian process under
specific membrane voltage. Under this assumption, the expression of the visual
cortex neural network under the floating threshold is derived, called CCNN:

$\left\\{{\begin{array}[]{*{20}{l}}{{F_{ij}}(n)={e^{-{\alpha_{f}}}}{F_{ij}}(n-1)+{V_{F}}{M_{ijkl}}{Y_{kl}}(n-1)+{S_{ij}}}\\\
{{L_{ij}}(n)={e^{-{\alpha_{l}}}}{L_{ij}}(n-1)+{V_{L}}{W_{ijkl}}{Y_{kl}}(n-1)}\\\
{{U_{ij}}(n)={F_{ij}}(n)\left({1+\beta{L_{ij}}(n)}\right)}\\\
{{Y_{ij}}(n)=\frac{1}{{1+{e^{-\left({{U_{ij}}(n)-{E_{ij}}(n)}\right)}}}}}\\\
{{E_{ij}}(n)={e^{-{\alpha_{e}}}}{E_{ij}}(n-1)+{V_{E}}{Y_{ij}}(n-1)}\end{array}}\right.$
(1)

CCNN consists of five main parts: dendritic coupling connection ${L_{ij}}(n)$,
feedforward input ${F_{ij}}(n)$, modulation product ${U_{ij}}(n)$, dynamic
threshold ${E_{ij}}(n)$, and continuous output ${Y_{ij}}(n)$. ${S_{ij}}$ is
the external input received by the receptive field. Parameters ${\alpha_{f}}$,
${\alpha_{l}}$ and ${\alpha_{e}}$ are exponential decay factors that record
the previous input state. ${V_{F}}$ and ${V_{L}}$ are weighting factors
regulating the action potential of peripheral neurons. In addition,
${M_{ijkl}}$ and ${W_{ijkl}}$ are the weights of input and connecting synapses
respectively, $\beta$ representing the connection strength, which directly
determines the coupling connection ${L_{ij}}(n)$ in the modulation product
${U_{ij}}(n)$. CCNN structure is shown in Fig. 3. CCNN is a mean-field model,
where the Heaviside function of PCNN output ${Y}$ is substituted with the
sigmoid function. This substitution facilitates the training process,
particularly for standard backpropagation. In fact, CCNN has demonstrated
superior performance compared to the most advanced visual cortex model in
image segmentation tasks [27].

Figure 3: Structural diagram of CCNN.

The characteristic of CCNN is to encode changing pixels into non-periodic
chaotic signals and static pixels into periodic signals. Skin pixels exhibit
changes through subtle color variations, while background regions remain
relatively static. The recognition of moving target objects is performed by
distinguishing the dynamic states corresponding to neuron clusters in the
video.

In the proposed framework, as depicted in Fig. 4, the ROI extraction phase
comprises three steps. Firstly, we convert the color space of the original
video from RGB to YIQ. Next, the CCNN model takes the time-varying I channel
signal of each pixel in a continuous frame sequence of length three as input.
The output is a time series of length three corresponding to the three-frame
sequence, representing the pixel encoding result of the CCNN. Subsequently,
the output signal from the CCNN neuron is processed by the continuous wavelet
transform, and the frequency characteristics of the output signal are analyzed
to distinguish between periodically changing pixels (skin area) and constant
pixels (environment area), ultimately obtaining the ROI.

Figure 4: Flow chart of ROI extraction.

In the continuous wavelet transform, we tried various continuous wavelets.
Based on the experimental results, we selected the Gaussian wavelets. Gaussian
wavelet function is a waveform function with local properties, and its
definition is based on the Gaussian function. The Gaussian function is a
continuously differentiable function with the following form:

$\psi(t)=\frac{1}{{\sqrt{2\pi}\sigma}}{e^{-\frac{{{t^{2}}}}{{2{\sigma^{2}}}}}}$
(2)

where $t$ is the input signal, $\psi(\bullet)$ is the Gaussian function, and
$\sigma$ is the standard deviation, which is used to control the width of the
Gaussian function.

The Gaussian wavelet function is obtained by translation and scale
transformation of the Gaussian function. Its definition is as follows:

${\psi_{a,b}}(t)=\frac{1}{{\sqrt{a}}}\psi\left({\frac{{t-b}}{a}}\right)=\frac{1}{{\sqrt{2\pi
a}\sigma}}{e^{-\frac{{{{(t-b)}^{2}}}}{{2{a^{2}}{\sigma^{2}}}}}}$ (3)

among them, $t$ is the input signal, ${\psi_{a,b}}(\bullet)$ is the Gaussian
wavelet function, and $a$ is the scale parameter, which controls the scale of
the wavelet function, and $b$ is the translation parameter, which controls the
position of the wavelet function.

The Gaussian wavelet function operates in two main steps: decomposition and
reconstruction. In the decomposition stage, the input signal or image
undergoes wavelet transformation to generate wavelet coefficients across
different scales and frequencies. This involves performing scale and frequency
transformations on the coefficient matrix. Subsequently, in the reconstruction
phase, the decomposed wavelet components are reconstructed back into the
original signal or image. This is achieved by reversing the scale and
frequency transformations and then applying inverse wavelet transformation.
This process enables the input signal or image to be decomposed into wavelet
components and subsequently reconstructed. Based on experimental findings, we
opted for three scales.

According to the resulting ROI, the non-skin pixels in the first original
frame among the three frames are set to be zero (a black pixel) while the RGB
values of the remaining pixels are retained. Finally, the ROI synthetic video
is obtained.

###  III-D Signal Time-frequency Analysis

In the phase, we perform signal time-frequency analysis for all pixel points
of ROI synthetic video. The phase consists of three steps, as shown in Fig. 5.

Figure 5: Flow chart of signal time-frequency analysis.

We extract the RGB temporal signal from each pixel point of the ROI synthetic
video, and then separate the green channel signal from the RGB temporal
signal. Verkruysse [28] highlighted that the green channel carries the most
pertinent physiological information. This observation aligns with the fact
that oxyhemoglobin exhibits greater absorption of green light compared to red
light. Furthermore, green light’s ability to penetrate deeper layers of the
skin, making it suitable for blood vessel detection, adds to its relevance.
Subsequently, we designed a third-order Butterworth band-pass filter to
eliminate signals outside the desired frequency range. The band-pass frequency
was defined as [0.7Hz, 2.5 Hz], corresponding to the typical heart rate range
of [42bpm, 150bpm]. The filtered signal is used for heart rate measurement.

###  III-E Heart Rate Measurement

Following the filtering process, we employed power spectral density (PSD)
estimation to calculate the heart rate value. Similar to Li [18], we utilized
the Welch method [29], which is an enhanced approach to classical power
spectrum density estimation. This method overcomes the shortcomings of
unsmooth power spectrum and large mean square error (MSE) obtained by
periodogram spectrum estimation. The specific PSD process of each pixel
point’s filtered signal is as follows:

The filtered signal of a pixel point is $x(n)$, $n$ denotes the time of the
signal, and $n=0,1,\cdots,N-1$, then $x(n)$ is divided with the time length of
$N$ into $L$ segments, each segment has $M$ data (i.e., $N=L\times M$), and
the data of the $i$-th segment is expressed as:

${x_{i}}(n)=x(n+iM-M),0\leq n\leq M,1\leq i\leq L$ (4)

then apply the window function $w(n)$ to each data segment, $w(n)$ is the
window function selected according to the task, and calculate the periodogram
of each segment. The periodogram of the $i$-th segment is:

${I_{i}}(\omega)=\frac{1}{U}{\left|{\mathop{\sum}\limits_{n=0}^{M-1}{x_{i}}(n)w(n){e^{-j\frac{{2\pi}}{M}in}}}\right|^{2}},i=1,2\ldots,M-1$
(5)

where $\omega$ is the signal frequency, $j={(-1)^{1/2}}$, and $U$ is called
the normalization factor,

$U=\frac{1}{M}\mathop{\sum}\limits_{n=0}^{M-1}{w^{2}}(n)$ (6)

the periodogram of each segment is approximately regarded as uncorrelated, and
the final power spectrum is estimated as:

${P_{xx}}\left({{e^{j\omega}}}\right)=\frac{1}{L}\mathop{\sum}\limits_{i=1}^{L}{I_{i}}(\omega)$
(7)

where ${P_{xx}}(\bullet)$ is the final PSD estimation. The frequency with the
maximum power spectral density is used as the frequency of heart rate. The
frequency multiplied by 60 is the beats per minute (bpm):

$H{R_{es}}=f[{\mathop{\rm
argmax}\nolimits}({P_{xx}}\left({{e^{j\omega}}}\right))]{\kern 1.0pt}{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}\times{\kern
1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern 1.0pt}{\kern
1.0pt}60{\rm{}}({\rm{bpm}})$ (8)

$H{R_{es}}$ is the estimated heart rate value, and $f(\bullet)$ is the heart
rate frequency.

Following heart rate computation from all ROI synthetic video points,
histogram analysis in Fig. 6A illustrates their distribution. Except for
values near zero, representing environmental background, most points cluster
around the ground truth heart rate of 69 bpm. Additionally, Fig. 6B presents a
thermographic representation of heart rate distribution, with points related
to the environment displaying higher errors compared to those aligned with
true values, mainly within the skin region, showing lower errors.
Consequently, we adopt the mode value among these values as the final output
result.

Figure 6: Statistical analysis of heart rate distribution.

##  IV EXPERIMENT

To evaluate the effectiveness of HR-RST, we performed two experiments: (a)
experiments involving data collected from various body parts collected by
ourselves, and (b) heart rate measurement experiments using the VIPL-HR
database.

###  IV-A Databases and evaluation indicators

VIPL-HR [30] is a comprehensive database established by the VIPL Research
Group at the Institute of Computing Technology, Chinese Academy of Sciences,
comprising data from 107 subjects across 9 distinct scenarios. These scenarios
encompass varied conditions such as static scenes, lighting changes, and head
movements, resulting in 2378 RGB videos captured using 4 different devices.
Ground truth heart rate measurements synchronized with the recordings are
included. To ensure fair comparisons with prior methods, we calculated the
average heart rate of each video and assessed measurement error alongside
statistical metrics like standard deviation (SD), mean absolute error (MAE),
and root mean square error (RMSE). For experiments involving non-facial body
parts, we conducted data collection due to the absence of relevant research
and public databases.

###  IV-B Experiments on Other Body Parts

We utilized CCNN to process every three consecutive frames of the original
video frames. Fig. 7 depicts the processing of two points in the initial three
frames: one on the palm and the other in the background. This involved
extracting the I-channel signal, obtaining a time series via CCNN, and
performing wavelet transform on the time series. Discrimination of ROI points
relied on the sum of real parts of the wavelet transform results: if the sum
exceeded zero, the pixel was classified as an ROI point; otherwise, it was
considered non-ROI. For the first frame, non-ROI pixel values were set to
grayscale zero, while RGB values remained for the rest. This process was
applied to all frames, yielding the final ROI synthetic video. In addressing
subject privacy, we conducted remote heart rate experiments on six non-facial
body parts, reducing individual identifiable information. These parts included
the palm, back of the hand, forearm, upper arm, back, and sole. Fig. 8
displays the results of ROI extraction for these body parts.

Figure 7: The discrimination process of HR-RST for ROI points on the palm and
background points. Figure 8: ROI extraction of other body parts, including (a)
palm and its extraction results; (b) back of hand and its extraction results;
(c) forearm and its extraction results; (d) upper arm and its extraction
results; (e) back and its extraction results; (f) sole and its extraction
results.

In the above results, except for only a small amount of skin area extracted
from the forearm and sole, relatively accurate ROIs were obtained in the other
four body parts. These images will be combined into ROI synthetic videos for
heart rate measurement. We argue that CCNN shows immense for potential ROI
extraction capabilities. Table I shows the heart rate measurement results
obtained using these body parts. Among them, $H{R_{gt}}$ is the ground truth
value of the heart rate, $H{R_{es}}$ is the heart rate estimate value, and
$H{R_{er}}$ is the heart rate error value, that is,
$H{R_{er}}=H{R_{es}}-H{R_{gt}}$.

TABLE I: Heart rate measurement results of six body parts ($H{R_{gt}}$=65 bpm) Body site |  Palm |  Back of hand |  Forearm |  Upper arm |  Back |  Sole
---|---|---|---|---|---|---
$H{R_{es}}$ (bpm) |  43 |  30 |  43 |  43 |  43 |  43
$H{R_{er}}$ (bpm) |  -22 |  -35 |  -22 |  -22 |  -22 |  -22

HR-RST extracts heart rate from unconventional non-facial areas like the back
of the palm, showcasing the potential of various body parts besides the face
for physiological measurement. This not only addresses the increasing concern
for privacy protection but also lays the groundwork for more advanced
physiological measurements in special clinical scenarios, such as infant
monitoring and care for burn victims.

###  IV-C Evaluation Experiment on VIPL-HR

We fed each set of three original video frames into CCNN for processing. Fig.
9 showcases the processing of two points on the forehead and background in the
initial three frames, involving I-channel signal extraction, time sequence
acquisition from CCNN, and wavelet transform result depiction. The ROI frame
was determined based on the sum of real parts of the wavelet transform: if the
sum exceeded zero, it was considered an ROI point; otherwise, a non-ROI point.
Non-ROI pixel values in the first frame were set to zero, while RGB values of
the remaining pixels were retained. This process was applied to all frames,
yielding the final ROI synthetic video. Experimental results in Fig. 10
demonstrate that compared to ROI extraction by the face detector, CCNN-
extracted ROIs comprise only facial and neck skin, excluding environmental
areas that may affect pulse signal extraction. Moreover, the model effectively
tracks subject head movements and accurately locates skin edge pixels,
segregating accessories like glasses and necklaces while capturing facial
expressions like smiling with an open mouth. Crucially, the proposed method
overcomes previous limitations in recognizing faces during head tilts due to
pitch and yaw rotation.

Figure 9: The discrimination process of HR-RST for ROI points on the forehead
and background points. Figure 10: Example of the result map of HR-RST
recognition of skin area, including ten subjects, (a) p1 and its extraction
results; (b) p2 and its extraction results; (c) p3 and its extraction results;
(d) p4 and its extraction results; (e) p5 and its extraction results; (f) p6
and its extraction results; (g) p7 and its extraction results; (h) p8 and its
extraction results; (i) p9 and its extraction results; (j) p10 and its
extraction results.

In the VIPL-HR database, we conducted experiments using all RGB videos from
ten subjects labeled as p1 to p10. To preserve physiological information
without loss due to video compression, we utilized the original videos from
the database. HR-RST obviates the need for complex image pre-processing and
training procedures, as it directly computes heart rates following the
acquisition of the ROI synthetic video. The results of these measurements for
the ten samples are presented in Table II.

TABLE II: HR estimation results of HR-RST on VIPL-HR database Sample |  SD↓ |  MAE↓ |  RMSE↓
---|---|---|---
p1 |  13.29 |  12.41 |  13.74
p2 |  12.69 |  26.41 |  29.18
p3 |  10.53 |  16.22 |  18.56
p4 |  17.63 |  15.05 |  19.54
p5 |  15.99 |  27.14 |  29.37
p6 |  13.85 |  13.00 |  16.17
p7 |  6.37 |  30.17 |  30.81
p8 |  7.38 |  32.17 |  32.97
p9 |  13.03 |  24.48 |  27.60
p10 |  14.56 |  19.22 |  21.36

From the experimental results, it’s evident that HR-RST consistently produces
comparable metrics across ten samples, indicating the robustness of our method
to different sample variations. Based on this, we contend that our framework
can overcome the challenges posed by complex scenarios in the VIPL-HR database
and yield reliable remote heart rate measurement results.

##  V CONCLUSION

Heart rate measurement using commonly available cameras is a welcome
technological advance. Although the popular deep learning methods have
achieved great success in the field, they encounter problems in real-world
applications such as the scarcity of large databases and the difficulty of
high cost of collecting data. Different from existing methods, this work
applies chaos theory to computer vision tasks for the first time, we have
proposed a robust motion-aware framework. It overcomes the problem that other
face detection methods cannot detect faces when the head moves significantly,
and it has strong interpretability. Concluding from the experimental results,
there is a gap between the indicators of the framework and the best deep
learning methods. Considering the dependence of deep learning methods on data,
the results obtained in the work are promising. We anticipate the work to be a
starting point for more sophisticated physiological measurement in special
clinical scenarios such as infant monitoring and burn victims.

## Acknowledgment

This work is supported by the Natural Science Foundation of Gansu Province
(Grants 21JR7RA510, 21JR7RA345, 22JR5RA543).

Some experiments are supported by the Supercomputing Center of Lanzhou
University.

Conflict of Interest: The authors declare that they have no conflicts of
interest.

## References

  * [1] T. A. Manzone, H. Q. Dam, D. Soltis, and V. V. Sagar, “Blood volume analysis: a new technique and new clinical interest reinvigorate a classic study,” _Journal of nuclear medicine technology_ , vol. 35, no. 2, pp. 55–63, 2007\.
  * [2] E. Lee, E. Chen, and C.-Y. Lee, “Meta-rppg: Remote heart rate estimation using a transductive meta-learner,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16_. Springer, 2020, pp. 392–409.
  * [3] H. Lu, H. Han, and S. K. Zhou, “Dual-gan: Joint bvp and noise modeling for remote physiological measurement,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 12 404–12 413.
  * [4] Z. Yu, Y. Shen, J. Shi, H. Zhao, P. H. Torr, and G. Zhao, “Physformer: Facial video-based physiological measurement with temporal difference transformer,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 4186–4196.
  * [5] H. Zhou, Y. Yuan, and C. Shi, “Object tracking using sift features and mean shift,” _Computer vision and image understanding_ , vol. 113, no. 3, pp. 345–352, 2009.
  * [6] V. Mirjalili, S. Raschka, and A. Ross, “Privacynet: Semi-adversarial networks for multi-attribute face privacy,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 9400–9412, 2020.
  * [7] J. Kranjec, S. Beguš, J. Drnovšek, and G. Geršak, “Novel methods for noncontact heart rate measurement: A feasibility study,” _IEEE transactions on instrumentation and measurement_ , vol. 63, no. 4, pp. 838–847, 2013.
  * [8] K. V. Madhav, M. R. Ram, E. H. Krishna, N. R. Komalla, and K. A. Reddy, “Robust extraction of respiratory activity from ppg signals using modified mspca,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 62, no. 5, pp. 1094–1106, 2013.
  * [9] S. Nabavi and S. Bhadra, “A robust fusion method for motion artifacts reduction in photoplethysmography signal,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 69, no. 12, pp. 9599–9608, 2020.
  * [10] N. Mora, F. Cocconcelli, G. Matrella, and P. Ciampolini, “Fully automated annotation of seismocardiogram for noninvasive vital sign measurements,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 69, no. 4, pp. 1241–1250, 2019.
  * [11] S. Chaichulee, M. Villarroel, J. Jorge, C. Arteta, G. Green, K. McCormick, A. Zisserman, and L. Tarassenko, “Multi-task convolutional neural network for patient detection and skin segmentation in continuous non-contact vital sign monitoring,” in _2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017)_. IEEE, 2017, pp. 266–272.
  * [12] A. Ni, A. Azarang, and N. Kehtarnavaz, “A review of deep learning-based contactless heart rate measurement methods,” _Sensors_ , vol. 21, no. 11, p. 3719, 2021.
  * [13] M. Lewandowska, J. Rumiński, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam—a non-contact method for evaluating cardiac activity,” in _2011 federated conference on computer science and information systems (FedCSIS)_. IEEE, 2011, pp. 405–410.
  * [14] M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” _IEEE transactions on biomedical engineering_ , vol. 58, no. 1, pp. 7–11, 2010.
  * [15] R. Macwan, Y. Benezeth, and A. Mansouri, “Heart rate estimation using remote photoplethysmography with multi-objective optimization,” _Biomedical Signal Processing and Control_ , vol. 49, pp. 24–33, 2019.
  * [16] G. De Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” _IEEE Transactions on Biomedical Engineering_ , vol. 60, no. 10, pp. 2878–2886, 2013.
  * [17] W. Wang, A. C. Den Brinker, S. Stuijk, and G. De Haan, “Algorithmic principles of remote ppg,” _IEEE Transactions on Biomedical Engineering_ , vol. 64, no. 7, pp. 1479–1491, 2016.
  * [18] X. Li, J. Chen, G. Zhao, and M. Pietikainen, “Remote heart rate measurement from face videos under realistic situations,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 4264–4271.
  * [19] S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2396–2404.
  * [20] G.-S. Hsu, A. Ambikapathi, and M.-S. Chen, “Deep learning with time-frequency representation for pulse estimation from facial videos,” in _2017 IEEE international joint conference on biometrics (IJCB)_. IEEE, 2017, pp. 383–389.
  * [21] X. Niu, Z. Yu, H. Han, X. Li, S. Shan, and G. Zhao, “Video-based remote physiological measurement via cross-verified feature disentangling,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_. Springer, 2020, pp. 295–310.
  * [22] C.-J. Hsieh, W.-H. Chung, and C.-T. Hsu, “Augmentation of rppg benchmark datasets: Learning to remove and embed rppg signals via double cycle consistent learning from unpaired facial videos,” in _European Conference on Computer Vision_. Springer, 2022, pp. 372–387.
  * [23] A. Zadeh, Y. Chong Lim, T. Baltrusaitis, and L.-P. Morency, “Convolutional experts constrained local model for 3d facial landmark detection,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 2519–2528.
  * [24] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE signal processing letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.
  * [25] L. Niu, J. Speth, N. Vance, B. Sporrer, A. Czajka, and P. Flynn, “Full-body cardiovascular sensing with remote photoplethysmography,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 5993–6003.
  * [26] J. Liu, J. Lian, J. C. Sprott, Q. Liu, and Y. Ma, “The butterfly effect in primary visual cortex,” _IEEE Transactions on Computers_ , vol. 71, no. 11, pp. 2803–2815, 2022.
  * [27] Z. Yi, J. Lian, Q. Liu, H. Zhu, D. Liang, and J. Liu, “Learning rules in spiking neural networks: A survey,” _Neurocomputing_ , vol. 531, pp. 163–179, 2023.
  * [28] W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmographic imaging using ambient light.” _Optics express_ , vol. 16, no. 26, pp. 21 434–21 445, 2008.
  * [29] P. Welch, “The use of fast fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms,” _IEEE Transactions on audio and electroacoustics_ , vol. 15, no. 2, pp. 70–73, 1967.
  * [30] X. Niu, S. Shan, H. Han, and X. Chen, “Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 2409–2423, 2019.


# Measurement based FHSS–type Drone Controller Detection at 2.4GHz: An STFT
Approach ††thanks: This paper has been accepted for the presentation in the
2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring).

Batuhan Kaplan12, İbrahim Kahraman13, Ali Görçin14, Hakan Ali Çırpan2, Ali
Rıza Ekti15  1 Informatics and Information Security Research Center (BİLGEM),
TÜBİTAK, Kocaeli, Turkey  2 Department of Electronics and Communication
Engineering, İstanbul Technical University, İstanbul, Turkey  3 Department of
Electrical and Electronics Engineering, Boğaziçi University, İstanbul, Turkey
4 Faculty of Electronics and Communications Engineering, Yıldız Technical
University, İstanbul, Turkey  5 Department of Electrical and Electronics
Engineering, Balıkesir University, Balıkesir, Turkey
Emails: batuhan.kaplan<EMAIL_ADDRESS>
agorcin@yildiz.edu.tr,hakan.cirpan<EMAIL_ADDRESS>
###### Abstract

The applications of the unmanned aerial vehicles increase rapidly in everyday
life, thus detecting the UAVs and/or its pilot is a crucial task. Many UAVs
adopt frequency hopping spread spectrum (FHSS) technology to efficiently and
securely communicate with their radio controllers where the signal follows a
hopping pattern to prevent harmful interference. In order to realistically
distinguish the frequency hopping (FH) RC signals, one should consider the
real–world radio propagation environment since many UAVs communicate with RCs
from a far distance in which signal faces both slow and fast fading
phenomenons. Therefore, in this study different from the literature, we
consider a system that works under real–conditions by capturing over–the–air
signals at hilly terrain suburban environments in the presence of foliages. We
adopt the short–time Fourier transform (STFT) approach to capture the hopping
sequence of each signal. Furthermore, time guards associated with each hopping
sequence are calculated using the autocorrelation function (ACF) of the STFT
which results in differentiating the each UAV RC signal accurately. In order
to validate the performance of the proposed method, the results of normalized
mean square error (MSE) respect to different signal–to–noise ratio (SNR),
window size and Tx–Rx separation values are given.

###### Index Terms:

short–time Fourier transform, frequency hopping, UAV remote controller
detection

##  I Introduction

Unmanned aerial vehicles have become a prevalent part of the daily life with
their applications to many fields such as mapping and surveying,
transportation, surveillance, law enforcement, aerial imaging and agriculture
[1]. Besides the aforementioned use of UAVs in many areas, one should keep in
mind that UAVs can also be used dangerously to create unwanted incidents
especially when they are diverted to the sensitive airspace near airports and
their presence may cause accidents which can result in fatal crashes [2].
Moreover, UAVs can be utilized for collecting information about people,
organisations, and companies without their consent. Therefore, identification
of UAV systems and their communication are great importance, especially to
prevent unwanted situations. In this context, it is known that most of the
communication between the UAVs and wireless RC utilize the spread spectrum
technology of FHSS on industrial, scientific, and medical (ISM) band at 2.4GHz
[3]. Therefore a method to detect and classify these kinds of signals in this
band would lead to the identification of the communication between the UAV and
the controller.

In literature, most well known FHSS signal detection methods adopt the
time–frequency analysis and wavelet analysis [4, 5, 6, 7]. It is shown that
Wigner Ville distribution, Wavelet transform method, and array signal
processing methods can be used to identify FHSS signals with the burden of
heavy computational complexity which make them hard to satisfy the real–time
implementation. Since STFT based detection method does not need any prior
information about received signal compared to computationally complex
algorithms, STFT becomes the designated optimum detector in the absence of
information regarding the received signal [8]. However, one should note that
in order to achieve good results with STFT, SNR value needs to be high. Thus,
in this study, we propose STFT based blind signal detection method for FHSS
UAV RC signals by using over–the–air signal measurements which accounts for
the signal imperfections present in the nature of real–world conditions (e.g.
fading, multipath, and so on) by considering the hilly terrain suburban
environments in the presence of foliage. Furthermore, the literature utilizes
the simulated data instead of over–the–air signals in general and these
simulations assume that there is no time guards between hops. This assumption
makes differentiation of FH signals easier, however, many hopping signals use
time guards and also these time guards are different for different signal
sources. The proposed approach is also able to detect the time guards between
each hopping sequence using the ACF [9] of the STFT which results in
differentiating the each drone RC signal accurately.

The paper is organized as follows, Section II details the system model.
Section III presents the proposed method. The measurement setup is explained
in Section IV. In Section V, measurement results are given and discussed.
Finally, Section VI concludes the paper.

##  II Signal Model

Drone controller signals might be FH signals that have temporal statistical
characteristics and they can be written as [6],

$x(t)=s(t)\sum_{m=0}^{M-1}e^{j2\pi f_{c_{m}}t_{m}+\theta_{m}}$ (1)

where $\theta_{m}$ and $f_{c_{m}}$ are the carrier phase and carrier frequency
of $m_{th}$ hop, respectively. Also, $s(t)$ denotes the complex baseband
equivalent of the information bearer for $t\in[0,T]$, $M$ stands for the total
number of hop of a signal, $t_{m}$ is the duration time of $m_{th}$ hop that
might be a uniform distribution or not.

The received controller signal which is complex baseband equivalent of the
received passband signal and can be expressed as,

$r(t)=\sum_{n=0}^{N-1}y_{n}(t)+n(t)+I(t)$ (2)

where $y_{n}(t)$ is an $n_{th}$ FH signal source, $n(t)$ denotes the complex
Additive White Gaussian Noise (AWGN) in which I and Q components are i.i.d
with $\mathcal{N}(0,\,\sigma^{2})$, $I(t)$ stands for the interference signal.

###  II-A Short–Time Fourier Transform

STFT approach is utilized to analyze the FH signals as a method to observe the
frequency content of this type of non–stationary signals over time.
Mathematical expression of STFT of the time-domain signal $z(t)$ can be
written as [8],

$STFT\Big{\\{}z(t)\Big{\\}}=\int_{-\infty}^{\infty}[z(t)w(t-\tau)]e^{-j2\pi
f\tau}d\tau$ (3)

where $w(t)$ is the window function. The STFT matrix
$S=[s_{1}[f],s_{2}[f],...,s_{K}[f]]$ such that $i_{th}$ element of this matrix
is a column vector determined by discrete Fourier transform of $r[n]w[n-iR]$,

$s_{i}[f]=\sum_{n=0}^{N-1}r[n]w[n-iR]e^{-j2\pi fn}$ (4)

where $r[n]$ is sampled version of $r(t)$ by considering the anti–aliasing
property and $R$ denotes the shifting length.

One should keep in mind that adjusting the time and frequency resolution is
crucial point for STFT analysis [10] due to the trade off between them. The
length of the time point can be calculated as [11]

$m=\left\lfloor\frac{N_{x}-L}{M-L}\right\rfloor$ (5)

where $N_{x}$ is length of the signal, $L$ denotes the number of overlap in
the Fourier transform, $M$ represents the window size, $\lfloor\cdot\rfloor$
stands for the floor operator.

##  III Proposed Method

The received controller signal, $r(t)$, analyzed by STFT method. As depicted
in Fig. 1, the flow graph explains how the system works in brief. After the
signal is received in the first stage, optimal window time length is decided
to get the optimum resolution at (5) based on maximizing the number of
elements on the matrix $S$ in the second stage of the flowchart. STFT is
calculated in a dBm unit according to power spectral density (PSD) in the same
step.

Figure 1: The flowchart of the proposed detection method.

A binarization process is conducted at the following steps of the flowchart;
the STFT matrix is converted to a binarized matrix, $Z(k,l)$, with the
utilization of a threshold $\mu$. Also, in the flowchart PSD refers to each
point in the STFT matrix. Based on the dynamically calculated threshold value,
whether the signal presents or not is decided. Therefore, the problem
statement can be indicated as the identification of the presence of the
unknown FH signal. When the signal is present, $Z(k,l)$ is evaluated as 1 and
the new binarized matrix, $Z(k,l)$, is given as,

$Z(k,l)=\begin{cases}1,&S(k,l)\geq\mu\\\ 0,&S(k,l)<\mu\end{cases}$ (6)

To determine the threshold value, each element of STFT matrix are concatenated
and a sorting algorithm implemented to list the power levels of each point of
the STFT matrix in a ascending order. Then, we assume that the majority of the
received signal comprise of noise, therefore, we take the mean value of the
top $20\%$ of the sorted values of STFT matrix to determine a lower bound for
the computation of the threshold. Thus, the threshold is calculated as

$\mu=\frac{S_{max}+\sigma_{20\%}}{2}$ (7)

where $S_{max}$ is the maximum value of the STFT matrix, $\sigma_{20\%}$
denotes the mean of the top $20\%$ samples. Fig. 2 shows $S_{max}$,
$\sigma_{20\%}$, and the threshold value. Please note that even for the very
low SNR regimes, the measurement results indicate the feasibility of this
threshold selection process.

Figure 2: Estimated threshold representation with $S_{max}$ and
$\sigma_{20\%}$ values over real recorded signal.

Due to the wireless impairments on the received signal, some portions of the
$Z$ matrix might be missing as shown in Fig. 3(a). In order to represent the
signal in a more plausible way, we adopt a widely used morphological dilation
and erosion processes from the domain of image processing [12, 13] to recover
the received signal properly. A signal with impairments and the output of
dilation and erosion processes are shown in Fig. 3.

(a) Recorded FH drone controller signal with channel impairments.

(b) FH drone controller signal after correction.

Figure 3: Result of dilation process.

After the recovery process, it becomes possible to extract the signal
parameters such as start time, stop time, center frequency and difference
between start time and stop time (dwell time) accurately. The parameter
estimation algorithm is given in Algorithm 1. This process is implemented to
the $Z$ matrix and the parameters of each signal in the spectrum are
extracted. The algorithm simply detects the signals from $Z$ and extracts the
duration of each independent transmission which can be part of same or
different signal sources.

Input: $Z(k,l)$

Output: start time, stop time, center frequency, dwell time, bandwidth

1

2for  _$i\leftarrow 1$ to row_ do

3 for  _$j\leftarrow 1$ to column_ do

4 $count\leftarrow 0$

5 $bandwidth\leftarrow 0$

6 if  _$Z(i,j)$ == to $1$_ then

7 start time $\leftarrow j$

8 $f_{start}\leftarrow i$

9 while  _$Z(i,j)$ $==$ $1$ _ do

10 $count\leftarrow count+1$

11 $j\leftarrow j+1$

12 while  _$Z(i,j-1)$ $==$ $1$ _ do

13 $bandwidth\leftarrow bandwidth+1$ $i\leftarrow i+1$

14 stop time $\leftarrow j-1$

15 $f_{stop}\leftarrow i-1$

16 center frequency $\leftarrow\frac{f_{start}+f_{stop}}{2}$

17 dwell time $\leftarrow count$

18

19 Assign $0$ to rectangle that founded above.

20

21

Algorithm 1 Parameter Extraction Algorithm

In the last step of the flowchart, controlling for each hop is done to decide
whether it belongs to FH signal or not. Due to the signal structure which will
be explained in Section IV, time guards are inserted during hopping and this
makes classification complicated. Time guards may lead to miss matches between
hops because there can be some other signals between them and these miss
matches should be corrected. In this paper ACF is utilized for this purpose
since the highest correlation peaks occur as the signal matches itself
perfectly and this is the case for drone controller FH signals based on the
fact that time and frequency characteristics are fixed. ACF is used on $Z$
matrix and result of ACF gives the highest peak at $T_{1}$, which is the
fundamental period (6.8ms) for the particular drone controller FH signal. The
rest of the peaks give information in regards to hops which are generated from
the same source. Thus, we define a set T = $\\{T_{1},T_{2},...T_{n}\\}$ which
represents the locations of all the peak values where $T_{1}=\sup\hskip
2.84526pt\text{T}$. In other words, we put all the local extremum for interval
$(0,T_{1})$ to the set T as shown in Fig. 4 meaning that we obtained all the
required T values (guard and dwell times) to track and classify hops. Finally,
the signal classification block in the flowchart is executed by the procedure:
$\exists T_{i}\in\text{T}$ and any two hops, $hop\\_a$ and $hop\\_b$, are
emitted from the same signal source if

$\text{start time($hop\\_a$)}-\text{start time($hop\\_b$)}\equiv T_{i}\hskip
2.84526pt(\text{mod }T_{1})$ (8)

where start time is the estimated parameter from Algorithm 1. Thus, hop
separation is also possible with the proposed method. Please note that even
though the method is applied to a particular set of FH signals, it can be
utilized to detect any kinds of FH signal with observable time and frequency
domain hopping pattern.

Figure 4: Time difference estimation between hops utilizing auto correlation
function.

##  IV Measurement Setup

Experimental setup for FH signal parameter estimation is performed in
Scientific and Technological Research Council of Turkey (TÜBİTAK) Informatics
and Information Security Research Center (BİLGEM). Data collection of FH
signal is conducted with different distances from 5m to 135m with 10m
intervals between each step. Fig. 5 shows the fixed location of our receiver
and locations of the drone controller (transmitter).

Figure 5: The aerial map of data collection locations.

###  IV-A Hardware Setup

The testbed used in the data acquisition procedure consists of signal source
(drone controller) and spectrum analyzer to record the signals.

Figure 6: The hopping pattern of Drone controller signal: Futaba T8J RC as
signal source.

Futaba T8J RC is used during the experiment as a FH signal source. Futaba T8J
RC operates in the 2.4 gigahertz (GHz) ISM band over the half of the spectrum
band (up to 2.45 GHz). When analyzing the signal, it is discovered that the RC
transmitter is behaved differently compared to the standard FH communication
systems (e.g. Bluetooth). An illustration of the periodic hopping sequence for
the Futaba T8J RC is shown in Fig. 6. Also, in the figure for $\tau_{dwell}$
represents dwell time and fundamental period becomes

$\Delta t_{1}<\Delta t_{2}<\Delta t_{3}$ (9) $3\tau_{dwell}+\Delta
t_{1}+\Delta t_{2}+\Delta t_{3}=T_{1}=0.0068sec$ (10)

TABLE I: FH signal characteristics  FH Signal |
---|---
Dwell Time | 1.44ms
Center Frequency Set |  | 2.4GHz-2.45GHz interval
---
Hopping Sequence | f1 f1 f2 f3 f3 f4 …

The parameters of Futaba T8J RC FH signal source are listed in Table I. In the
receiver side, Rohde&Schwarz FSW 26 signal and spectrum analyzer (SSA) is
utilized to record FH signals. SSA can support the frequency range from 2
hertz (Hz) to 26.5GHz. The device provides real–time spectral analysis up to
160 megahertz (MHz) bandwidth. The signals are recorded over the 2.4GHz ISM
spectrum band with an omnidirectional antenna.

###  IV-B Experimental Procedures

In data collection from real–world, it is assumed that transmitter operates
between 2.4GHz–2.48GHz ISM spectrum band. The center frequency of SSA is set
to 2.44GHz and bandwidth of interest is adjusted to 80MHz for the purpose of
full coverage. Also, SSA is connected to external computer via an Ethernet
cable in favor of achieving data storage with ease. The sampling rate depends
on the analysis bandwidth of real–time spectrum which is selected as 80MS/s.
Each measurement is captured as an I/Q samples and collected 20M I/Q samples
during 250ms. However, considering the processing limit, the collected data is
divided into pieces with each has 4M samples. Each divided data was considered
when calculating the performance of the proposed method. Finally, captured I/Q
data is fed into the computer which runs MATLAB R2015b software.

##  V Measurement Results

Over–the–air data collection is realized and the performance of time–frequency
analysis method is evaluated. Please note that all the captured data includes
real–world propagation effects such as multipath fading, interference, carrier
frequency offset (CFO). Fig. 7 shows how over–the–air recorded FH signal
source behaves during the observation time. Please also note that some of the
estimated parameters of the real signal which measured at 25m distance with
5dB SNR can be found in Table II.

Figure 7: Recorded 2.4GHz spectrum that comprised of drone controller FH
signals.

In order to validate the performance of the system, normalized MSE was
considered. Normalized MSE can be calculated as[14]

$\text{NMSE}=\frac{1}{N}\sum_{i=1}^{N}\Big{(}\frac{\hat{t}_{i}-t}{t}\Big{)}^{2}$
(11)

where $\hat{t}_{i}$ is the estimated hopping time of $i_{th}$ hop and $t$
denotes the true value of hopping time. $N$ represents the total number of hop
that must be found. When calculating the error, it is assumed that $N$ equals
to $22$ in the observation time. Also, $0$ values have been added to extracted
parameters if there is some undetected signal.

TABLE II: Estimated parameters of FH signal | Start Time
---
(ms)
| Stop Time
---
(ms)
| Dwell Time
---
(ms)
| Center Frequency
---
(GHz)
1.7930 | 3.2403 | 1.4472 | 2.4271
5.1742 | 6.6086 | 1.4344 | 2.4414
6.7623 | 8.1967 | 1.4344 | 2.4414
8.6066 | 10.0410 | 1.4344 | 2.4211
11.9749 | 13.4221 | 1.4472 | 2.4039
Figure 8: Estimated hopping time errors vs. distance in terms of normalized
MSE.

The error curve of an estimated hopping time from measured data is plotted in
Fig. 8. In here, error values are determined for three different SNR values
and distances. It is clearly seen that as the distance increases, the error of
estimation increases. Also poor SNR condition adversely affects the
performance. Moreover, even with the same SNR condition, there may be a
missing hop of an FH signal in the signal received from the farther point.

As discussed before, another important issue is selection of window size
($M$). In this regard, precision of window size is also studied. It can be
seen that window size directly effects the accuracy of the estimation. While
window size decreases, we get high resolution in time but low resolution in
frequency. In contrast, increasing window size implies high resolution in
frequency but low resolution in time. Because both the frequency and time
information are main features to classify the hopping signals, there should be
some optimum window size providing better performance. It is empirically shown
in Fig. 9 that the optimum window size for STFT is $M=2048$.

Since the previous works utilize STFT only for deciding whether a signal is
hopping between different frequencies, ACF is applied only in time–domain for
the signals with no guard times, and the simulations were considered instead
of measurements, comparison of the performance of the proposed method with
these simulations are avoided in this work.

Figure 9: Estimated hopping time errors vs. distance for different window
sizes and for SNR $=$ 0 dB.

##  VI Concluding Remarks and Future Directions

In this work, an FH drone controller signal detection algorithm is proposed
and the performance of the algorithm is evaluated using measurements of UAV RC
in real–world wireless environment. The algorithm can estimate the guards of
the each hopping sequence successfully by using the ACF of STFT, which also
leads to the accurate identification whether a FH signal is present or not.
The performance of the proposed method is also quantified by normalized MSE
for over–the–air signal which are recorded at different distances and SNRs.
Measurement results show that reasonable input parameters will improve the
performance of frequency hopping signal parameter estimation. In future
studies, we will consider adopting the recently emerging deep learning
algorithms to distinguish multiple standard based wireless hopping signals.

##  VII Acknowledgement

This publication was made possible by NPRP12S-0225-190152 from the Qatar
National Research Fund (a member of The Qatar Foundation). The statements made
herein are solely the responsibility of the author[s].

## References

  * [1] DHL, “DHL launches first commercial drone ’parcelcopter’ delivery service,” Available: https://www.theguardian.com/technology/2014/sep/25/german-dhl-launches-first-commercial-drone-delivery-service, Accessed: Oct. 29, 2019.
  * [2] FAA, “Federal Aviation Administration UAS Sightings Report,” Available: https://www.faa.gov/uas/resources/public_records/uas_sightings_report/, Accessed: Oct. 29, 2019.
  * [3] P. Popovski, H. Yomo, and R. Prasad, “Strategies for adaptive frequency hopping in the unlicensed bands,” _IEEE Wireless Communications_ , vol. 13, no. 6, pp. 60–67, Dec 2006.
  * [4] A. Kanaa and A. Z. Sha’ameri, “A robust parameter estimation of fhss signals using time–frequency analysis in a non-cooperative environment,” _Physical Communication_ , vol. 26, pp. 9–20, 2018.
  * [5] B. Boashash, _Time-frequency signal analysis and processing: a comprehensive reference_. Academic Press, 2015.
  * [6] S. Wei, M. Zhang, G. Wang, X. Sun, L. Zhang, and D. Chen, “Robust multi-frame joint frequency hopping radar waveform parameters estimation under low signal-noise-ratio,” _IEEE Access_ , 2019.
  * [7] X. Zhang, X. Wang, and X.-m. Du, “Blind parameter estimation of frequency-hopping signals based on wavelet transform [j],” _Journal of Circuits and Systems_ , vol. 4, 2009.
  * [8] X. Ouyang and M. G. Amin, “Short-time fourier transform receiver for nonstationary interference excision in direct sequence spread spectrum communications,” _IEEE Trans. Signal Process._ , vol. 49, no. 4, pp. 851–863, 2001.
  * [9] C.-D. Chung and A. Polydoros, “Parameter estimation of random fh signals using autocorrelation techniques,” _IEEE Trans. Commun._ , vol. 43, no. 2/3/4, pp. 1097–1106, 1995.
  * [10] N.-K. Kim and S.-J. Oh, “Comparison of methods for parameter estimation of frequency hopping signals,” in _International Conference on Information and Communication Technology Convergence (ICTC)_ , 2017, pp. 567–569.
  * [11] J. Smith, S. U. C. for Computer Research in Music, Acoustics, and S. U. D. of Music, _Spectral Audio Signal Processing_. W3K, 2011, online book, 2011 edition. [Online]. Available: https://ccrma.stanford.edu/~jos/sasp/
  * [12] R. M. Haralick, S. R. Sternberg, and X. Zhuang, “Image analysis using mathematical morphology,” _IEEE Trans. Pattern Anal. Mach. Intell._ , no. 4, pp. 532–550, 1987.
  * [13] L. Luo _et al._ , “Detection of an unknown frequency hopping signal based on image features,” in _2nd International Congress on Image and Signal Processing_ , 2009, pp. 1–4.
  * [14] Y. Ma and Y. Yan, “Blind detection and parameter estimation of single frequency-hopping signal in complex electromagnetic environment,” in _2016 Sixth International Conference on Instrumentation & Measurement, Computer, Communication and Control (IMCCC)_, 2016, pp. 370–374.

  *[UAVs]: Unmanned aerial vehicle
  *[FHSS]: frequency hopping spread spectrum
  *[FH]: frequency hopping
  *[RC]: radio controller
  *[RCs]: radio controller
  *[STFT]: short–time Fourier transform
  *[ACF]: autocorrelation function
  *[UAV]: Unmanned aerial vehicle
  *[MSE]: mean square error
  *[SNR]: signal–to–noise ratio
  *[ISM]: industrial, scientific, and medical
  *[AWGN]: Additive White Gaussian Noise
  *[PSD]: power spectral density
  *[TÜBİTAK]: Scientific and Technological Research Council of Turkey
  *[BİLGEM]: Informatics and Information Security Research Center
  *[GHz]: gigahertz
  *[SSA]: signal and spectrum analyzer
  *[Hz]: hertz
  *[MHz]: megahertz
  *[CFO]: carrier frequency offset


11institutetext: Institute of Computer Engineering, Technische Universität
Wien (TU Wien), Austria
11email<EMAIL_ADDRESS>
22institutetext: eBrain Lab, Division of Engineering, New York University Abu
Dhabi (NYUAD), United Arab Emirates (UAE)
22email<EMAIL_ADDRESS>
# ReFit: A Framework for Refinement of Weakly Supervised Semantic Segmentation
using Object Border Fitting for Medical Images

Bharath Srinivas Prabakaran These authors contributed to this work equally.11
Erik Ostrowski⋆ 11 Muhammad Shafique 22

###### Abstract

Weakly Supervised Semantic Segmentation (WSSS) relying only on image-level
supervision is a promising approach to deal with the need for Segmentation
networks, especially for generating a large number of pixel-wise masks in a
given dataset. However, most state-of-the-art image-level WSSS techniques lack
an understanding of the geometric features embedded in the images since the
network cannot derive any object boundary information from just image-level
labels. We define a boundary here as the line separating an object and its
background, or two different objects. To address this drawback, we are
proposing our novel ReFit framework, which deploys state-of-the-art class
activation maps combined with various post-processing techniques in order to
achieve fine-grained higher-accuracy segmentation masks. To achieve this, we
investigate a state-of-the-art unsupervised segmentation network that can be
used to construct a boundary map, which enables ReFit to predict object
locations with sharper boundaries. By applying our method to WSSS predictions,
we achieved up to 10% improvement over the current state-of-the-art WSSS
methods for medical imaging. The framework is open-source, to ensure that our
results are reproducible, and accessible online at
https://github.com/bharathprabakaran/ReFit.

Weak Supervision, Semantic Segmentation, Masks, Medical Imaging,

Framework, CAM, Activation Maps, Boundary, Refinement

##  1 Introduction

State-of-the-art approaches for the semantic segmentation task require a deep
learning model trained on a large amount of heavily annotated data, a task
referred to as Fully Supervised Semantic Segmentation (FSSS). The generation
of pixel-wise masks for large datasets is a resource- and time-consuming
process. For instance, one frame of the Cityscapes dataset, which contains
thousands of pixel-wise frame annotations of street scenes from cities,
requires more than an hour of manual user-driven annotation [1]. Furthermore,
medical imaging and molecular biology fields require the knowledge of highly
qualified and experienced individuals capable of interpreting and annotating
the images. The interpretation of these individuals may vary based on their
experience and know-how, further inducing a problem of varying masks, known as
annotator bias. Therefore, to reduce the time and resources required for
generating pixel-wise masks, a wide range of research works focus on
developing approaches that focus on weaker kinds of supervision. This is where
Weakly Supervised Semantic Segmentation (WSSS) can be highly beneficial.

WSSS approaches focus on generating masks with minimum supervision, such as
image-level labels [2, 3, 4], bounding boxes [5, 6], point annotations [7],
and scribbles [8, 9]. This work focuses on generating semantic segmentation
masks for medical images using the most straightforward, and least-supervised,
image-level labels. We limit the scope of our work to just using image-level
labels since they are the most inexpensive form of annotation. To the best of
our knowledge, the current state-of-the-art in image-level-based WSSS methods
use class activation maps (CAMs) [10] to generate the pixel-level masks of an
object from its image-level label. The central idea of CAMs is to use any
model trained with classification loss to generate activation maps that
highlight the image regions responsible for the prediction decision. This
results mostly in a rough localization of the objects rather than precise
pixel-wise masks. The most popular CAM approaches focus on adding
regularization loss to improve the quality of the CAM prediction [11, 12] or
utilizing refinement methods that aim to enhance the CAM afterward [13, 14].
For example, adversarial erasing [15] erases the most discriminative part of
the CAM to force the model to consider different parts of an object. Chang et
al. [11] use clustering to automatically sub-divide every class into sub-
classes, implicitly generating distinctive classes for less discriminative
parts of the CAM. Several other techniques focus on exploiting the ability of
state-of-the-art classifiers to obtain viable object masks for the dataset
[16, 17, 18]. However, these approaches are not primarily feasible for medical
imaging use-cases, due to the classifier’s inability to learn relevant
information from the smaller and complex datasets. We propose to investigate
the applicability of CAMs for automating the generation of pixel-wise
segmentation masks for medical images given their image labels.

Although recognizing only the most discriminative parts of an object is enough
to achieve a high classification accuracy, the activation map might leave out
a sizeable portion of the object or include irrelevant background information,
thus leading to an inaccurate mask. The most common approaches to acquiring a
complete segmentation mask include adding more demanding conditions to the
loss function, which improves the output CAM. This can be further refined,
with respect to the initial CAM output, usually with pixel similarity
calculations. Although more complex loss functions may succeed in covering the
complete object, their downside is that the highlighted area remains blurry
and is still far from the sharp predictions achieved using FSSS. On the other
hand, the refinement-based methods can predict sharp object borders, but they
heavily rely on the quality of the initial predictions since they cannot spot
parts of the object with a different color gradient than the rest. This could
also lead to potential scenarios wherein a generated mask is inaccurate in
certain instances when incorrect anchors are used to generate the masks with
respect to an identified object. Fig. 1 presents an overview of these
differences with the help of illustrative examples from the BRATS [19] and
Decathlon [20] datasets. As shown, the CAM prediction successfully generates a
mask almost around the complete object but includes a lot of background
pixels. In contrast, a similarity-based pixel clustering instance achieves
sharper borders but may fail to capture the object due to changes in color
gradients with respect to the background. It might also fail entirely in cases
where wrong anchors are used due to a bad base prediction.

Figure 1: Comparing the CAM’s masks (a and b) and their ground truths (c and
d) against the use of similarity-based clustering strategies when incorrect (e
and f) and accurate (g and h) anchors are identified on the BRATS and
Decathlon datasets.

Therefore, state-of-the-art frameworks combine the two approaches to help
alleviate each other’s shortcomings to generate segmentation labels, known as
pseudo-labels, to train an FSSS network, which proved to be the most accurate.
For medical imaging datasets, state-of-the-art CAM models are not effective
because the classifier fails to learn on them due to the smaller size and
complexity of the dataset. Hence, we improve upon this strategy by extracting
information obtained from saliency maps and their boundaries, to differentiate
between the object and its surroundings through our novel BoundaryFit module
that acts as an additional step between the initial CAM prediction and mask
refinement.

As we have already discussed, the improved CAM output generally captures the
complete object but still includes many background pixels in its predictions.
These background pixels tend to deteriorate the quality of the refinement
method, if they achieve higher logits/certainty than the actual target object.
This may occur if the classifier fails to make confident predictions on the
given dataset. Hence, our proposed BoundaryFit module aims to refine the
borders of the initial CAM prediction without relying on pixel similarities
and instead focuses on Unsupervised Semantic Segmentation (USS) and saliency
methods to generate a simplified edge map of the input image, which is the
first step of our framework. Second, we use the edge map to determine the
image’s response map using GradCAM [21], which is a network used to obtain
CAMs. Finally, we combine the two using our BoundaryFit module, which refines
the object boundaries to obtain a fine-grained segmentation mask. We perform
extensive experiments on three different medical imaging datasets, namely, the
breast cancer ultrasound (BUSI) [22], the BRATS 2020 [19, 23], and the
Decathlon [20] datasets, to prove the effectiveness of the proposed framework
and illustrate the benefits of our approach. To sum up, the key contributions
of this work are:

  1. 1.

Our novel ReFit framework can generate fine-grained segmentation masks of
medical images using just image-level labels.

  2. 2.

The BoundaryFit module, which can be incorporated into any conventional WSSS
pipeline to obtain fine-grained segmentation masks.

  3. 3.

We have illustrated the benefits and improvements of using ReFit on three
real-world medical imaging datasets. The ReFit framework, including the
BoundaryFit module, is open-source and accessible
online***https://github.com/bharathprabakaran/ReFit.

Figure 2: Overview of our ReFit Framework, which starts with the unsupervised
segmentation (USS) stage clustering the pixels of an image based on their
similarity to the surrounding pixels; Next, we extract the edge map based on
the input cluster, which determines a preliminary boundary of the target
object; Finally, our BoundaryFit module conforms the two to obtain a fine-
grained mask.

##  2 Our ReFit Framework

Fig. 2 presents an overview of our framework, which currently incorporates a
generic CAM to obtain the necessary activation maps. First, an unsupervised
segmentation (USS) network is used to cluster the image pixels into bigger
groups based on the Quickshift [24] and SLIC [25] algorithms to create
superpixels. Second, we use a trained classifier to generate a CAM, which acts
as a preliminary segmentation mask, based on clusters from the USS, for each
class in a given input image. Followed by which our BoundaryFit module fits
the fuzzy CAM prediction to the boundaries outlined by the USS map to generate
a refined segmentation mask for each object in the image. If necessary, these
pseudo-labels can be subsequently used to train a fully supervised semantic
segmentation network to improve its precision.

###  2.1 Unsupervised Segment Detection

Our approach focuses on pre-refining the CAM prediction to the nearest edge
inside the mask in order to reduce the model’s identification of false-
positive pixel predictions. To achieve this, we propose to generate a helpful
edge map using a USS network to break down the image into simpler parts as a
function for the input $I$ as $I_{EM}=g_{USS}(I)$. The USS will cluster
similar pixels together, removing unimportant details in the input image,
leaving us with a less detailed image. The USS method we use for image
simplification consists of two convolutional layers with ReLU, Batch Norm, a
continuity loss, and a similarity loss. The continuity loss prevents the
network from using an arbitrary number of pixel clusters and constrains them
with a user-defined maximum value $q$. Meanwhile, the similarity loss
encourages the network to only cluster pixels close to each other in the
feature space.

For the ReFit framework, we used SLIC and Quickshift as the continuity loss
and performed an extended hyperparameter search to generate suitable
simplified images. On the one hand, clustering too many pixels into one group
will merge the object with the background, while clustering too few pixels
together will not simplify the image enough. Note that optimizing the hyper-
parameters for a particular image would not work for a different image.
Therefore our goal was to find hyperparameters that work well enough for all
images.

To elaborate further, we iteratively cycle through all possible hyperparameter
values in small step-sizes, while evaluating their results on the dataset to
obtain the parameter value that offers the best results for a small sample of
up to $100$ random images. we have observed such a sample represents the
dataset very well. The hyperparameter search could be mathematically optimized
to find the optimal combination if an annotated dataset for validation is
provided. But sufficient hyperparameters can be found by testing a few images
and evaluating their performance. We can find better hyperparameters when
using an annotated dataset for validation. But sufficient hyperparameters can
be found by testing less than $100$ images and evaluating their performance.
The goal is to simplify the images as much as possible without losing the
shape of the target objects. The use of other clustering algorithms is
orthogonal to our approach and can easily be included in the framework.

###  2.2 Class Activation Map - CAM

Next, we start by training a conventional classifier model on the target
dataset, which is subsequently used by a GradCAM [21] to extract the response
map for each input edge map obtained from the USS stage. The generated
segmentation masks for each class $n$ in $I_{EM}$ are considered together to
be the initial response map, which is defined as
$M_{CAM}:=\\{M_{CAM}^{0},M_{CAM}^{1},...,M_{CAM}^{n}\\}$. The original CAM
predictions highlight the most discriminative object parts and a lot of
background pixels around them, as illustrated in Fig. 1. For example, the
initial response map from our CAM covers a significant portion of the image to
ensure that the complete object is inside along with some background pixels.
The next step, i.e., the BoundaryFit module, is used to eliminate the
irrelevant pixels and obtain a fine-grained segmentation mask of the object.

###  2.3 The BoundaryFit Module

Once we successfully generate the edge map, we can use it as an additional
supervision source, which provides us with geometrical guidance for improved
boundary detection. Hence, we can combine it with the initial response map
from CAMs, which usually captures the complete object at the cost of adding
many background pixels. Next, with a combination of the edge map obtained from
the previous stage and a Floodfill algorithm [26], we successfully remove some
irrelevant background pixels from the mask.

The Floodfill algorithm accomplishes this by starting from any negative pixel
$(x,y)$ in the initial response map and then turning all pixels belonging to
the same cluster as $(x,y)$ negative as well. Note that we define a cluster as
the set of pixels inside an area closed by the boundaries in an edge map
$I_{EM}$, including the pixels that demarcate the image borders. Since some
pixel clusters also reach into the CAM prediction, only pixel clusters
completely inside the CAM mask will remain positive. We define this step as
follows:

$\begin{split}h_{BF}(M_{CAM},I_{EM})&=(M_{CAM}(I_{EM})\otimes I_{EM})\\\
&=M_{BF}\end{split}$ (1)

where $h_{BF}(\cdot)$ and $\otimes$ denote our BoundaryFit module and the
combination process of edge map and CAM prediction, respectively, which
provides us with the final segmentation mask ($M_{BF}$) that fits the
estimated boundaries of the object.

##  3 Results and Discussion

The experiments are completed on a CentOS 7.9 Operating System executing on an
Intel Core i7-8700 CPU with 16GB RAM and 2 Nvidia GeForce GTX 1080 Ti GPUs. We
executed our scripts using CUDA 11.5, Pytorch 3.7.4.3, torchvision 0.11.1, and
Pytorch-lightning 1.5.1. We use the Dice Similarity Coefficient (DSC) metric,
as shown by the state-of-the-art [27, 14], to illustrate the benefits of using
our approach to generate segmentation masks for the medical imaging datasets.
We also consider the mean Intersection-over-Union (mIoU) ratio as an
evaluation metric to analyze the framework’s benefits. For all experiments, we
evaluate the Dice Similarity Coefficient as:

$\centering DSC=\frac{2|A\cdot B|}{|A|+|B|}\@add@centering$ (2)

where $A$ and $B$ are binary matrices denoting the ground truths and
classification results, respectively, with each element corresponding to $1$
for elements inside a group and $0$ otherwise. We also evaluate the mean
Intersection-over-Union (mIoU) ratio, which is used as the evaluation metric,
and defined as

$\centering
mIoU=\frac{1}{N}\sum_{i=1}^{N}\frac{p_{i,i}}{\sum_{j=1}^{N}p_{i,j}+\sum_{j=1}^{N}p_{j,i}-p_{i,i}}\@add@centering$
(3)

where $N$ is the total number of classes, $p_{i,i}$ the number of pixels
classified as class $i$ when labelled as class $i$. $p_{i,j}$ and $p_{j,i}$
are the number of pixels classified as class $i$ that were labelled as class
$j$ and vice-versa, respectively.

We have evaluated our framework on the following three different medical
imaging datasets:  (1) the breast cancer ultrasound dataset BUSI [22],  (2)
the Multimodal Brain Tumor Segmentation dataset BRATS 2020 [19, 23, 28], and
(3) the Decathlon dataset [20].  For Decathlon, the dataset is divided in a
$3:1$ ratio for generating the training and validation sets, respectively; the
subdivided dataset is already accessible as part of the code repository
published in the WSS-CMER work. Every image contains either just the
background or the background and the singular positive class. The base
classifiers were trained on the training set and then evaluated on the
validation set. The decathlon dataset contained the T2 and ADC phases for each
image. We only used the ADC phase as discussed in the paper. For the BraTS
dataset, the 3D images of the dataset were divided into slices and cropped.
All slices were randomly subdivided into one of three sets to perform three-
fold cross-validation. For training, the classifiers used a combination of two
sets for training and were validated on the remaining set. For the BUSI
dataset, since we do not train a model on the data, we directly evaluate the
ImageNet pre-trained model on it. BUSI contains three classes, normal, benign,
and malignant. We excluded the malignant class for evaluation since our method
struggled to detect the malignant cases, as stated in the main article.

Method | Avg. DCS | Avg. mIoU
---|---|---
Blank | 64.1 | 47.2
GradCAM | 65.5 | 48.5
ReFit | 67.2 | 50.6
Table 1: Comparison of GradCAM and ReFit (w/ GradCAM backbone) on BUSI. Figure
3: (a) Image from the breast ultrasound dataset [22]; (b) ground truth; (c)
GradCAM prediction; (d) ReFit prediction.

BUSI Dataset: We start by examining the results of GradCAM, built using a
state-of-the-art classifier for the dataset, and compare them with the results
from ReFit. Note, since learning the other classes proved too difficult with
the low amount of data, we have only evaluated the benign cases. We use an
ImageNet [29] pre-trained GradCAM with a ResNet34 [30] backbone as a baseline,
instead of a state-of-the-art CAM model for natural images, because the
GradCAM achieved better results on the dataset. This is probably because the
BUSI dataset is relatively small and does not contain enough data for a larger
model to learn all the relevant information effectively. After generating the
response maps with GradCAM, we have refined them using our BoundaryFit module
to achieve the results presented in Table 1.

Conventional user-level evaluation of the BUSI images provides us with the
understanding that the cancer objects in each image are relatively small
compared to the background (see Fig. 3). Therefore, predicting a simple blank
mask achieves an mIoU of $47.2$%, while using GradCAM improves the results to
$48.5$%. However, with the ReFit framework, we improved the best prediction
mask to the highest possible value. Similarly, the average DCS for ReFit is
the highest among all three with a value of $67.2\%$. The state-of-the-art
SEAM [14] and WSS-CMER [27] techniques are unable to learn sufficient
information from this dataset to make predictions.

Method | Avg. DCS | Avg. mIoU
---|---|---
SEAM [14] | 56.1 | 39.0
WSS-CMER [27] | 59.7 | 42.6
Blank | 64.6 | 47.7
GradCAM (w/ ResNet34) | 67.3 | 50.7
GradCAM (w/ ResNet50) | 68.5 | 52.1
ReFit (w/ ResNet34) | 70.3 | 54.2
ReFit (w/ ResNet50) | 71.2 | 55.7
FSSS | 81.8 | 69.2
Table 2: Comparison of ReFit with state-of-the-art techniques on BRATS. Method | Avg. DCS | Avg. mIoU
---|---|---
Blank | 64.6 | 47.7
GradCAM (w/ ResNet101) | 65.9 | 49.1
SEAM [14] | 65.9 | 49.1
GradCAM (w/ ResNet18) | 67.0 | 50.3
WSS-CMER [27] | 71.3 | 55.4
GradCAM (w/ ResNet50) | 76.1 | 61.4
GradCAM (w/ ResNet34) | 78.0 | 63.9
ReFit (w/ ResNet34) | 79.4 | 65.8
FSSS | 86.8 | 76.7
Table 3: Comparing ReFit and state-of-the-art techniques on Decathlon. Figure
4: Visual comparison of the BRATS and Decathlon dataset images (a and b,
respectively); their ground truth segmentation masks (c and d); GradCAM
predictions (e and f); and ReFit predictions (g and h).

BRATS 2020 Dataset: Next, we evaluate the effectiveness of our ReFit framework
on the BRATS 2020 dataset and compare it with state-of-the-art techniques like
SEAM and WSS-CMER. These results are shown in Table 2. Like with the BUSI
dataset, we additionally evaluate the effectiveness of a GradCAM constructed
using trained ResNet34 and ResNet50 classifiers as the baseline. As expected,
our ReFit framework (with ResNet50) outperforms all other techniques to
achieve average DCS and mIoU values of $71.2\%$ and $55.7\%$, respectively.
The FSSS evaluation is the result reached when using a traditional, fully
supervised semantic segmentation network is trained on the provided ground-
truth pixel-wise masks. This network acts as the upper bound, when compared to
the use of other WSSS approach using pseudo-labels,, i.e., the goal of WSSS
approaches is to reach or surpass the accuracy of FSSS.  Like in the WSS-CMER
[27] approach, we have combined all instances into one single ‘positive’
class. We have followed a similar approach for the Decathlon dataset, which is
the next use-case.

Decathlon Dataset: Finally, we analyze the efficacy of the our framework on
the Decathlon Prostate dataset and compare it with the state-of-the-art and
simple GradCAMs built using various classifier variants. Out of all these
approaches, the best outcome was achieved using a GradCAM built with a
ResNet34 model, surpassing the state-of-the-art approaches. Hence, we evaluate
the ReFit framework with a ResNet34-based GradCAM, which achieves the best
outcome with a DCS score of $79.4\%$. We have illustrated the results of these
experiments in Table 3.

Although we have illustrated that ReFit is successful in improving upon the
current state-of-the-art, the literature on WSSS for medical imaging is not
comprehensive and comparison partners are rather lacking. We are working with
a set of very low-quality state-of-the-art predictions, trying to improve them
to a certain degree. More investigation might be required into this to
determine the actual benefits of using CAMs and the BoundaryFit module.
Similarly, the BoundaryFit module works on the understanding that the object
is clearly distinguishable from the background based on its boundaries; in
unlikely scenarios where this is no longer the case, this could lead to
potential problems.

Note, the state-of-the-art results for the last two datasets were extracted
directly from [27], since we were unable to reproduce them at our end.

###  3.1 Ablation Studies

Fig. 4 illustrates the visual comparison of the ReFit predictions for the
BRATS and Decathlon datasets. As anticipated, the GradCAM over-estimates the
object mask by including a significant number of background pixels for both
datasets (Figs. 4 (e) and (f)). And, by design, a significant portion of these
pixels are eliminated by our BoundaryFit module to obtain a fine-grained
segmentation mask (Figs. 4 (g) and (h)).

##  4 Conclusion

In this paper, we have proposed the novel ReFit framework, which introduces a
novel BoundaryFit module between CAM and the final FSSS model, to achieve
finer segmentation masks, which can be used to improve the overall accuracy of
the model. The BoundaryFit module provides additional saliency by utilizing
unsupervised semantic segmentation models, which refine the CAM predictions to
obtain higher-quality training labels for state-of-the-art FSSS models. The
BoundaryFit module can be incorporated into any pre-existing WSSS framework to
boost the quality of its predictions, as illustrated by three real-world
medical imaging datasets. Finally, we showed that the predictions generated by
ReFit achieve state-of-the-art performance, proving its effectiveness compared
to other approaches by up to $10\%$. ReFit is open-source and is accessible
online at https://github.com/bharathprabakaran/ReFit.

## Acknowledgments

This work is part of the Moore4Medical project funded by the ECSEL Joint
Undertaking under grant number H2020-ECSEL-2019-IA-876190. This work was also
supported in parts by the NYUAD’s Research Enhancement Fund (REF) Award on
“eDLAuto: An Automated Framework for Energy-Efficient Embedded Deep Learning
in Autonomous Systems”, and by the NYUAD Center for Artificial Intelligence
and Robotics (CAIR), funded by Tamkeen under the NYUAD Research Institute
Award CG010.

## References

  * [1] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213–3223, 2016.
  * [2] A. Kolesnikov and C. H. Lampert, “Seed, expand and constrain: Three principles for weakly-supervised image segmentation,” in European conference on computer vision, pp. 695–711, Springer, 2016.
  * [3] D. Pathak, P. Krahenbuhl, and T. Darrell, “Constrained convolutional neural networks for weakly supervised segmentation,” in Proceedings of the IEEE international conference on computer vision, pp. 1796–1804, 2015.
  * [4] P. O. Pinheiro and R. Collobert, “From image-level to pixel-level labeling with convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1713–1721, 2015.
  * [5] A. Khoreva, R. Benenson, J. Hosang, M. Hein, and B. Schiele, “Simple does it: Weakly supervised instance and semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 876–885, 2017.
  * [6] Q. Li, A. Arnab, and P. H. Torr, “Weakly-and semi-supervised panoptic segmentation,” in Proceedings of the European conference on computer vision (ECCV), pp. 102–118, 2018.
  * [7] A. Bearman, O. Russakovsky, V. Ferrari, and L. Fei-Fei, “What’s the point: Semantic segmentation with point supervision,” in European conference on computer vision, pp. 549–565, Springer, 2016.
  * [8] D. Lin, J. Dai, J. Jia, K. He, and J. Sun, “Scribblesup: Scribble-supervised convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3159–3167, 2016.
  * [9] P. Vernaza and M. Chandraker, “Learning random-walk label propagation for weakly-supervised semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7158–7166, 2017.
  * [10] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921–2929, 2016.
  * [11] Y.-T. Chang, Q. Wang, W.-C. Hung, R. Piramuthu, Y.-H. Tsai, and M.-H. Yang, “Weakly-supervised semantic segmentation via sub-category exploration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8991–9000, 2020.
  * [12] S. Jo and I.-J. Yu, “Puzzle-cam: Improved localization via matching partial and full features,” in 2021 IEEE International Conference on Image Processing (ICIP), pp. 639–643, IEEE, 2021.
  * [13] J. Ahn and S. Kwak, “Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4981–4990, 2018.
  * [14] Y. Wang, J. Zhang, M. Kan, S. Shan, and X. Chen, “Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12275–12284, 2020.
  * [15] Y. Wei, J. Feng, X. Liang, M.-M. Cheng, Y. Zhao, and S. Yan, “Object region mining with adversarial erasing: A simple classification to semantic segmentation approach,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1568–1576, 2017.
  * [16] L. Xu, W. Ouyang, M. Bennamoun, F. Boussaid, and D. Xu, “Multi-class token transformer for weakly supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4310–4319, 2022.
  * [17] T. Zhou, M. Zhang, F. Zhao, and J. Li, “Regional semantic contrast and aggregation for weakly supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4299–4309, 2022.
  * [18] L. Ru, Y. Zhan, B. Yu, and B. Du, “Learning affinity from attention: end-to-end weakly-supervised semantic segmentation with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16846–16855, 2022.
  * [19] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest, et al., “The multimodal brain tumor image segmentation benchmark (brats),” IEEE transactions on medical imaging, vol. 34, no. 10, pp. 1993–2024, 2014.
  * [20] M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze, O. Ronneberger, R. M. Summers, et al., “The medical segmentation decathlon,” Nature communications, vol. 13, no. 1, pp. 1–13, 2022.
  * [21] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE international conference on computer vision, pp. 618–626, 2017.
  * [22] W. Al-Dhabyani, M. Gomaa, H. Khaled, and A. Fahmy, “Dataset of breast ultrasound images,” Data in brief, vol. 28, p. 104863, 2020.
  * [23] S. Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempfler, A. Crimi, R. T. Shinohara, C. Berger, S. M. Ha, M. Rozycki, et al., “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge,” arXiv preprint arXiv:1811.02629, 2018.
  * [24] Q. Zhu, D. Wu, Y. Xie, and L. Wang, “Quick shift segmentation guided single image haze removal algorithm,” in 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014), pp. 113–117, IEEE, 2014.
  * [25] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “Slic superpixels,” tech. rep., 2010.
  * [26] K. P. Fishkin and B. A. Barsky, “An analysis and algorithm for filling propagation,” in Computer-generated images, pp. 56–76, Springer, 1985\.
  * [27] G. Patel and J. Dolz, “Weakly supervised segmentation with cross-modality equivariant constraints,” Medical Image Analysis, vol. 77, p. 102374, 2022\.
  * [28] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, “Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,” Scientific data, vol. 4, no. 1, pp. 1–13, 2017.
  * [29] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  * [30] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.