# TinyRadarNN: Combining Spatial and Temporal Convolutional Neural Networks
for Embedded Gesture Recognition with Short Range Radars

Moritz Scherer, Michele Magno, Jonas Erb, Philipp Mayer, Manuel Eggimann, Luca
Benini M. Scherer, P. Mayer, M. Eggimann and L. Benini are with the Dept. of
Information Technology and Electrical Engineering, ETH Zürich, Switzerland
(e-mail: {scheremo, magnom, mayerph, meggiman, lbenini}@iis.ee.ethz.ch).M.
Magno is with the Dept. of Project-based Learning, ETH Zürich, Switzerland
(e-mail: magnom@pbl.ee.ethz.ch).J. Erb was with the Dept. of Information
Technology and Electrical Engineering, ETH Zürich, Switzerland at the time of
his work on the TinyRadar project.L. Benini is also with the Dept. of
Electrical, Electronic and Information Engineering, University of Bologna,
Italy.

###### Abstract

This work proposes a low-power high-accuracy embedded hand-gesture recognition
algorithm targeting battery-operated wearable devices using low power short-
range RADAR sensors. A 2D Convolutional Neural Network (CNN) using range
frequency Doppler features is combined with a Temporal Convolutional Neural
Network (TCN) for time sequence prediction. The final algorithm has a model
size of only 46 thousand parameters, yielding a memory footprint of only
$92\text{\,}\mathrm{K}\mathrm{B}$. Two datasets containing 11 challenging hand
gestures performed by 26 different people have been recorded containing a
total of 20’210 gesture instances. On the 11 hand gesture dataset, accuracies
of 86.6% (26 users) and 92.4% (single user) have been achieved, which are
comparable to the state-of-the-art, which achieves 87% (10 users) and 94%
(single user), while using a TCN-based network that is 7500$\times$ smaller
than the state-of-the-art. Furthermore, the gesture recognition classifier has
been implemented on a Parallel Ultra-Low Power Processor, demonstrating that
real-time prediction is feasible with only $21\text{\,}\mathrm{mW}$ of power
consumption for the full TCN sequence prediction network, while a system-level
power consumption of less than $120\text{\,}\mathrm{mW}$ is achieved. We
provide open-source access to example code and all data collected and used in
this work on tinyradar.ethz.ch.

###### Index Terms:

gesture recognition, machine learning, internet of things, ultra-low power

††publicationid: pubid: ©2020 IEEE

##  I Introduction

Human-computer Interface (HCI) and Human activity recognition (HAR) systems
provide a plethora of attractive application scenarios with a wide array of
solutions, strategies, and technologies [1, 2]. Hand gestures are one of the
most natural ways for people to interact, control and engage with devices and
machines in the Internet of Things (IoT) paradigm [3]. For this reason it is
not surprising that one of the emerging technologies in the context of
wearable devices is gesture recognition [4]. Traditionally, the approaches for
capturing human gestures are based on image data or direct measurements of
movement, i.e. by using motion sensors [4, 5, 6]. The main types of sensors
used in literature are cameras with and without depth perception, force-
sensitive resistors, capacitive elements and accelerometers to measure the
movement of the subject directly [7]. While these approaches have been shown
to work well in controlled settings, robustness remains a challenge in real-
world application scenarios. Image-based approaches have to deal with well-
known environmental challenges like subject occlusion and variability in
brightness, contrast, exposure and other parameters [8]. Another drawback of
image-based solutions is the comparatively high power consumption, with
commercial sensors like the Kinect sensors having power consumptions in the
order of watts [9]. Wearable systems using motion-based sensing are much less
affected by environmental variability and typically use significantly less
power, but are more difficult to adapt to differences in user physique and
behaviour. Approaches based on Wi-Fi have also been studied. On single
subjects they have been shown to achieve high accuracy [6, 10], but are
generally restricted to coarse or full-body gestures, due to the low spatial
resolution, signal strength and susceptibility to electromagnetic interference
and multi-path reflections [11, 12, 13].

A very promising, novel sensing technology for hand gesture recognition is
based on high frequency and short-range pulsed RADAR sensors [14]. RADAR
technology can leverage the advantages of image-based recognition with reduced
challenges from environmental variability. The electromagnetic RADAR waves can
propagate through matter, such that it can potentially record responses even
if placed behind clothing. Furthermore, recently proposed designs based on
novel sensor implementations can fit within a low power budget [14, 15],
compatible with the constraints of wearable devices. However, achieving RADAR-
based gesture recognition on the highly constrained computing platforms
available on wearables remains an open challenge.

Battery-operated wearable devices for the Internet of Things, especially those
used for machine learning and data mining applications, typically host an ARM
Cortex-M or RISC-V based microcontroller, which can achieve power consumption
in the order of a few milliwatts and computational speeds in the order of
hundreds of MOp/s [16, 17, 18, 19], while offering memory storage of at most a
few megabytes. Fitting within these limited computational resources to run
machine learning algorithms especially for high-bandwidth sensors, such as
imagers or RADARs, remains challenging [20, 21]. Recently, several research
efforts have started to focus on specialized hardware to run machine learning
algorithms, and in particular neural networks on power-constrained devices
[22, 20, 23, 24]. Parallel architectures leveraging near-threshold operation
and multi-core clusters, enabling significant increases in energy efficiency,
have been explored in recent years with different application workloads [25]
and low-power systems [26].

The main state-of-the-art approaches to machine learning-based time-sequence
modelling for gesture recognition are Hidden Markov Models (HMM) [27] and Long
Short-Term Memory (LSTM) [28] networks, which both use an internal state to
model the temporal evolution of the signal. In recent years especially,
Artificial Neural Networks (ANNs) have seen a rapid increase in popularity,
with most recent works relying on LSTM-based approaches [29, 30]. On the
processing side, previous work has shown the potential of RADAR signals for
use with machine learning algorithms to classify static as well as dynamic
hand gestures [15]. The recently proposed RADAR sensing platform Soli, jointly
developed by Infineon and Google, has been studied in different works, most
prominently by Wang et al. [31]. They propose an LSTM model that achieves an
accuracy of above 90% over 11 classes.

In contrast to the state-based modelling of the input signal, Temporal
Convolutional Neural Networks (TCNs) are stateless in the sense that their
computation model does not depend on the input. This means that they can
compute sequential outputs in parallel, unlike LSTMs or HMMs [32].
Furthermore, since they only use stateless layers, TCNs use significantly less
memory for buffering feature maps compared to LSTMs, defusing the memory
bottleneck on embedded platforms. TCNs have increasingly been adopted in many
application scenarios where the classification of data is heavily linked to
its temporal properties, for example, biomedical data [33] or audio data [34].

This paper proposes a novel embedded, highly accurate temporal convolutional
neural network architecture, optimized for low-power microcontrollers. The
proposed model achieves both a memory footprint of less than
$100\text{\,}\mathrm{K}\mathrm{B}$, as well as achieving a per-sequence
inference accuracy of around 86.6% for 11 challenging gesture classes, trained
on a multi-user dataset, and 92.4% for a single-user dataset. This work
exploits novel, low-power short-range A1 RADAR sensors from
Acconeer111https://www.acconeer.com/products to acquire two rich and diverse
datasets, one for a single user and one for a total of 26 users, each
containing 11 gestures. Further, we leverage a multi-core RISC-V based
embedded processor taking advantage of the emerging parallel ultra-low power
(PULP) computing paradigm to enable the execution of complex algorithmic flows
on power-constrained devices. Similar to Soli, possible deployment scenarios
for the algorithm and processing platform include smart devices like
smartphones222https://ai.googleblog.com/2020/03/soli-radar-based-perception-
and.html and smart thermostats333https://www.gearbrain.com/new-google-nest-
hub-soli-2649744781.html and even wearables with small form factor like
smartwatches and hearing aids [26]. Due to the small footprint of less than
$30\text{\,}{\mathrm{mm}}^{2}$, the RADAR sensor can be easily integrated into
most wearable devices [35].

We show that highly-accurate, real-time hand gesture recognition within a
power budget of around $120\text{\,}\mathrm{mW}$, including the sensor and
processing consumption, is possible with the proposed sensor and computing
platform. Experimental evaluations with a working prototype demonstrate both
the power consumption and the high accuracy and are presented in the paper.

The main contribution of this paper can be summarized as follows.

  * •

Design and implementation of a TCN network architecture optimized for low-
power hand gesture recognition on microcontrollers, achieving state-of-the-art
accuracy with a total memory footprint of less than
$512\text{\,}\mathrm{K}\mathrm{B}$.

  * •

Acquistion and labeling of an open-source gesture recognition dataset
featuring 11 challenging, fine-grained hand gestures recorded with the low-
power Acconeer A1 pulsed RADAR sensor to provide a baseline dataset for future
research.

  * •

Implementation of the proposed model in a novel parallel RISC-V based
microcontroller, featuring 8 specialized parallel cores for processing and
$512\text{\,}\mathrm{K}\mathrm{B}$ of on-chip memory. The novel, power-
optimized architecture of the processors enables a full-system power
consumption below $100\text{\,}\mathrm{mW}$ in full active mode.

  * •

Evaluation of the benefits of the algorithm in terms of accuracy, energy
efficiency and inference speed, showing that the processor consumes only
$21\text{\,}\mathrm{mW}$, which is orders of magnitude less power for real-
time prediction compared to the state-of-the-art, at a comparable level of
accuracy.

##  II Related Work

Hand gesture recognition is a widely investigated field. However, it is
difficult to put all the research into context, as there are many different
categories of hand gestures, which vary in complexity. Also, depending on the
number of modelled gestures, the sensor used, and how well diversified the
studied dataset is, accuracies vary greatly. In this section we review RADAR
based approaches which are most directly comparable with our work. We refer
the interested reader to [36, 37, 38, 39, 40, 41, 42] for image-based,
inertial and RF-based gesture recognition.

###  II-A RADAR-based gesture recognition

Some research has been conducted to exploit RADAR systems or radio signals to
predict hand gestures. The approaches vary in terms of the application
scenario, as well as accuracy and power efficiency. Different models without
explicit sequence modelling have been employed in the past, a sample of which
is discussed here. Kim et al. use pulsed radio signals to determine static
hand gestures by analysing the differences between reflected waveforms with
the help of a 1D CNN. Accuracies of over 90% are achieved for American Sign
Language (ASL) hand signs using a Convolutional Neural Networks (CNN) and
micro-Doppler signatures [43]. In their feasibility analysis, Kim and
Toomajian use deep convolutional neural networks to classify ten hand-gestures
using micro-doppler signatures from a pulsed RADAR. Their offline prediction
algorithm reaches an accuracy of 85.6% on a single participant [44]. Using a
similar approach based on micro-doppler signatures and a Frequency-Modulated
Continuous Wave (FMCW) RADAR, Sun et al. showed that inference accuracy of
over 90% on a nine gesture dataset recorded from a stationary RADAR for
driving-related gestures is possible [45].

Different works have used combinations of LSTM cells or Hidden Markov Models
combined with different pre-processing strategies and convolutional layers to
classify both coarse- and fine-grained gestures with the help of time-sequence
modelling. Hazra et al. present a FMCW-based system which is trained to
recognize eight gestures, reaching an accuracy of over 94% [46]. Targeting
embedded, low-power applications, Lien et al. developed a high-frequency
short-range RADAR specifically for the purpose of hand-gesture recognition,
called Soli. They implement a neural network to classify four hand gestures.
Their final implementation uses a random forest classifier on those features
with an optional bayesian filter of the random forest output. They use four
micro-gestures, which they call ”virtual button” (pinch index), ”virtual
slider” (sliding with index finger over thumb), ”horizontal swipe” and
”vertical swipe”. On those four gestures, they achieve a per-sample accuracy
of 78.22% and a per-sequence accuracy of 92.10% for the bayesian filtered
random forest output [14]. Choi et al. used the Soli sensor and a self-
recorded 10 gesture dataset featuring ten participants to train an LSTM-based
neural network. They achieve an accuracy of over 98% using a GPU for inference
computation [47]. Using the Soli sensor, Wang et al. propose a machine
learning model to infer the hand motions contained in the RADAR signal, based
on an ANN network containing both convolutional layers and LSTM cells. They
employ a fine-grained eleven gesture dataset recorded using the Soli sensor.
While their approach shows a high average statistical accuracy of 87.17%,
their proposed model uses more than $600\text{\,}\mathrm{M}\mathrm{B}$ of
memory which is several orders of magnitude more than most low-power
microcontrollers offer. Moreover, the Soli sensors are consuming more than
$300\text{\,}\mathrm{mW}$ of power, which will drain any reasonably sized
battery for a wearable device in a few minutes of use [31].

While it has been shown that TCNs can outperform LSTMs for action segmentation
tasks both in terms of accuracy and inference speed [48, 32], the use of TCNs
for gesture recognition remains a relatively unexplored field of research.
However, one work by Luo et al. indicates that classical 2D-TCNs can perform
equally well and even outperform approaches based on LSTM cells and HMMs for
gesture recognition tasks [49].

This paper presents a combination of TCN and CNN models to improve energy
efficiency, reduce memory requirements and maximize the accuracy of gesture
recognition using sensor data from a short-range RADAR. The hardware
implementation and the benefits of the combination of TCN and CNN have briefly
been discussed in the authors’ previous work [15]. In this paper, we
significantly extend the contribution of the previous work by fully discussing
the model architecture and comparing it against other state-of-the-art gesture
recognition algorithms, showing that the proposed TCN-based model performs
significantly better in terms of accuracy per operation than the state-of-the-
art LSTM-based approach. We further evaluate in-depth the selection of
features starting from the raw sensor data. We also show a full working
implementation on an embedded platform and in-field measurements from a
demonstrator. To the best of the authors’ knowledge, there is no previous work
that evaluates the use of TCNs for embedded, real-time hand-gesture
recognition based on RADAR sensors.

##  III Background

###  III-A Range Frequency Doppler Map

Feature maps based on the Fourier transform of the time axis, like the Range
Frequency Doppler Map (RFDM), similarly to micro-Doppler signatures, have been
proven to be effective for machine learning applications in previous research
on gesture recognition [44, 45, 46, 47, 50]. It relies on the Doppler effect,
which quantifies the shift of frequency in a signal that is reflected from a
moving object. This shift of the frequency is correlated to the velocity of
the object in the direction of the sensor. In order to detect changes in
velocity, the I/Q signal is Fourier transformed into the frequency space,
where changes in frequency can be observed. In order to detect the movement of
objects in front of the sensor, multiple sweeps (i.e. time steps) are joined
together and the time signal is Fourier transformed for each range point. As
the sampled signal from each sweep $S(t,r)$ is time and range discrete the
Discrete Fourier Transform (DFT) is used. The transformed feature map $S(f,r)$
can be calculated according to the following equation:

$S(f,r)=\sum_{t=0}^{T}S(t,r)e^{-\frac{2\pi ift}{T}}$

Where $T$ is the total number of sample points per recorded distance point. In
this work, only the absolute values of this function are considered. An
example RFDM is shown in Figure 1.

Figure 1: Range-Frequency Doppler Map for an example recording. The width and
height dimensions correspond to the temporal frequency and the sampling range,
respectively.

###  III-B Temporal Convolutional Networks

Temporal Convolutional Networks are a modelling approach for time series using
dilated 1D-convolutional neural networks, proposed by Lea et al. [48], which
has been used for a multitude of tasks, but very prominently in speech
modelling [51, 52] and general human action recognition [53]. The basis of
TCNs are causal, dilated 1D-convolutions. Causal refers to the fact that for
the prediction of any time step no future inputs are considered. Thus, the
support pixel of the kernel is always chosen to be the last pixel. This is
needed in a real-time prediction scenario, as in that case only the current
and past data values are available at prediction time. To weigh past data for
sequence predicition, TCNs employ dilated convolutions over the temporal
dimension. By increasing the dilation factor for consecutive layers the
receptive field can be increased rapidly and very long effective memory of the
network can be achieved. Figure 2 shows the data flow of the TCN as used in
this work. The input data for the TCN used in this work are the flattened, 1D
outputs of a 2D-CNN.

Figure 2: Layer structure of the TCN used in this work. Each input time step
is one 1D vector that is generated by flattening the 2D CNN’s output. The
dilation factors used in the network are 1, 2 and 4. The kernel size for all
convolutional filters in the TCN is 2.

Naturally, the TCN produces one output per time step. In the following, we
will refer to metrics considering each individual time step as per-frame and
to metrics considering the time step and all previous time steps modelled in
the TCN as per-sequence.

##  IV Low Power Short Range Radar and Dataset

This chapter describes the properties of the Acconeer low power short-range
RADAR sensor that was used in this work and the parameters of the datasets
that were acquired using the sensor.

###  IV-A Short Range RADAR for Gesture Recognition

The RADAR devices used in this work are novel short-range pulsed Radio
Detection and Ranging (RADAR) from Acconeer, pulsed with
$60\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$. These low power devices use only
one transmitter and receiver which reduces the power consumption to tens of
Milliwatts. The data returned by these sensors are sampled values of the I/Q
signals. The RADAR sensor is configured to continuously emit pulses at a fixed
frequency of $f_{sweep}$, called RADAR Repetition Frequency (RRF). The time
interval between two pulses is called RADAR Repetition Interval (RRI).

Let $t=0$ be the time at which the sensor sends out a pulse. Assuming that the
transmitter and receiver are at the same position, i.e. being the same
antenna, the response received at $t+2\Delta t$ corresponds to the reflection
echo of an object located at a distance of $d=\frac{c}{2\Delta t}$ from the
emitter/receiver, where $\Delta t$ is the time-of-flight of the pulse to the
location of the object.
By regularly sampling the signal received after sending a pulse, a sweep
vector containing reflections of objects at different distances can be
computed. The distance resolution $\Delta d$ of the Acconeer sensor amounts to
$0.483\text{\,}\mathrm{mm}$, which corresponds to a time-of-flight of
$1.6\text{\,}\mathrm{ns}$.

###  IV-B Dataset Specification and Acquistion

To train and evaluate the sensor for hand gesture recognition, two datasets
were gathered in this work: One 5-gesture dataset and two 11-gesture
datasets.444The 5G and 11G datasets and code for feature extraction are
available for research purposes at https://tinyradar.ethz.ch The 11-gesture
data set features the same gestures as Wang et al. [31] and the 5-gesture
dataset uses a subset of the same 11 gestures, consisting of the ”Finger
Slide”, ”Slow Swipe”, ”Push”, ”Pull” and ”Palm Tilt” gestures. Using the same
gestures as Wang et al. [31] allows for an effective comparison. All eleven
gestures are depicted in Figure 3.

Figure 3: Overview of the gestures used in the dataset by Wang et al. [31] and
this work. The eleven gestures contain fine-grained gestures like ”Finger
Slide”, as well as coarser gestures like ”Push” or ”Pull”.

The 11-gesture dataset uses two Acconeer sensors with a sweep rate of
$160\text{\,}\mathrm{Hz}$ each, while the 5-gesture dataset uses a single
sensor with a sweep rate of $256\text{\,}\mathrm{Hz}$. Participants were shown
Figure 3, the approximate height, $20\text{\,}\mathrm{cm}$ above the sensor
board, at which to perform the gesture, but were given minimal instructions on
how to perform the gestures. The gestures were performed in sitting position,
without any additional inclination. The recording setup was not systematically
varied between different persons and recordings. The 11-gesture dataset
contains a total of 45 recording sessions of 26 different individuals, out of
which 20 recordings are recorded from the same person to evaluate single-user
accuracy, while the other 25 recordings are each recorded from different
individuals. Subsets of the 11-gesture dataset are used to evaluate single
user (SU) performance and multi-user (MU) performance. For the single-user
dataset, the aforementioned 20 recordings from one single individual are used.
For the multi-user dataset, one recording of the same individual is merged
with the remaining 25 recordings of different individuals, which results in a
dataset of 26 recordings of 26 different individuals. Thus, the multi-user and
single-user datasets overlap by one recording of one individual.

A complete overview of the dataset parameters can be found in Table I.

TABLE I: Overview of the parameters used to record the dataset Parameters | 5-G | 11-G (SU) | 11-G (MU)
---|---|---|---
Sweep frequency | $256\text{\,}\mathrm{Hz}$ | $160\text{\,}\mathrm{Hz}$ | $160\text{\,}\mathrm{Hz}$
Sensors | 1 | 2 | 2
Gestures | 5 | 11 | 11
Recording length | $3\text{\,}\mathrm{s}$ |  $\leq$ $3\text{\,}\mathrm{s}$ |  $\leq$ $3\text{\,}\mathrm{s}$
# of different people | 1 | 1 | 26
Instances per Session | 50 | 7 | 7
Sessions per recording | 10 | 5 | 5
Recordings | 1 | 20 | 26
Instances per gesture | 500 | 710 | 910
Instances per person | 2500 | 7700 | 35
Total Instances | 2500 | 7700 | 10010
Sweep ranges | 10 – $30\text{\,}\mathrm{cm}$ | 7 – $30\text{\,}\mathrm{cm}$ | 7 – $30\text{\,}\mathrm{cm}$
Sensor modules used | XR111 | XR112 | XR112

##  V Energy-Efficient and High Accuracy Gesture Recognition Algorithm

One of the major contributions of this paper is the proposal of a model to
accurately classify hand gestures recorded with a short-range RADAR sensor.
The proposed model enables the reduction of memory and computational
resources, which pose the biggest challenge for the deployment of a model for
small embedded devices such as microcontrollers.

The constraints for peak memory use and throughput in this work were chosen to
work with microcontrollers like the ARM Cortex-M7 series and RISC-V based
devices with a power budget in the order of tens of milliwatts. These
microprocessors are very memory-constrained, usually offering below
$512\text{\,}\mathrm{K}\mathrm{B}$ of memory, and achieve optimal operating
conditions when using 8-Bit quantization for the activations and 16- or 8-Bit
quantization for the weights [54].

###  V-A Preprocessing

Since the dataset consists of periodic samples of distance sweep vectors, we
chose to use the well-known approach of stacking a number $TW$ of sweep
vectors into one feature map window of raw data, which is called a frame. For
the proposed network, the number of sweep vectors was chosen to be 32. This
corresponds to a total time resolution of $200\text{\,}\mathrm{ms}$ per frame
for 11-G datasets and $125\text{\,}\mathrm{ms}$ for the 5-G dataset. These
frames are then processed by normalizing them and computing their RFDM. While
the 2D range-frequency spectrum contains a real and an imaginary component,
only the absolute value of each bin is used, since the phase component of the
spectral representation, while having the same number of values as the
magnitude, did not add any significant improvement to the overall inference
accuracy.

###  V-B Neural Network Design

For the 11-G dataset, the input feature map size is $492\times 32\times 2$
values, as each sensor contributes one channel, the number of time steps
considered are 32 and the number of range points per sweep is 492. Even when
compressing each value to $8\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$, the
total required buffer memory for each frame amounts to
$246\text{\,}\mathrm{K}\mathrm{B}$. For successful time-sequence modelling,
the information of multiple frames needs to be stored and processed. Using the
raw frame for multiple time steps would lead to buffer space requirements in
the order of megabytes, which is not available in commercial microcontrollers.

To solve this issue, the proposed model is based on a combination of a 2D CNN
and a 1D TCN, which are designed to separate the spatial-temporal modelling
problem into two parts; a short-term, spatial modelling problem, which
captures little temporal information and can be solved on the level of
individual frames, and a sequence modelling problem which can be solved on the
level of extracted features from the first network. The overall data flow is
depicted in Figure 4.

Figure 4: Overview of the processing algorithm. The raw I/Q sensor data is
first processed by applying a Fourier transform, after which features are
extracted from the frequency maps by processing them using a 2D CNN. The
results of the feature extraction stage are flattened and five time steps are
processed using a dilated TCN network.

###  V-C Spatial and Short-Term Temporal Modelling

Spatial and short-term temporal modelling in this work can be seen as the task
of extracting spatial and short-term temporal information from a single frame
of RADAR data into a 1D feature vector containing spatial features that can be
accurately classified with a sequence modelling algorithm. This approach
compresses each frame by a factor of 82$\times$, which allows the extracted
features to be stored on the low-memory microcontrollers for multiple time
steps, which is required for accurate time-sequence prediction. The proposed
network for spatial feature extraction is depicted in Figure 5.

Figure 5: Layer structure of the 2D CNN. Each Convolutional layer is followed
by a ReLU activation.

Since the width direction of the data frames corresponds to the spatial
dimension, i.e. the distance from the sensor and the height direction
corresponds to the temporal dimension of the frame, the frame width is
considerably greater than the frame height. Since the distance sampling is
chosen to be very fine-grained, wide kernels are used, both for pooling and
convolutions. The layer parameters are shown in Table II. The total required
buffer memory size for inference for algorithms using a static allocation of
memory is given by the maximum of the sum of the buffer space required for the
input and output feature map of any layer. For the proposed network, the total
required buffer size is reached in the first layer and amounts to $(492\cdot
32\cdot 2+98\cdot 10\cdot 16)\cdot 8\text{ Bit}=$
$368\text{\,}\mathrm{K}\mathrm{B}$.

TABLE II: Layer architecture of the 2D CNN Layer | Input | Output | Kernel | Padding
---|---|---|---|---
2D Conv | 32$\times$492$\times$2 | 32$\times$492$\times$16 | 3$\times$5 | Same
Max Pooling | 32$\times$492$\times$16 | 10$\times$98$\times$16 | 3$\times$5 | Valid
2D Conv | 10$\times$98$\times$16 | 10$\times$98$\times$32 | 3$\times$5 | Same
Max Pooling | 10$\times$98$\times$32 | 3$\times$19$\times$32 | 3$\times$5 | Valid
1D Conv | 3$\times$19$\times$32 | 3$\times$19$\times$64 | 1$\times$7 | Same
Max Pooling | 3$\times$19$\times$64 | 3$\times$2$\times$64 | 1$\times$7 | Valid
Flatten | 3$\times$2$\times$64 | 384 | - | -

###  V-D Long-Term Temporal Modelling

The features computed by the 2D CNN are processed further with a TCN. The TCN
uses an exponentially increasing dilation factor to combine features from
different time steps into a single feature vector which can then be passed to
a classifier consisting of fully-connected layers. For the proposed network,
five time steps are considered by the TCN, i.e. five consecutive output
feature vectors of the 2D CNN are used as the input of the TCN. This
corresponds to a total effective time window of $1\text{\,}\mathrm{s}$ for the
11-G datasets and $0.625\text{\,}\mathrm{s}$ for the 5-G dataset. The overall
TCN structure, taking into account the exponential dilation steps, is depicted
in Figure 2.

In this work, each TCN filter in the TCN consists of residual blocks, each
consisting of one depthwise convolution layer followed by a ReLU [55]
activation, the result of which is then added to the original input. This is
slightly different from the original definition of residual blocks in Lea et
al. [48], as normalization layers, dropout layers and one depthwise
convolutional layer are removed to save memory space and execution time. A
graphical comparison of the residual blocks as proposed by Lea et al. and as
used in this work can be seen in Figures 6 and 6.

Figure 6: Comparison of the TCN residual blocks. The proposed network blocks
(right) require a factor 2$\times$ less computations and memory than the
original blocks (left), due to using only one convolutional layer instead of
2.

To reduce dimensionality, the output of the 2D CNN is filtered with a 1D
Convolution which compresses the number of channels by a factor of 12$\times$.
The compressed features are then collected for a total of five time steps
before being passed to the dilated network. For the final output
classification, the output of the dilated network is passed to three fully-
connected layers. The resulting network structure is described in Table III.

TABLE III: Layer architecture of the TCN Layer | Input | Output | Kernel | Dilation
---|---|---|---|---
Causal 1D Convolution | 5$\times$384 | 5$\times$32 | 1 | -
Causal 1D Convolution | 5$\times$32 | 5$\times$32 | 2 | 1
Adding Layer | 5$\times$32 | 5$\times$32 | - | -
Causal 1D Convolution | 5$\times$32 | 5$\times$32 | 2 | 2
Adding Layer | 5$\times$32 | 5$\times$32 | - | -
Causal 1D Convolution | 5$\times$32 | 5$\times$32 | 2 | 4
Adding Layer | 5$\times$32 | 5$\times$32 | - | -
Fully connected | 5$\times$32 | 5$\times$64 | - | -
Fully connected | 5$\times$64 | 5$\times$32 | - | -
Fully connected | 5$\times$32 | 5$\times$11 | - | -

###  V-E Training Setup

Both the 2D CNN as well as the TCN were implemented using the Keras/Tensorflow
framework. The RFDM features were extracted from the dataset and saved before
training. Both network parts were trained together, using a batch size of 128
for a total of 100 epochs. The optimizer chosen for training is Adam [56].
Both 5-fold cross-validation (CV5) and leave-one-user-out cross-validation
(LOOCV) training runs were performed and are shown in the results section
(Section VI).

##  VI Results and Discussion

We evaluated the proposed model and its implementation on embedded hardware in
terms of power consumption and inference performance on the system-scale. In
particular, we present the test setup and the evaluation of the proposed model
in terms of accuracy, memory and computational requirements in the first
subsections, comparing different features and processing alternatives, while
we present an evaluation of the implementation on a novel RISC-V-based
parallel processor in a later subsection.

###  VI-A Experimental Setup

The GAP8 from Greenwaves Technologies555https://greenwaves-
technologies.com/ai_processor_gap8/ is an off-the-shelf RISC-V-based multicore
embedded microcontroller developed for IoT applications. At its heart, the
GAP8 features one RISC-V microcontroller and an octa-core RISC-V processor
cluster with support for specialized DSP instructions, derived from the PULP
open-source project [57]. The GAP8 memory architecture features two levels of
on-chip memory hierarchy, containing $512\text{\,}\mathrm{K}\mathrm{B}$ of L2
memory and $64\text{\,}\mathrm{K}\mathrm{B}$ of L1 memory.

Figure 7 shows the hardware test setup, using evaluation boards for the GAP8
and A111 RADAR sensor, connected with an ARM Cortex-M4 evaluation board, which
is used to broadcast the data to both a connected PC and the GAP8.

Figure 7: Picture of the hardware setup used to evaluate the system. The
central board is a STM32L4 development board used to interface the RADAR
sensor board (right) with the GAP8 development board (left).

The trained model was deployed onto the GAP8 with the AutoTiler tool666
https://greenwaves-technologies.com/manuals/BUILD/AUTOTILER/html/index.html,
which generates C Code optimized for parallel execution of the model on the
hardware platform.

###  VI-B Accuracy of the Algorithm

The inference accuracy of the algorithm can be discussed both in terms of per-
frame accuracy, i.e. considering every frame for only one time step or in
terms of per-sequence accuracy, i.e. the prediction for each frame taking into
account the prediction for the individual frame at all time steps. To fairly
compare results on the same dataset and frame definition, the per-frame metric
is preferable, since it allows to accurately compare different approaches and
the impact of sequence modelling versus single-frame processing. For comparing
to other datasets and frame definitions, the per-sequence accuracy is the
preferable metric, since it levels out the impact of using frames with higher
time resolution and represents more accurately how the network behaves in a
practical setting. The final results for the proposed network, both in terms
of per-frame and per-sequence accuracy are shown in Table IV.

TABLE IV: Per-frame and per-sequence inference accuracy of the full algorithm on the respective test/validation set Metric | Per-Frame Accuracy | Per-Sequence Accuracy
---|---|---
5-G SU-CV5 | 93.83% | 95.00%
11-G SU-CV5 | 89.52% | 92.39%
11-G MU-CV5 | 81.52% | 86.64%
11-G MU-LOOCV | 73.66% | 78.85%

For the following paragraphs, the per-frame accuracy is used to discuss the
impact of changes in architecture and pre-processing, while the per-sequence
accuracy is used to compare to other research.

###  VI-C Evaluation of Pre-Processing Methods

To increase classification performance, different pre-extracted features were
evaluated in combination with the features extracted by the convolutional
neural network. The pre-extracted features are the signal energy, both for the
Signal-over-Range (SOR) as well as the Signal-over-Time (SOT), the signal
variation for the SOR and SOT and the centre of mass, which measures the
intensity of the signal over the range of the sensor. An important
consideration for embedded systems is the size of the feature maps since
memory is the most common bottleneck for neural network implementations on
microcontrollers and similar devices. An overview of the number of values per
feature with respect to the number of sampling windows $TW$ and the number of
range points $RP$ can be found in Table V.

TABLE V: Overview of the size of different input features Feature | Data Format | 5-G | 11-G
---|---|---|---
Raw I/Q Signal | TW $\times$ RP $\times$ 2 | 26496 | 62976
Signal Variation 2D | (TW-1) $\times$ RP $\times$ 2 | 25668 | 61008
RFDM | TW $\times$ RP | 13248 | 31488
Signal Energy SOR | RP | 414 | 492
Signal Energy SOT | TW | 32 | 32
Signal Variation SOR | RP | 414 | 492
Signal Variation SOT | TW | 32 | 32
Centre of mass | TW $\times$ 3 | 96 | 96

Due to the splitting of the data into windows containing both spatial and
temporal information, an evaluation of the preprocessing and pre-extracted
feature performance using the 2D CNN and a fully-connected layer to estimate
the feature quality can be given. Using this setup, the per-frame training
accuracy results in Table VI were achieved.

TABLE VI: Overview of the per-frame performance of different features for the 2D-CNN Feature Combination | 5-G SU-CV5 | 11-G MU-CV5
---|---|---
Raw I/Q Signal | 90.35% | 69.09%
Signal Variation 2D | 89.93% | 65.32%
RFDM | 91.08% | 69.37%
Signal Energy SOR & SOT | 70.25% | 51.90%
Signal Energy SOR | 65.67% | 49.95%
Signal Energy SOT | 64.40% | 40.72%
Signal Variation SOR | 38.10% | 17.92%
Signal Variation SOT | 20.92% | 10.57%
Centre of mass | 47.56% | 33.81%

The RFDM features provide the best baseline in terms of pre-processed feature
maps, both in terms of memory efficiency as well as classification
performance. The raw data shows similar performance as the RFDM in the case of
a single-frame model, which makes it important to consider as using the raw
data needs no pre-processing, while all other features do. However, the
required energy to calculate the RFDM features is around 34$\times$ less than
what is used for one inference of the 2D-CNN, so the impact of pre-processing
on energy efficiency is negligible. To further increase the accuracy,
combinations of the RFDM with signal energy, variation and centre of mass were
also studied. The per-frame performance of the RFDM features combined with
other features can be seen in Table VII.

TABLE VII: Overview of the 2D-CNN per-frame network performance with combined features Feature Combination | 5-G SU-CV5 | 11-G MU-CV5
---|---|---
RFDM baseline | 91.08% | 69.37%
RFDM & signal variation 2D | 91.05% | 71.93%
RFDM & signal energy SOR | 90.99% | 70.24%
RFDM & signal variation SOR | 91.08% | 69.16%
RFDM & centre of mass | 91.34% | 70.35%
RFDM & signal variation SOT | 76.93% | 59.33%
RFDM & signal energy SOT | 91.20% | 70.33%

As already shown in the evaluation of pre-processing methods, the added
features do not increase accuracy by a significant margin, which substantiates
the choice not to add them for the proposed network.

###  VI-D Hyperparameter Tuning of the TCN

The performance of the network with the added TCN was evaluated against the
performance of the 2D CNN alone. As explained in section V-D, the number of
TCN filters is independent of the rest of the network and can be tuned to fit
the constraints of the application and target hardware. To find the optimal
operating point for the number of filters, the correlation between the number
of filters and the increase in accuracy was evaluated for the 11 gesture
dataset and is shown in Figure 8.

Figure 8: Classification performance vs. number of TCN filters on the 11-G
dataset, using 5-fold cross-validation (blue) and leave-one-out cross-
validation (grey). Even with exponential scaling of the number of filters, the
accuracy stagnates after around 32 filters.

As can be seen in the graph, the classification accuracy plateaus after 32 TCN
filters. The averaged per-frame accuracy for different selections of features
using 32 TCN filters and five time steps can be seen in Table VIII.

TABLE VIII: Overview of the averaged per-frame accuracy of the whole network with combined features Feature Combination | 5-G SU-CV5 | 11-G MU-CV5
---|---|---
Raw I/Q Signal | 91.90% | 76.91%
RFDM | 93.83% | 81.52%
RFDM & signal variation 2D | 92.75% | 78.84%
RFDM & signal energy SOR | 93.22% | 80.92%
RFDM & centre of mass | 91.81% | 78.45%
RFDM & signal energy SOT | 93.38% | 78.99%

As previously discussed in the evaluation of the pre-processing methods,
adding manually extracted features does not positively impact the overall
accuracy of the network.

Further, for all combinations of features, especially with respect to the 11
gesture multi-user dataset, the TCN improves the per-frame accuracy of the
overall network by a significant margin.

###  VI-E Comparison to LSTM-based Networks

The proposed model’s time-sequence modelling network using custom TCN layers
was also evaluated against a modelling approach based on LSTM s as proposed by
Schmidhuber et al. [28] and a network using standard TCN layers.

The performance for all three alternatives was evaluated using the same number
of filters and time steps. The per-frame test accuracies for 32 and 128
filters are shown in Table IX.

TABLE IX: Per-frame test accuracy of the whole network for different sequence modelling approaches using 32 filters Time steps | 5 | 10 | 20
---|---|---|---
LSTM, 32 filters | 79.24% | 79.69% | 80.71%
LSTM, 128 filters | 79.29% | 80.23% | 81.77%
Original TCN, 32 filters | 80.50% | 80.46% | 81.49%
Original TCN, 128 filters | 80.55% | 80.26% | 82.09%
Proposed TCN, 32 filters | 80.13% | 80.17% | 81.45%
Proposed TCN, 128 filters | 80.79% | 81.32% | 82.79%

The number of time steps beyond five does not significantly increase the
inference performance of the network neither for the TCN version nor for the
LSTM version. Besides accuracy, the focus for embedded deployment is always on
network size. Table X shows the number of parameters for 32 and 128 filters.
Note that the number of time steps does not impact the number of parameters.

TABLE X: Number of parameters required for sequence modelling using LSTM vs. TCN broken down by number of filters Filters | 32 | 64 | 96 | 128
---|---|---|---|---
LSTM | 25.4k | 99.8k | 223.5k | 396.3k
Original TCN | 12.4k | 49.6k | 111.2k | 197.4k
Proposed TCN | 6.2k | 24.8k | 55.6k | 98.7k

The number of parameters for the TCN-based implementations is much lower than
the number of parameters required for the LSTM-based implementations. Taking
into account the superior accuracy, smaller memory footprint achieved with the
TCN-based implementations, the TCN models perform better by all evaluated
metrics. Furthermore, using the proposed TCN variant, the number of parameters
for the sequence modelling part can be reduced by a factor of 4$\times$
compared to LSTM-based variants.

###  VI-F Experimental Results

The proposed algorithm, as explained in section V, was implemented and
evaluated on a GAPuino evaluation board and power measurements were taken for
both the microcontroller as well as the RADAR sensor. The overall number of
weights of the model is split between the 2D CNN, requiring 22’368 weights and
the TCN, requiring 22’917 weights. Using 16 bit quantization and considering
the implementation overheads, the network requires just under
$92\text{\,}\mathrm{K}\mathrm{B}$ on the GAP8. In terms of operations, the 2D
CNN dominates the overall algorithm, taking up more than 99% of the overall
computations, which total around
$42\text{\,}\mathrm{M}\mathrm{O}\mathrm{p}\mathrm{s}$ per inference, taking a
total of
$5.8\text{\,}\mathrm{M}\mathrm{C}\mathrm{y}\mathrm{c}\mathrm{l}\mathrm{e}\mathrm{s}$
per inference on the GAP8.

An overview of the energy consumption with respect to operating frequency is
given in Figure 9.

Figure 9: Overview of the microprocessor’s energy efficiency while running the
algorithm vs. its cluster frequency. To achieve real-time operation, at least
100 MHz are required.

For the system to work in real-time at $5\text{\,}\mathrm{Hz}$ prediction
rate, including the sampling of the RADAR sensor and execution of the
algorithm, the cluster frequency should be chosen to be at least
$100\text{\,}\mathrm{MHz}$. This leads to an average power consumption of
$21\text{\,}\mathrm{mW}$ of the GAP8 microcontroller measured during 2
inference/sleep cycles, with peak power consumption of
$98\text{\,}\mathrm{mW}$ while running the inference. An overall breakdown of
operations, energy and cycles per inference at a clock frequency of
$100\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$ using 8 cores is shown in Table
XI.

TABLE XI: Energy breakdown of the algorithm on GAP8 at 100 MHz Algorithm step | Energy per Frame | Cycles | MACs
---|---|---|---
FFT | $0.12\text{\,}\mathrm{m}\mathrm{J}$ | $176\cdot 10^{3}$ | -
2D CNN | $4.07\text{\,}\mathrm{m}\mathrm{J}$ | $5^{\prime}100\cdot 10^{3}$ | $20^{\prime}470\cdot 10^{3}$
TCN | $0.32\text{\,}\mathrm{m}\mathrm{J}$ | $458\cdot 10^{3}$ | $256\cdot 10^{3}$
Dense | $0.006\text{\,}\mathrm{m}\mathrm{J}$ | $86\cdot 10^{3}$ | $22\cdot 10^{3}$
Full Network | $4.52\text{\,}\mathrm{m}\mathrm{J}$ | $5^{\prime}820\cdot 10^{3}$ | $20^{\prime}750\cdot 10^{3}$

To consider the overall system performance, the power consumption of the RADAR
sensor has to be taken into account. Measuring the power consumption of the
development board used in this work results in an upper bound, shown in Table
XII.

TABLE XII: Power consumption of the RADAR sensor development board at different sweep frequencies Sweep frequency | Power consumption | Samples
---|---|---
$100\text{\,}\mathrm{Hz}$ | $80\text{\,}\mathrm{mW}$ | 300
$160\text{\,}\mathrm{Hz}$ | $95\text{\,}\mathrm{mW}$ | 480
$256\text{\,}\mathrm{Hz}$ | $144\text{\,}\mathrm{mW}$ | 768

Taking into account the power consumption for the RADAR sensors, we arrive at
a system-level power consumption of around $200\text{\,}\mathrm{m}\mathrm{W}$
when using two RADAR sensors at $160\text{\,}\mathrm{H}\mathrm{z}$, and
$115\text{\,}\mathrm{m}\mathrm{W}$ when using one RADAR sensor at
$160\text{\,}\mathrm{H}\mathrm{z}$.

###  VI-G Comparison to Previous Work

A direct comparison of this work is most directly possible with previous work
in Wang et al. [31] since this work uses the same set of gestures and
evaluation metrics. In Table XIII we compare our results with those reported
by Wang et al. All accuracies are reported per-sequence, as the definition of
frames is different in [31].

TABLE XIII: Comparison of the proposed implementation with previous work Metric | Interacting with Soli | This work
---|---|---
Model size | $689\text{\,}\mathrm{M}\mathrm{B}$ | $91\text{\,}\mathrm{K}\mathrm{B}$
Single sensor power consumption | $300\text{\,}\mathrm{m}\mathrm{W}$ | $95\text{\,}\mathrm{m}\mathrm{W}$
Total sensor power consumption | $300\text{\,}\mathrm{m}\mathrm{W}$ | $190\text{\,}\mathrm{m}\mathrm{W}$
Network inference power | $-$ | $21\text{\,}\mathrm{m}\mathrm{W}$
11-G SU Accuracy | 94.5% | 92.39%
11-G MU-CV5 Accuracy | - | 86.64%
11-G MU-LOOCV Accuracy | 88.27% | 78.85%
Number of different users | 10 | 26

The direct comparison shows that our proposed network performs comparably
accurately, if slightly worse, in all but leave-one-subject-out cross-
validation, to the network proposed by [31]. Nonetheless, our network size is
smaller by a factor of 7’500$\times$ and our power consumption is lower by
several orders of magnitudes, as [31] use a GPU for inference, which operates
at tens to hundreds of Watts of power consumption.

##  VII Conclusion

This work presented a high-accuracy and low-power hand-gesture recognition
model combining a TCN and CNN model to achieve accuracy and low memory
footprint. The model targets data processing with short-range RADAR. The paper
proposed also a hand-gesture recognition system that uses low-power RADAR
sensors from Acconeer combined with a GAP8 Parallel Ultra-Low-Power processor
and can be battery operated. Two large datasets with 11 challenging hand-
gestures performed by 26 different people containing a total of 20’210 gesture
instances were recorded, on which the proposed algorithm reaches an accuracy
of up to 92.4%. The model size is only $92\text{\,}\mathrm{k}\mathrm{B}$ and
the implementation in GAP8 shows that live-prediction is feasible with a power
consumption of the prediction network of only
$21\text{\,}\mathrm{m}\mathrm{W}$. The results show the effectiveness and
potential of RADAR-based hand-gesture recognition for embedded devices, as
well as the network design, using the TCN approach. Further, we provide all
necessary data and code to train the TinyRadarNN on tinyradar.ethz.ch.

## Acknowledgment

The authors would like to thank armasuisse Science & Technology for funding
this research. The authors also thank Michael Rogenmoser and Cristian Cioflan
for their valuable contributions to the research project.

## References

  * [1] F. Naujoks, Y. Forster, K. Wiedemann, and A. Neukum, “A human-machine interface for cooperative highly automated driving,” in _Advances in Human Aspects of Transportation_. Springer, 2017, pp. 585–595.
  * [2] Q. Li, R. Gravina, Y. Li, S. Alsamhi, F. Sun, and G. Fortino, “Multi-user activity recognition: Challenges and opportunities,” _Information Fusion_ , vol. 63, 06 2020.
  * [3] Y. Zhang, Y. Chen, H. Yu, X. Yang, and W. Lu, “Learning effective spatial-temporal features for semg armband based gesture recognition,” _IEEE Internet of Things Journal_ , 2020.
  * [4] J. Wu and R. Jafari, “Orientation independent activity/gesture recognition using wearable motion sensors,” _IEEE Internet of Things Journal_ , vol. 6, no. 2, pp. 1427–1437, 2018.
  * [5] R. Xu, S. Zhou, and W. J. Li, “Mems accelerometer based nonspecific-user hand gesture recognition,” _IEEE sensors journal_ , vol. 12, no. 5, pp. 1166–1173, 2011.
  * [6] A. Dementyev and J. Paradiso, “Wristflex: low-power gesture input with wrist-worn pressure sensors,” _Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology_ , 10 2014.
  * [7] A. Bandini and J. Zariffa, “Analysis of the hands in egocentric vision: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [8] X. Zabulis, H. Baltzakis, and A. A. Argyros, “Vision-based hand gesture recognition for human-computer interaction,” in _The Universal Access Handbook_ , 2009.
  * [9] P. Fankhauser, M. Bloesch, D. Rodriguez, R. Kaestner, M. Hutter, and R. Siegwart, “Conference presentation slides on kinect v2 for mobile robot navigation: Evaluation and modeling,” _International Conference on Advanced Robotics (ICAR)_ , 2015.
  * [10] Y. Zhang and C. Harrison, “Tomo: Wearable, low-cost electrical impedance tomography for hand gesture recognition,” in _UIST’15_ , 11 2015, pp. 167–173.
  * [11] J. Yang, H. Zou, Y. Zhou, and L. Xie, “Learning gestures from wifi: A siamese recurrent convolutional architecture,” _IEEE Internet of Things Journal_ , vol. 6, no. 6, pp. 10 763–10 772, 2019.
  * [12] J. Wang, L. Zhang, C. Wang, X. Ma, Q. Gao, and B. Lin, “Device-free human gesture recognition with generative adversarial networks,” _IEEE Internet of Things Journal_ , vol. 7, no. 8, pp. 7678–7688, 2020.
  * [13] F. Adib, Z. Kabelac, D. Katabi, and R. Miller, “3d tracking via body radio reflections,” _Usenix NSDI_ , 12 2013.
  * [14] J. Lien, N. Gillian, M. E. Karagozler, P. Amihood, C. Schwesig, E. Olson, H. Raja, and I. Poupyrev, “Soli: Ubiquitous gesture sensing with millimeter wave radar,” _ACM Trans. Graph._ , vol. 35, no. 4, Jul. 2016. [Online]. Available: https://doi.org/10.1145/2897824.2925953
  * [15] M. Eggimann, J. Erb, P. Mayer, M. Magno, and L. Benini, “Low power embedded gesture recognition using novel short-range radar sensors,” in _2019 IEEE SENSORS_. IEEE, 2019, pp. 1–4.
  * [16] A. Rahimi, P. Kanerva, and J. M. Rabaey, “A Robust and Energy-Efficient Classifier Using Brain-Inspired Hyperdimensional Computing,” in _Proc. ACM/IEEE ISLPED_ , 2016, pp. 64–69.
  * [17] M. Magno, M. Pritz, P. Mayer, and L. Benini, “DeepEmote: Towards multi-layer neural networks in a low power wearable multi-sensors bracelet,” in _Proc. IEEE IWASI_ , 2017, pp. 32–37.
  * [18] C. Savaglio, P. Gerace, G. Di Fatta, and G. Fortino, “Data mining at the iot edge,” in _2019 28th International Conference on Computer Communication and Networks (ICCCN)_ , 2019, pp. 1–6.
  * [19] G. Fortino, S. Galzarano, R. Gravina, and W. Li, “A framework for collaborative computing and multi-sensor data fusion in body sensor networks,” _Information Fusion_ , vol. 22, 01 2014.
  * [20] J. Chen and X. Ran, “Deep learning with edge computing: A review,” _Proceedings of the IEEE_ , vol. 107, no. 8, pp. 1655–1674, 2019.
  * [21] F. Samie, L. Bauer, and J. Henkel, “From cloud down to things: An overview of machine learning in internet of things,” _IEEE Internet of Things Journal_ , vol. 6, no. 3, pp. 4921–4934, 2019.
  * [22] S. Deng, H. Zhao, W. Fang, J. Yin, S. Dustdar, and A. Y. Zomaya, “Edge intelligence: The confluence of edge computing and artificial intelligence,” _IEEE Internet of Things Journal_ , vol. 7, no. 8, pp. 7457–7469, 2020.
  * [23] L. Cavigelli and L. Benini, “CBinfer: Exploiting Frame-to-Frame Locality for Faster Convolutional Network Inference on Video Streams,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2019.
  * [24] L. Cavigelli, D. Gschwend, C. Mayer, S. Willi, B. Muheim, and L. Benini, “Origami: A Convolutional Network Accelerator,” in _Proc. IEEE GLSVLSI_. New York, New York, USA: ACM Press, 2015, pp. 199–204.
  * [25] M. Gautschi, P. D. Schiavone, A. Traber, I. Loi, A. Pullini, D. Rossi, E. Flamand, F. K. Gurkaynak, and L. Benini, “Near-Threshold RISC-V Core With DSP Extensions for Scalable IoT Endpoint Devices,” _IEEE TVLSI_ , vol. 25, no. 10, pp. 2700–2713, 2017.
  * [26] M. Eggimann, S. Mach, M. Magno, and L. Benini, “A risc-v based open hardware platform for always-on wearable smart sensing,” in _2019 IEEE 8th International Workshop on Advances in Sensors and Interfaces (IWASI)_. IEEE, 2019, pp. 169–174.
  * [27] L. Rabiner and B. Juang, “An introduction to hidden markov models,” _IEEE ASSP Magazine_ , vol. 3, no. 1, pp. 4–16, 1986.
  * [28] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, pp. 1735–80, 12 1997.
  * [29] S. Mitra and T. Acharya, “Gesture recognition: A survey,” _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , vol. 37, no. 3, pp. 311–324, 2007.
  * [30] A. K. H. Al-Saedi and A. H. H. Al-Asadi, “Survey of hand gesture recognition systems,” _Journal of Physics: Conference Series_ , vol. 1294, p. 042003, sep 2019. [Online]. Available: https://doi.org/10.1088%2F1742-6596%2F1294%2F4%2F042003
  * [31] J. Song, S. Wang, J. Lien, I. Poupyrev, and O. Hilliges, “Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum,” in _ACM Symposium on User Interface Software and Technologies_ , ser. UIST ’16. New York, NY, USA: ACM, 2016.
  * [32] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” 2018.
  * [33] D. Jarrett, J. Yoon, and M. van der Schaar, “Dynamic prediction in clinical survival analysis using temporal convolutional networks,” _IEEE journal of biomedical and health informatics_ , 2019.
  * [34] E. MatthewDavies and S. Böck, “Temporal convolutional networks for musical audio beat tracking,” in _2019 27th European Signal Processing Conference (EUSIPCO)_. IEEE, 2019, pp. 1–5.
  * [35] S. Ahmed, K. D. Kallu, S. Ahmed, and S. H. Cho, “Hand gestures recognition using radar sensors for human-computer-interaction: A review,” _Remote Sensing_ , vol. 13, no. 3, 2021. [Online]. Available: https://www.mdpi.com/2072-4292/13/3/527
  * [36] J. Wan, G. Guo, and S. Z. Li, “Explore efficient local features from rgb-d data for one-shot learning gesture recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 8, pp. 1626–1639, 2016.
  * [37] D. Wu, L. Pigou, P. Kindermans, N. D. Le, L. Shao, J. Dambre, and J. Odobez, “Deep dynamic neural networks for multimodal gesture segmentation and recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 38, no. 8, pp. 1583–1597, 2016.
  * [38] A. D. Calin, “Gesture recognition on kinect time series data using dynamic time warping and hidden markov models,” in _2016 18th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC)_ , 2016, pp. 264–271.
  * [39] S. Ji, W. Xu, M. Yang, and K. Yu, “3d convolutional neural networks for human action recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 1, pp. 221–231, 2013.
  * [40] O. Koller, C. Camgoz, H. Ney, and R. Bowden, “Weakly supervised learning with multi-stream cnn-lstm-hmms to discover sequential parallelism in sign language videos,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2019.
  * [41] B. Kellogg, V. Talla, and S. Gollakota, “Bringing gesture recognition to all devices,” in _11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14)_. Seattle, WA: USENIX Association, Apr. 2014, pp. 303–316. [Online]. Available: https://www.usenix.org/conference/nsdi14/technical-sessions/presentation/kellogg
  * [42] J. Yang, H. Zou, Y. Zhou, and L. Xie, “Learning gestures from wifi: A siamese recurrent convolutional architecture,” _IEEE Internet of Things Journal_ , vol. 6, no. 6, pp. 10 763–10 772, 2019.
  * [43] S. Y. Kim, H. G. Han, J. W. Kim, S. Lee, and T. W. Kim, “A hand gesture recognition sensor using reflected impulses,” _IEEE Sensors Journal_ , vol. 17, no. 10, pp. 2975–2976, 2017.
  * [44] Y. Kim and B. Toomajian, “Hand gesture recognition using micro-doppler signatures with convolutional neural network,” _IEEE Access_ , vol. 4, pp. 7125–7130, 2016.
  * [45] Y. Sun, T. Fei, S. Gao, and N. Pohl, “Automatic radar-based gesture detection and classification via a region-based deep convolutional neural network,” in _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 4300–4304.
  * [46] S. Hazra and A. Santra, “Radar gesture recognition system in presence of interference using self-attention neural network,” in _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_ , 2019, pp. 1409–1414.
  * [47] J.-W. Choi, S.-J. Ryu, and J.-H. Kim, “Short-range radar based real-time hand gesture recognition using lstm encoder,” _IEEE Access_ , vol. PP, pp. 1–1, 03 2019.
  * [48] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager, “Temporal convolutional networks for action segmentation and detection,” 2016.
  * [49] F. Luo, S. Poslad, and E. Bodanese, “Temporal convolutional networks for multi-person activity recognition using a 2d lidar,” _IEEE Internet of Things Journal_ , pp. 1–1, 2020.
  * [50] Y. Sun, T. Fei, F. Schliep, and N. Pohl, “Gesture classification with handcrafted micro-doppler features using a fmcw radar,” in _2018 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM)_ , 2018, pp. 1–4.
  * [51] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” 2016.
  * [52] A. Pandey and D. Wang, “Tcnn: Temporal convolutional neural network for real-time speech enhancement in the time domain,” in _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 6875–6879.
  * [53] T. S. Kim and A. Reiter, “Interpretable 3d human action analysis with temporal convolutional networks,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2017, pp. 1623–1631.
  * [54] L. Lai, N. Suda, and V. Chandra, “Cmsis-nn: Efficient neural network kernels for arm cortex-m cpus,” _arXiv preprint arXiv:1801.06601_ , 2018.
  * [55] R. H. R. Hahnloser, R. Sarpeshkar, M. A. Mahowald, R. J. Douglas, and H. S. Seung, “Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit,” _Nature_ , vol. 405, no. 6789, pp. 947–951, Jun. 2000.
  * [56] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _International Conference on Learning Representations_ , 12 2014.
  * [57] M. Gautschi, P. D. Schiavone, A. Traber, I. Loi, A. Pullini, D. Rossi, E. Flamand, F. K. Gürkaynak, and L. Benini, “Near-threshold risc-v core with dsp extensions for scalable iot endpoint devices,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 25, no. 10, pp. 2700–2713, 2017.

|  Moritz Scherer (GS’20) received the B.Sc. and M.Sc. degree in electrical
engineering and information technology from ETH Zürich in 2018 and 2020,
respectively, where he is currently pursuing a Ph.D. degree at the Integrated
Systems Laboratory. His current research interests include the design of
ultra-low power and energy-efficient circuits and accelerators as well as
system-level and embedded design for machine learning and edge computing
applications. He is the recipient of the Best Presentation Award at the 2019
IEEE Sensors and Applications Symposium. Moritz Scherer received the ETH Medal
for his Master’s thesis in 2020.
---|---
|  Michele Magno (SM’13) received his Masters and Ph.D. degrees in electronic
engineering from the University of Bologna, Italy, in 2004 and 2010,
respectively. Currently, he is a senior researcher at ETH Zurich, Switzerland,
and Head of the Project-Based Learning Center at ETH Zurich. The key topics of
his research are wireless sensor networks, wearable devices, machine learning
at the edge, energy harvesting, power management techniques, and extended
lifetime of batterie operated devices. He has collaborated with several
universities and research centers, such as Mid University Sweden, where he is
a guest full professor. He has published more than 150 papers in international
journals and conferences, in which he got multiple best paper and best poster
awards.
---|---
|  Jonas Erb received the B.Sc. and M.Sc. degree in electrical engineering and
information technology from ETH Zurich in 2017 and 2019, respectively. After
working in the Vast AG car rental start-up as an IT project manager in 2019,
he started his own coaching business at the end of 2019 and works
independently as an Embodied Authenticity coach since then, while following
his passion for engineering in his spare time.
---|---
|  Philipp Mayer (GS’17) received his B.Sc. degree in electrical engineering
and information technology from the TU Wien, Austria in 2016, and a
consecutive M.Sc. degree from the ETH Zurich, Switzerland in 2018. He is
currently pursuing a Ph.D. degree at the ETH Zurich Integrated System
Laboratory. His research interests include low-power system design, energy
harvesting, and edge computing. He is the recipient of the best paper award at
the 2017 IEEE International Workshop on Advances in Sensors and Interfaces and
the best student paper award at the 2018 IEEE Sensors Applications Symposium.
---|---
|  Manuel Eggimann (GS’18) Received his B.Sc. and consecutive M.Sc. degree in
electrical engineering and information technology from the ETH Zurich,
Switzerland in 2018. He is currently pursuing a Ph.D. degree at the ETH Zurich
Integrated Systems Laboratory. His research interests include low-power
hardware design, edge-computing and VLSI. He is the recipient of the best
paper award at the 2019 IEEE 8th International Workshop on Advances in Sensors
and Interfaces.
---|---
|  Luca Benini (F’07) is the Chair of Digital Circuits and Systems at ETH
Zürich and a Full Professor at the University of Bologna. He has served as
Chief Architect for the Platform2012 in STMicroelectronics, Grenoble. Dr.
Benini’s research interests are in energy-efficient system and multi-core SoC
design. He is also active in the area of energy-efficient smart sensors and
sensor networks. He has published more than 1’000 papers in peer-reviewed
international journals and conferences, four books and several book chapters.
He is a Fellow of the ACM and of the IEEE and a member of the Academia
Europaea.
---|---


**footnotetext: Equal contribution. Corresponding author email:
<EMAIL_ADDRESS>This work was conducted and fully
financed by the Corporate Research and Technology department of Carl Zeiss
AG.11institutetext: Carl Zeiss AG, Germany 22institutetext: Technische
Universität München, Germany 33institutetext: Munich Center for Machine
Learning, Germany

# SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow
Prediction

Çağhan Köksal ∗ 11 2 2 Ghazal Ghazaei∗ 11 Felix Holm 1122 Azade Farshad 2233
Nassir Navab 2233

###### Abstract

Graph-based holistic scene representations facilitate surgical workflow
understanding and have recently demonstrated significant success. However,
this task is often hindered by the limited availability of densely annotated
surgical scene data. In this work, we introduce an end-to-end framework for
the generation and optimization of surgical scene graphs on a downstream task.
Our approach leverages the flexibility of graph-based spectral clustering and
the generalization capability of foundation models to generate unsupervised
scene graphs with learnable properties. We reinforce the initial spatial graph
with sparse temporal connections using local matches between consecutive
frames to predict temporally consistent clusters across a temporal
neighborhood. By jointly optimizing the spatiotemporal relations and node
features of the dynamic scene graph with the downstream task of phase
segmentation, we address the costly and annotation-burdensome task of semantic
scene comprehension and scene graph generation in surgical videos using only
weak surgical phase labels.Further, by incorporating effective intermediate
scene representation disentanglement steps within the pipeline, our solution
outperforms the SOTA on the CATARACTS dataset by $8\%$ accuracy and $10\%$ F1
score in surgical workflow recognition.

###### Keywords:

Surgical Phase Segmentation Scene Graph Generation Unsupervised Video
Segmentation Surgical Scene Understanding

##  1 Introduction

Surgical videos capture pivotal moments of surgery, providing a valuable
source of information that can facilitate better insights into the quality of
surgery. Automated analysis of these videos can significantly enhance surgical
procedures via online or offline feedback. Surgical workflow prediction has
been a focal point of numerous studies, highlighting its critical role in
enhancing surgical precision and efficiency through video analysis [27, 22, 4,
23, 11, 17] Recently, methods based on scene graph representations
overperformed non-graph methods thanks to its holistic scene understanding
capabilities[11, 24, 16, 9, 17]. A primary barrier to developing such
technologies is the lack of dense annotations. Moreover, surgical video
annotation is not only inherently burdensome but requires specialized
annotation platforms and expert annotators, which can be prohibitively
expensive for specialized surgeries. Given the inherent interdependence of
surgical scene understanding and workflow prediction during surgical
procedures, it is imperative to address these two tasks simultaneously. In
this regard, we introduce a novel approach of SurgicAl SceNe GRaph
OptImizAtion (SANGRIA) that tackles both problems using solely surgical phase
labels.

Figure 1: Our end-to-end surgical scene graph generation and workflow
prediction pipeline, SANGRIA, comprising: 1) Spectral Temporal Clustering
converts input frames into a dynamic patchified graph leveraging graph
partitioning and local feature matching to produce an initial dynamic scene
graph. 2) DSG Optimization optimizes the edge weights between clusters for the
end task. 3) Task prediction a GCN-based architecture to predict surgical
phases.

Prior works on unsupervised scene and video segmentation take advantage of
optical flow [10] or similar to CutLER [28, 29] use variations of
NormalizedCut [25] and self-training [30] to find salient objects in the
scene. For surgical scene understanding, similarly, optical flow or shape
priors have been used [31, 21]. Shape priors lack adaptability and flexibility
to new tools or setups and struggle when visually similar tools are present.
While optical flow is more generalizable, it introduces noise from anatomical
movement or fluid during surgery. The main shortcoming with solutions such as
[28, 29] is the burdensome process of mask generation followed by rounds of
self-training, which would be needed on every new dataset. The advent of
foundation models [6, 18, 13] has opened new opportunities for scene
understanding with minimal annotations. In the context of surgical scenes,
this impact is still less pronounced, considering the significant domain gap
between common computer vision use cases and medical applications, requiring
further fine-tuning. Previous research demonstrated that (dynamic) scene graph
representations provide a more holistic and interpretable representation of
surgical procedures for surgical workflow recognition [11, 17]. To mitigate
the annotation scarcity in surgical videos and address both scene
understanding and surgical workflow recognition simultaneously, we propose a
novel data-agnostic pipeline that integrates the benefits of semantic scene
graphs and graph-based unsupervised semantic scene segmentation techniques.
Our approach comprises two main components:

First, a task- and data-agnostic spectral and temporal clustering module
leveraging DMON layer [26] for unsupervised clustering of scene components
reinforced with LightGlue [15] for sparse and lightweight local feature
matching among neighboring frames. Acting as a zero-shot semantic scene
segmenter, this module ensures local and temporal consistency while optimizing
scene representation for downstream tasks. Our solution builds upon insights
from recent works [1, 5], formulating semantic segmentation as a graph
partitioning task. While these models indicate promising performance on still
images, they lack temporal consistency among neighboring frames when applied
to video sequences. To overcome this challenge, we leverage LightGlue [15]
into our dynamic graph construction, which predicts an assignment between
points based on their pairwise similarity and unary matchability. This leads
to sparse dynamic links among close frames and significantly reduced
computations.

Second, a dynamic scene graph (DSG) generation module optimizing graph
relations based on the downstream task. The end-to-end pipeline is jointly
optimized for the specific downstream task, which is surgical phase
segmentation in this work. As the predicted semantic segmentation maps and the
generated scene graphs lack object class identities, we propose a simple yet
efficient prototype matching strategy via which only a few prototypes are
exploited to retrieve class identities. In this work, we introduce SANGRIA, a
novel graph-based pipeline harnessing the holisticness and flexibility of
graph-based learning to overcome the challenge of annotation scarcity in
surgical video analysis.

Our contributions are as follows: 1) Unsupervised semantic scene segmentation
and dynamic scene graph generation leveraging a spectral and temporal
clustering module equipped with lightweight correspondence matching. 2)
Demonstrating the importance of understanding scene components and their
spatiotemporal relations in contributing to the downstream task. By
disambiguating video representations in an end-to-end optimization setup,
SANGRIA achieves state-of-the-art performance on surgical phase segmentation
on the CATARACTS [2] dataset. 3) Offering a few-shot prototype matching
mechanism enabling further refinement of the predictions and demonstrating
promising performance in annotation-efficient scene graph generation.

##  2 Methodology

We tackle the problem of semantic scene understanding and scene graph
generation with an end-to-end graph-based pipeline. Formulating semantic
segmentation as a graph partitioning task, we patchify input images and
establish sparse temporal links with the neighboring frame patches via
correspondence matching, constructing a dynamic graph (the patch-based graph
hereafter). We then perform a spectral, temporal clustering of the patch-based
graph to generate a dynamic semantic scene graph. This DSG is augmented with a
dynamic relation prediction module to be further refined for the downstream
task of surgical phase segmentation. Finally, a prototype matching mechanism
is developed for a final refinement, few-shot segmentation, and scene graph
generation evaluations.

###  2.1 Dynamic Scene Graph Generation

For an input image $I$, DINO key features ${f}$ are obtained by partitioning
${I}$ into $n$ patches and passing them to DINO. The adjacency matrix ${A}$ is
then generated by patchwise dot product as follows:

${\mathcal{A}}_{ij}=\cases{f}_{i}\cdot{f}_{j}&\text{if}{f}_{i}\cdot{f}_{j}>0\\\
0\text{otherwise}$ (1)

Next, we threshold the values in ${\mathcal{A}>\tau}$ and connect highly
similar nodes together, generating a static patch graph representation
$\mathcal{G}_{t}=(\mathcal{V}_{t},\mathcal{A}_{t})$ for a given frame t. An
extension of graph clustering to sequences of frames without considering the
temporal inter-dependencies leads to inconsistent clusters among frames with
close proximity. A naïve solution could be an expansion of adjacency matrix
calculation across the third dimension of time to find spatiotemporal
similarities across patches. This leads to high computational costs
$\mathcal{O}(wn^{2}d$) specifically with the increasing length of the temporal
window for $n$ number of patches, patch feature of length $d$, and $w$ time
steps. As temporal relations require a coarser level of attention compared to
spatial dependencies [7], we suggest a sparse dynamic linking mechanism
between patches along the time dimension.

In this work, we leverage correspondence matching to find prominent features
within frames and match those efficiently. We incorporate LightGlue [15], a
distilled deep neural network powered with self- and cross-attention, into our
patch graph construction setup. It is designed explicitly for low-latency
problems and sparse inputs by predicting matches from two sets of local
features. Next, for a clip with $w$ frames, we construct a dynamic patch-based
graph, $\mathcal{G}_{t_{i}\xrightarrow{}t_{i+w}}=(\mathcal{V},\mathcal{E})$
with node set $\mathcal{V}$, edge set $\mathcal{E}$, node features
$X\in\mathbb{R}^{w\times n\times d}$. Spatial edges, $\mathcal{E}_{t_{i}}$ are
established using pairwise correlation similarity between those features
(Equation 1), while for temporal edges
$\mathcal{E}_{t_{i}\xrightarrow{}t_{i+1}}$, LightGlue correspondences between
frames within a temporal sequence of $w$ time stamps are exploited. Dynamic
graph edges can be represented as follows: $\mathcal{E}=\bigcup_{1\leq t\leq
w}\mathcal{E}_{t_{i}}+\mathcal{E}_{t_{i}\xrightarrow{}t_{i+1}}$

We further reinforce the graph nodes with temporal and spatial encodings to
accentuate the dynamic relations between objects in the scene. Temporal
encodings capture the temporal order of object interactions and actions in a
video sequence, while spatial encodings capture objects’ relative positions
and orientations in a scene. For temporal encoding, we incorporate the
location of each frame along the temporal window by adding a temporal feature
vector to the node feature matrix $X$. For spatial encoding, we incorporate
the position of patches within the frame by adding a spatial feature vector to
feature matrix $X$.

The graph clustering is performed by employing deep modularity networks (DMON)
[26] featuring a collapse regularization objective to improve unsupervised
graph clustering in real-world scenarios. The DMON module comprises multiple
graph convolutional layers [12], MLP layers with softmax, as well as a SeLU
activation function. The input graph $X$ is fed into the DMON module to be
clustered, optimizing the modularity and collapse regularization objectives.
This leads to a pooled scene graph
$G^{pool}_{t_{i}\xrightarrow{}t_{i+w}}=(\mathcal{V}^{pool}_{t_{i}\xrightarrow{}t_{i+w}},\mathcal{E}^{pool}_{t_{i}\xrightarrow{}t_{i+w}})$
with pooled node features
$X^{pool}_{t_{i}\xrightarrow{}t_{i+w}}\in\mathbb{R}^{K\times d}$ and pooled
adjacency matrix $A^{pool}_{t_{i}\xrightarrow{}t_{i+w}}\in\mathbb{R}^{K\times
K}$, where $K$ is the number of clusters. This DSG incorporates the summary of
a window, that is, the scene components, their relations, and motion along the
time. Figure 2 illustrates the DSG generation workflow.

Figure 2: DSG generation: A patchified input image is fed into DINO to
construct a patch-wise affinity matrix. The static patch-based input graphs
for neighboring frames within a window of $w$ are temporally linked via sparse
matches provided by LightGlue [15]. The dynamic patch-based graph is then
clustered to predict a DSG for the last frame of the window.

DSG Optimization We establish a probabilistic estimation of edge weights
within the DSGs via equation $W^{pool}_{t_{i}\rightarrow
t_{i+w}}=\sigma(\text{MLP}(X^{pool}_{t_{i}\rightarrow
t_{i+w}};\Theta_{\text{MLP}}))$, where $W^{pool}_{t_{i}\xrightarrow{}t_{i+w}}$
refers to the edge weights between the clusters of DSG, in which
$w_{ij}\in[0,1]$ indicating the strength of the relations between clusters
$i$, $j$. $\Theta_{\text{MLP}}$ represents the set of trainable parameters.
The probabilistic setup allows for flexibility in optimizing the edge weights
while learning the downstream tasks and equips the DSG generation to account
for the inherent uncertainty and variability present in the unsupervised graph
clustering results, leading to more robust and accurate inference.

###  2.2 End-to-end Pipeline

To tackle the task of phase segmentation, we propose a multi-layer GCN [12,
11] that takes the DSG as input. The GCN consists of multiple layers, each of
which enables learning increasingly complex representations of the scene
graph. The output of the GCN is fed to a global sum-pooling layer aggregating
features from all nodes in the graph. A fully-connected layer and a softmax
function predict probabilities for each phase class. A cross-entropy objective
function $L_{CE}$ is employed to optimize the model parameters for the
surgical phase segmentation task. The final objective function of the end-to-
end pipeline can therefore be formulated as $L_{joint}=L_{u}+L_{CE}$. The
first term corresponds to DMON loss, while the second term can be replaced
with any other downstream task optimization loss for each specific use case.
This joint optimization provides a tailored representation of the scene for
the downstream task and reduces the noises introduced during unsupervised
steps by taking advantage of the classification labels.

Prototype Matching To assign semantic classes to DSG nodes, we create
prototypes by leveraging DINO patch features and ground truth (GT)
segmentation annotations. Exploiting only 5 annotations per class, patch
features corresponding to GT segmentation masks are used as prototype patches.
We use the mean of the patch features as a prototype for that object class. To
predict the semantic category of a node (cluster), a pairwise cosine
similarity with prototypes is calculated. Since different clusters might
represent the same region, such as surgical tape, we used argmax to assign
class labels to clusters.

##  3 Implementation Details

Datasets We experiment on 3 datasets: CATARACTS [3, 2] consists of 50 cataract
surgery videos of $1920\times 1080$ pixels at $30$ fps. The dataset is split
25-5-20 for training, validation, and testing, with videos annotated on 19
surgical phases. CaDIS dataset, a subset of CATARACTS, consists of $4670$
pixel-wise annotated images. We use Task II of CaDIS, which defines $17$
classes of objects, including surgical tools, anatomical structures, and
miscellaneous. Cataract101 (C101) comprises 101 videos of $720\times 540$
pixels and 25 fps performed by surgeons with various levels of expertise. The
videos are annotated based on the 11 most common phases of cataract surgery
and used with 45-5-50 train-validation-test splits.
Training details Frames are resized to 224x224 to generate DINO-B embeddings.
For graph generation, a frame similarity threshold of 0.9 is chosen. We
trained phase segmentation models for 100 epochs using an Adam optimizer with
a learning rate of 0.0001 and a batch size of 32 on a single A40 GPU.

Metrics For phase segmentation, we compute the accuracy and F1 score. For
semantic segmentation, we measure the mean intersection over union (mIoU) and
pixel-wise accuracy (PAC).

I’m

##  4 Results & Discussion

Surgical Workflow Prediction section 4 presents an ablation study on window
size and spatial and temporal embeddings as well as a comparative analysis of
our proposed method against existing techniques on phase segmentation tasks
for CATARACTS and Cataract101 datasets. We show that increasing window size
together with temporal embeddings improves phase segmentation performance,
while spatial embeddings have minimal impact. Our method demonstrates superior
performance in terms of accuracy and F1 score by indicating $8\%$ accuracy and
$10\%$ F1 score improvement over previous graph-based phase segmentation SOTA
[11]. As there are no other benchmarks on the CATARACTS dataset, we add an
additional non-graph-based solution representing the best practices in the
surgical workflow. It exploits DINO as a strong spatial feature extractor
followed by long-range temporal learning via TCN++[14]. Our dynamic end-to-end
setup outperforms the CNN baseline with $6\%$ accuracy and $4\%$ F1 score. Our
method consistently yields closely competitive results on Cataract101. This
demonstrates the robustness and adaptability of our approach, providing a
solid foundation for further refinement and application in diverse contexts.

Table 1:  Comparison of DMON-based Phase segmentation performance on CATARACTS
[2] and CAT101 [20] datasets. * indicates our implementation.

cccccccc Method Graph SpatialTempWS Accuracy F1

14-1<>CATARACTS DINO-TCN++* [14] Non-graph full video 77.02 74.37
Holm [11] Static 164.34 50.04
Holm [11]Dynamic 30 75.15 68.56

SANGRIA Static 177.53 69.29
SANGRIA Dynamic 475.50 67.76
SANGRIA Dynamic 874.18 68.50
SANGRIA Dynamic 475.96 67.07
SANGRIA Dynamic 4 80.67 75.35
SANGRIA Dynamic 481.62 74.62
SANGRIA Dynamic 8 74.44 64.44
SANGRIA Dynamic 8 81.85 78.47
SANGRIA Dynamic 8 82.13 75.17
SANGRIA Dynamic 16 83.36 78.24

4-1<>C101 ViT [8] Non-graph 184.56 -
TimesFormer [4] Non-graph 890.76 -
GLSFormer [22] Non-graph 892.91 \-

SANGRIA (Ours) Dynamic 8 91.26 85.02
Semantic Segmentation Table 2 indicates that increasing the number of temporal
connections (lowering temporal similarity threshold) in fully-connected graph
increases segmentation performance on both anatomy and tools. Conversely, due
to high computational costs with the increasing length of the temporal window,
fully-connected graphs have limited use for end tasks since surgical phase
recognition performance heavily depends on temporal information. LightGlue
correspondences effectively facilitate better temporal learning while
maintaining comparable performance and considerably reducing the amount of
computation. Table 3 indicates the prior performance of the unsupervised
segmentation performance of our proposed dynamic graph clustering algorithm to
SOTA object discovery method MaskCut [28]. Both tables emphasize the
difficulty of tool localization via mIoUins while mIoUana indicates higher
segmentation performance for anatomical structures. In all setups, DMON
consistently outperforms other solutions. In Figure 3 A, we qualitatively
evaluate MaskCut [28] and variations of our segmentation solution. Our method
leverages temporal connections and predicts temporally consistent semantic
segmentation maps. Unlike MaskCut, it also predicts separate masks for anatomy
and surgical tools.
DSG Generation Figure 3 B manifests improvement of joint optimization of DSG
generation with phase segmentation on surgical tool segmentation. The jointly
optimized model classifies the "Primary Knife" better than the per frame
optimized model. We hypothesize that guidance towards the downstream task
significantly improves attention to tools with high importance in the current
surgical phase. Our DSG also indicates better explainability by highlighting
the importance of graph edges in the end task.

Figure 3: A) Comparison of various graph clustering setups (WS corresponds to
window size). B) End-to-end optimization improves the classification of
’Primary Knife’ since it plays a critical role in predicting the current
phase.

Table 2: Ablation study on the temporal connections.

WS | Temp. Sim | Temp. Conn | PAC | mIoU | mIoUana | mIoUins
---|---|---|---|---|---|---
1 | - | - | 65.34 | 33.86 | 51.44 | 9.46
3 | - | LightGlue | 65.25 | 34.03 | 50.36 | 8.68
3 | 0.9 | FC | 65.21 | 33.79 | 51.27 | 9.51
3 | 0.5 | FC | 65.37 | 34.09 | 51.50 | 9.98

Table 3: Comparison of semantic segmentation on CaDIS sequences.

Pool Method | WS | Temp. Conn | PAC | mIoU | mIoUana | mIoUins
---|---|---|---|---|---|---
Maskcut* | 1 | - | 60.56 | 24.75 | 33.34 | 6.23
MincutPool | 1 | - | 61.77 | 29.82 | 46.00 | 4.32
DMON | 1 | - | 65.34 | 33.86 | 51.44 | 9.46
MincutPool | 3 | LightGlue | 62.66 | 29.84 | 45.04 | 3.20
DMON | 3 | LightGlue | 65.25 | 34.03 | 50.36 | 8.68

##  5 Conclusion

We introduce SANGRIA, an end-to-end graph-based solution for concurrent
surgical workflow recognition, semantic scene segmentation, and dynamic scene
graph generation. Our jointly optimized setup featuring sparse temporal
connections and graph clustering, prioritizes the graph generation for the
downstream task by disambiguating the graph and highlighting the most
influential components and their connections. By focusing on downstream task-
specific features, we achieve state-of-the-art results in surgical phase
segmentation on the CATARACTS dataset while generating scene explanations with
minimal annotation.

####  5.0.1 Acknowledgements

This work was conducted and fully financed by the Corporate Research and
Technology department of Carl Zeiss AG.

####  5.0.2

The authors have no competing interests to declare that are relevant to the
content of this article.

## References

  * [1] Aflalo, A., Bagon, S., Kashti, T., Eldar, Y.: Deepcut: Unsupervised segmentation using graph neural networks clustering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 32–41 (2023)
  * [2] Al Hajj, H., Lamard, M., Conze, P.H., Roychowdhury, S., Hu, X., Maršalkaitė, G., Zisimopoulos, O., Dedmari, M.A., Zhao, F., Prellberg, J., et al.: Cataracts: Challenge on automatic tool annotation for cataract surgery. Medical image analysis 52, 24–41 (2019)
  * [3] ALHAJJ, H., Lamard, M., Conze, P.h., Cochener, B., Quellec, G.: Cataracts (2021). https://doi.org/10.21227/ac97-8m18, https://dx.doi.org/10.21227/ac97-8m18
  * [4] Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: ICML. vol. 2, p. 4 (2021)
  * [5] Bianchi, F.M., Grattarola, D., Alippi, C.: Spectral clustering with graph neural networks for graph pooling. In: International conference on machine learning. pp. 874–883. PMLR (2020)
  * [6] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9650–9660 (2021)
  * [7] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6299–6308 (2017)
  * [8] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2020)
  * [9] Farshad, A., Yeganeh, Y., Chi, Y., Shen, C., Ommer, B., Navab, N.: Scenegenie: Scene graph guided diffusion models for image synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 88–98 (2023)
  * [10] Francesco, L., Dirk, W., Thomas, U., Aravindh, M., Georg, H., Jakob, U., Alexey, D., Thomas, K.: Object-centric learning with slot attention. Adv. Neural Inform. Process. Syst 33, 11525–11538 (2020)
  * [11] Holm, F., Ghazaei, G., Czempiel, T., Özsoy, E., Saur, S., Navab, N.: Dynamic scene graph representation for surgical video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 81–87 (2023)
  * [12] Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
  * [13] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [14] Li, S.J., AbuFarha, Y., Liu, Y., Cheng, M.M., Gall, J.: Ms-tcn++: Multi-stage temporal convolutional network for action segmentation. IEEE transactions on pattern analysis and machine intelligence (2020)
  * [15] Lindenberger, P., Sarlin, P.E., Pollefeys, M.: Lightglue: Local feature matching at light speed. arXiv preprint arXiv:2306.13643 (2023)
  * [16] Murali, A., Alapatt, D., Mascagni, P., Vardazaryan, A., Garcia, A., Okamoto, N., Mutter, D., Padoy, N.: Latent graph representations for critical view of safety assessment. arXiv preprint arXiv:2212.04155 (2022)
  * [17] Murali, A., Alapatt, D., Mascagni, P., Vardazaryan, A., Garcia, A., Okamoto, N., Mutter, D., Padoy, N.: Encoding surgical videos as latent spatiotemporal graphs for object and anatomy-driven reasoning. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 647–657. Springer (2023)
  * [18] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
  * [19] Pissas, T., Ravasio, C.S., Da Cruz, L., Bergeles, C.: Effective semantic segmentation in cataract surgery: What matters most? In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. pp. 509–518 (2021)
  * [20] Schoeffmann, K., Taschwer, M., Sarny, S., Münzer, B., Primus, M.J., Putzgruber, D.: Cataract-101: video dataset of 101 cataract surgeries. In: Proceedings of the 9th ACM multimedia systems conference. pp. 421–425 (2018)
  * [21] Sestini, L., Rosa, B., De Momi, E., Ferrigno, G., Padoy, N.: Fun-sis: A fully unsupervised approach for surgical instrument segmentation. Medical Image Analysis 85, 102751 (2023)
  * [22] Shah, N.A., Sikder, S., Vedula, S.S., Patel, V.M.: Glsformer: Gated-long, short sequence transformer for step recognition in surgical videos. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 386–396. Springer (2023)
  * [23] Sharma, S., Nwoye, C.I., Mutter, D., Padoy, N.: Rendezvous in time: an attention-based temporal fusion approach for surgical triplet recognition. International Journal of Computer Assisted Radiology and Surgery pp. 1–7 (2023)
  * [24] Sharma, S., Nwoye, C.I., Mutter, D., Padoy, N.: Surgical action triplet detection by mixed supervised learning of instrument-tissue interactions. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 505–514. Springer (2023)
  * [25] Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence 22(8), 888–905 (2000)
  * [26] Tsitsulin, A., Palowitch, J., Perozzi, B., Müller, E.: Graph clustering with graph neural networks. Journal of Machine Learning Research 24(127), 1–21 (2023)
  * [27] Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N.: Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging 36(1), 86–97 (2016)
  * [28] Wang, X., Girdhar, R., Yu, S.X., Misra, I.: Cut and learn for unsupervised object detection and instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3124–3134 (2023)
  * [29] Wang, X., Misra, I., Zeng, Z., Girdhar, R., Darrell, T.: Videocutler: Surprisingly simple unsupervised video instance segmentation. arXiv preprint arXiv:2308.14710 (2023)
  * [30] Wang, Y., Shen, X., Yuan, Y., Du, Y., Li, M., Hu, S.X., Crowley, J.L., Vaufreydaz, D.: Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
  * [31] Zhao, Z., Jin, Y., Gao, X., Dou, Q., Heng, P.A.: Learning motion flows for semi-supervised instrument segmentation from robotic surgical video. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23. pp. 679–689. Springer (2020)

##  Appendix 0.A SANGRIA: Supplementary Material

Table 1: Comparison of fully-supervised segmentation(OCRNET[19])-based [11] and SANGRIA (Ours) on scene graph node classification on CaDIS sequences. While the supervised model demonstrates superior performance, our weakly-supervised approach offers promsing results on most influential objects within a surgical scene and highlight the importance of their interactions. Class | Base[11] | End-to-end |
---|---|---|---
Precision | Recall | F1-Score | Precision | Recall | F1-Score | Support
Pupil | 1.00 | 1.00 | 1.00 | 1.00 | 0.98 | 0.99 | 1381
Tape | 0.94 | 0.96 | 0.95 | 0.93 | 0.47 | 0.62 | 1101
Hand | 0.64 | 0.32 | 0.43 | 0.00 | 0.00 | 0.00 | 218
Retractors | 0.93 | 0.91 | 0.92 | 1.00 | 0.02 | 0.04 | 1030
Iris | 1.00 | 1.00 | 1.00 | 1.00 | 0.78 | 0.88 | 1382
Skin | 1.00 | 1.00 | 1.00 | 1.00 | 0.97 | 0.98 | 1381
Cornea | 1.00 | 1.00 | 1.00 | 1.00 | 0.99 | 0.99 | 1382
Cannula | 0.80 | 0.43 | 0.56 | 0.95 | 0.07 | 0.12 | 628
Cystotome | 0.34 | 0.55 | 0.42 | 0.06 | 0.18 | 0.09 | 33
T. Forceps | 0.77 | 0.43 | 0.55 | 1.00 | 0.00 | 0.02 | 227
Pr. Knife | 0.79 | 0.32 | 0.46 | 0.66 | 0.14 | 0.24 | 228
Phaco | 0.50 | 1.00 | 0.67 | 0.03 | 0.03 | 0.33 | 4
Lens Inj. | 0.86 | 0.45 | 0.59 | 0.98 | 0.27 | 0.42 | 223
I/A | 0.19 | 0.82 | 0.31 | 0.50 | 0.29 | 0.37 | 19
Sec. Knife | 0.66 | 0.30 | 0.42 | 1.00 | 0.01 | 0.02 | 187
Micromanip. | 0.14 | 0.45 | 0.22 | 0.00 | 0.00 | 0.00 | 13
Cap. Forceps | 0.08 | 0.15 | 0.11 | 0.00 | 0.00 | 0.00 | 20
micro avg | 0.94 | 0.87 | 0.91 | 0.96 | 0.61 | 0.75 | 9457
macro avg | 0.68 | 0.65 | 0.62 | 0.65 | 0.32 | 0.34 | 9457
weighted avg | 0.94 | 0.87 | 0.89 | 0.95 | 0.61 | 0.66 | 9457
Figure 1: Comparison of phase segmentation performance on CATARACTS [3, 2]
test videos. A) Ground Truth phases. B) Predictions of the best dynamic model
of [11]. C) Best model of the SANGRIA(Ours). Our model predicts surgical
phases such as Irrigation/Aspiration, OVD Aspiration and Nucleus Breaking more
consistently thanks to scene representation and understanding capabilities of
the SANGRIA. Table 2: Comparison of per frame optimization of MincutPool [5]
and DMON [26] with our end2end solution on scene graph node classification of
CaDIS sequences. The end-to-end model significantly improves the node
classification performance of the Lens Injector, Irrigation/Aspiration tools,
which are the main components in OVD Aspiration, Irrigation/Aspiration, and
Implant Injection phases. DMON consistently demonstrates better scene graph
node classification performance than MincutPool.

Class | MincutPool | Dmon | End-to-end |
---|---|---|---|---
Precision | Recall | F1-Score | Precision | Recall | F1-Score | Precision | Recall | F1-Score | Support
Pupil | 1.00 | 0.93 | 0.97 | 1.00 | 0.96 | 0.98 | 1.00 | 0.98 | 0.99 | 1381
Tape | 0.94 | 0.54 | 0.69 | 0.93 | 0.60 | 0.73 | 0.93 | 0.47 | 0.62 | 1101
Hand | 0.91 | 0.29 | 0.44 | 0.95 | 0.38 | 0.54 | 0.00 | 0.00 | 0.00 | 218
Retractors | 1.00 | 0.03 | 0.05 | 1.00 | 0.03 | 0.06 | 1.00 | 0.02 | 0.04 | 1030
Iris | 1.00 | 0.71 | 0.83 | 1.00 | 0.81 | 0.90 | 1.00 | 0.78 | 0.88 | 1382
Skin | 1.00 | 0.88 | 0.94 | 1.00 | 0.94 | 0.97 | 1.00 | 0.97 | 0.98 | 1381
Cornea | 1.00 | 0.94 | 0.97 | 0.90 | 0.98 | 0.99 | 1.00 | 0.99 | 0.99 | 1382
Cannula | 0.96 | 0.04 | 0.08 | 0.93 | 0.02 | 0.05 | 0.95 | 0.07 | 0.12 | 628
Cystotome | 0.09 | 0.15 | 0.11 | 0.08 | 0.15 | 0.11 | 0.06 | 0.18 | 0.09 | 33
T. Forceps | 0.98 | 0.18 | 0.30 | 1.00 | 0.30 | 0.47 | 1.00 | 0.00 | 0.02 | 227
Pr. Knife | 0.55 | 0.10 | 0.17 | 0.54 | 0.11 | 0.19 | 0.66 | 0.14 | 0.24 | 228
Phaco | 0.00 | 0.00 | 0.00 | 0.02 | 0.25 | 0.03 | 0.03 | 0.33 | 0.03 | 4
Lens Inj. | 0.93 | 0.17 | 0.29 | 0.98 | 0.23 | 0.37 | 0.98 | 0.27 | 0.42 | 223
I/A | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.50 | 0.29 | 0.37 | 19
Sec. Knife | 1.00 | 0.06 | 0.11 | 0.79 | 0.06 | 0.11 | 1.00 | 0.01 | 0.02 | 187
Micromanip. | 0.03 | 0.15 | 0.05 | 0.04 | 0.18 | 0.06 | 0.00 | 0.00 | 0.00 | 13
Cap. Forceps | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 20
micro avg | 0.96 | 0.59 | 0.73 | 0.95 | 0.64 | 0.77 | 0.96 | 0.61 | 0.75 | 9457
macro avg | 0.67 | 0.30 | 0.35 | 0.66 | 0.35 | 0.40 | 0.65 | 0.32 | 0.34 | 9457
weighted avg | 0.97 | 0.59 | 0.66 | 0.96 | 0.64 | 0.70 | 0.95 | 0.61 | 0.66 | 9457