# T-WaveNet$:$ Tree-Structured Wavelet Neural Network for Sensor-Based
Time Series Analysis

Minhao Liu1111Contact Author Ailing Zeng1 Qiuxia Lai1&Qiang Xu1
1The Chinese University of Hong Kong
{mhliu, alzeng, qxlai<EMAIL_ADDRESS>
###### Abstract

Sensor-based time series analysis is an essential task for applications such
as activity recognition and brain-computer interface. Recently, features
extracted with deep neural networks (DNNs) are shown to be more effective than
conventional hand-crafted ones. However, most of these solutions rely solely
on the network to extract application-specific information carried in the
sensor data. Motivated by the fact that usually a small subset of the
frequency components carries the primary information for sensor data, we
propose a novel tree-structured wavelet neural network for sensor data
analysis, namely _T-WaveNet_. To be specific, with T-WaveNet, we first conduct
a power spectrum analysis for the sensor data and decompose the input signal
into various frequency subbands accordingly. Then, we construct a tree-
structured network, and each node on the tree (corresponding to a frequency
subband) is built with an invertible neural network (INN) based wavelet
transform. By doing so, T-WaveNet provides more effective representation for
sensor information than existing DNN-based techniques, and it achieves state-
of-the-art performance on various sensor datasets, including UCI-HAR for
activity recognition, OPPORTUNITY for gesture recognition, BCICIV2a for
intention recognition, and NinaPro DB1 for muscular movement recognition.

##  1 Introduction

Sensors play an essential role in our everyday life. For example, inertial
sensors in the smartwatch are used for human activity logging Chen et al.
(2020). Electroencephalography (EEG) signals for brain intention
classification can help disabled people communicate effectively Zhang et al.
(2020). Surface electromyography signals are widely used in muscular movement
analysis for clinical diagnosis and prosthetic device control Ahmad Nasrul and
Mohd Hanafi (2009).

Generally speaking, sensor-based time series analysis consists of three steps:
(i). _data segmentation_ , wherein the continuous sensor data is partitioned
into segments using fixed- or variable-sized windows; (ii). _feature
extraction_ , wherein various techniques are applied on each segment to
extract discriminative features.; and (iii). _downstream tasks_ , which
perform the required tasks (e.g., classification or prediction) with the
extracted features. No doubt to say, feature extraction is the most critical
step among them.

Sensor features can be broadly categorized as hand-crafted features and deep
learning-based features. The former includes temporal features (e.g., the mean
and the variance of the sensor signal) and frequency features obtained by
performing signal processing techniques such as discrete wavelet transform
(DWT), mean frequency (MNF), and power spectrum ratio (PSR). Temporal features
are simple to calculate Janidarmian et al. (2017); Qian et al. (2019), but
they are not effective for complicated tasks. By contrast, frequency features
are more informative by preserving the context information in different
frequency components Atzori et al. (2014); Phinyomark et al. (2013); Duan et
al. (2015), but the feature extraction process is often laborious and time-
consuming. Recently, deep neural networks (DNNs) have become the dominant
approaches for feature extraction in various fields, and DNN-based features
also show superiority for sensor-based time-series analysis, e.g., Rahimian et
al. (2020); Ignatov (2018); Yao et al. (2017).

Existing DNN-based solutions typically extract features from the raw sensor
signals directly without considering different frequency components’ roles. In
practice, a small subset of the frequency components always carries the
primary information for sensor data, known as the dominant energy range of the
signal Rioul and Vetterli (1991). Motivated by it, we take signal energy
ranges into consideration and propose a tree-structured wavelet neural
network, namely _T-WaveNet_ , to extract more effective features. The main
contributions of this work are threefold:

  * •

We perform _power spectrum analysis_ to decompose the input signals into
multiple frequency subbands and utilize them to configure the tree structure
in T-WaveNet. Such flexible design is applicable to various types of sensor
data.

  * •

The tree node in T-WaveNet is realized as a novel invertible neural network
(INN) based wavelet transform unit, which can effectively separate the input
signal’s frequency subbands. To the best of our knowledge, this is the first
attempt to model the wavelet transform using INN.

  * •

Finally, we propose an instrumental _feature fusion module_ according to the
feature dependencies across different frequency components to deal with the
personalized heterogeneity from the sensor signal.

With the above, T-WaveNet enhances task-related subband feature extraction,
thereby dramatically improving the performance of downstream tasks. Extensive
experiments on four real-world sensor-based recognition tasks, namely activity
recognition, gesture recognition, intention recognition, and muscular movement
recognition, show that T-WaveNet consistently outperforms the state-of-the-art
solutions by a considerable margin.

In Section 2, we give a brief overview of related works. Section 3 details our
proposed T-WaveNet. Next, we introduce performance metrics and our
experimental settings in Section 4 and compare T-WaveNet with state-of-the-art
methods and conduct ablation study in Section 5. Finally, Section 6 concludes
this work.

Figure 1: Given an input signal, we first perform (a) Power Spectrum Analysis
to decompose it into multiple frequency subbands with comparable energy, where
each subband contains at most one formant. Then, based on these frequency
subbands, we construct (b) T-WaveNet, a tree-structured network, where each
node is a (c) Frequency Bisection Operator built with an INN-based wavelet
transform unit. The operator outputs the high- and low-frequency components of
the input if its binary gate value is “1”, and otherwise bypass the input.
Considering the personalized heterogeneity of the input, we utilize a (d)
Feature Fusion Module to fuse the subband features $\\{h_{i}\\}$ according to
the feature dependencies across various frequency components.

##  2 Related Work

Sensor-based time-series analysis is a common task and has been studied in
applications such as human activity recognition Geng et al. (2016), brain
activity identification Lawhern et al. (2018), heart rate monitoring Labati et
al. (2019), and muscular assessment Wei et al. (2019a). Our proposed T-WaveNet
model advances the development of sensor-based time series analysis by
addressing two key challenges: (1) Feature extraction of time-series sensor
data, and (2) Effective modeling of wavelet transform.

###  2.1 Feature Extraction of Sensor Data

Sensor features can be broadly classified into hand-crafted features and deep
learning-based features, and the former can be further divided into temporal
and frequency features.

Temporal features (e.g., mean, variance, and root mean square (RMS)) are
widely used in early studies Vepakomma et al. (2015); Janidarmian et al.
(2017); Qian et al. (2019). Though simple to calculate, these temporal
features are not effective for complicated tasks. Frequency features are more
informative than temporal features because they preserve the context
information in different frequency components, and have shown improved
performances various tasks. Duan et al. (2015) performs discrete wavelet
transform on surface-Electromyography (sEMG) signal for hand motion
classification. Geng et al. (2016); Wei et al. (2019b); Jiang and Yin (2015);
Li et al. (2017) applies the Short-time Discrete Fourier Transform (STDFT) to
extract features for human activity recognition. However, this is often a
labor-intensive and time-consuming process Qian et al. (2019); Wang et al.
(2019).

Recently, deep learning-based features have shown superiority in various tasks
in both performance and efficiency. For example, Zeng et al. (2014); Yang et
al. (2015); Ignatov (2018); Zeng et al. (2020); Lee et al. (2017); Xing et al.
(2019) consider using convolutional neural networks (CNNs) to extract feature
along time dimension. Guan and Plötz (2017); Yao et al. (2017); Chen et al.
(2016); Ordóñez and Roggen (2016) combine CNNs with Recurrent neural networks
(RNNs) to extract local and long-term temporal dependency features. Later,
Zhang et al. (2020) integrates the graph neural network to represent the
positioning information of EEG nodes, which achieves superior performance.

However, the existing methods usually extract features from the raw sensor
data, and do not distinguish the roles of various frequency components. In
contrast, our proposed framework not only utilizes deep learning techniques
for efficient feature extraction, but alleviates the above problems by
constructing a tree-structured network with the input of multiple split
frequency subbands to differentiate the fused components in sensor data
analysis.

##  3 Proposed T-WaveNet Solution

Instead of solely relying on the network structure for information extraction
from sensor data, our T-WaveNet utilize the knowledge within the data itself
to construct a task-specific tree-structured network for feature learning.

An overview of the _T-WaveNet_ is shown in Fig. 1. We first perform a power
spectrum analysis on the sensor data (Fig. 1(a)) to decompose it into multiple
frequency subbands with comparable energy, where each subband contains at most
one formant. Then, based on these frequency subbands, we construct a tree-
structured network (Fig. 1(b)), where each node is a frequency bisection
operator (Fig. 1(c)) built with an INN-based wavelet transform unit. The
operator is conditioned on a binary gate. It outputs the high-frequency and
low-frequency components of the input if the gate value is “1”, and otherwise
bypass the input. Additionally, since different users have their unique style,
resulting in the diversity of signal patterns. Thus, to deal with such
personalized heterogeneity problem, after obtaining the set of leaf features
for each frequency subband, we utilize an effective feature fusion method
which enhances the task-related subband features with larger fusion weights
(Fig. 1(d)).

###  3.1 Power Spectrum Analysis

Considering that only a small subset of the frequency components carries the
primary information for sensor data, we first perform power spectrum analysis
to obtain the set of frequency subbands that covers the dominant energy range
of the signal, where each subband contains approximately equivalent energy.
Then, we construct T-WaveNet, the tree-structured network wherein each of the
leaf nodes corresponds to a frequency subband.

Alg. 1 shows the power spectrum analysis consisting of two phases: a _formant-
guided frequency band splitting_ phase to obtain the initial subbands set, and
an _energy-guided advanced subband splitting_ to further balance the energy of
each subband. In the first phase, given the input signal $X\in\mathbb{R}^{N}$,
we first calculate the spectrum with Fourier transform (Fig. 1(a)). Next, we
obtain the set of formants $\mathcal{P}=\\{f_{p}\\}_{p}$, where $f_{p}$ is a
local maximum of the envelope of spectrum. Such formants represent the most
direct source of the signal information. Then, we recursively bisecting the
frequency band until there is at most one formant falls in each frequency
subband $[f_{s},f_{e}]$, where $f_{s}$ and $f_{e}$ are the starting and ending
frequency, respectively. All these subbands are collected in set
$\mathcal{Q}$. In the second phase, we calculate the energy of each subband in
$\mathcal{Q}$ with Eq.(1), and recursively bisecting the subband whose energy
exceeds twice the minimum subband energy $E_{\text{min}}$, which is to ensure
each subband contains a relatively same amount of information. Because we
argue that _evenly-distributed information would enhance network
representation ability and reduce the burden of the learning process_.

$Energy=\int_{f_{s}}^{f_{e}}Amp\left(f\right)d_{f}$ $None$

where $Amp(f)$ is the amplitude of the frequency $f$.

Algorithm 1 Power Spectrum Analysis

1:

2:Phase 1 - Formants guided frequency band splitting

3:

4:Input signal $X$, frequency subband set $\mathcal{Q}=\emptyset$

5:$\mathcal{Q}$ (updated)

6:procedure Itv_Bisect($[f_{s},f_{e}],\mathcal{P},\mathcal{Q}$)

7: $f_{m}=(f_{s}+f_{e})/2$

8: if Count$(f_{p}\\!\in\\![f_{s},f_{m}])\\!>\\!1$ for
$f_{p}\\!\in\\!\mathcal{P}$ then

9: Itv_Bisect($[f_{s},f_{m}],\mathcal{P},\mathcal{Q}$)

10: else

11: $\mathcal{Q}\leftarrow\mathcal{Q}\cup\\{[f_{s},f_{m}]\\}$

12: end if

13: if Count$(f_{p}\\!\in\\![f_{m},f_{e}])\\!>\\!1$ for
$f_{p}\\!\in\\!\mathcal{P}$ then

14: Itv_Bisect($[f_{m},f_{e}],\mathcal{P},\mathcal{Q}$)

15: else

16: $\mathcal{Q}\leftarrow\mathcal{Q}\cup\\{[f_{m},f_{e}]\\}$

17: end if

18: return $\mathcal{Q}$

19:end procedure

20:$Amp(f)\leftarrow FFT(X),f\\!\in\\![0,F]$ $\triangleright$ Perform Fourier
transform on $X$ to get the power spectrum

21:Obtain the set of formants $\mathcal{P}=\\{f_{p}\\}_{p}$ $\triangleright$
The local maximum of the envelope of $Amp(f)$.

22:Itv_Bisect($[0,F],\mathcal{P},\mathcal{Q}$)

23:

24:Phase 2 - Energy guided advanced subband splitting

25:

26:$\mathcal{Q}=\\{[f_{i},f_{i+1}]\\}_{i}$, $Amp(f)$

27:$\mathcal{Q}$ (updated)

28:procedure
E_Bisect($[f_{i},f_{i+1}],E_{i},E_{\text{min}},\mathcal{Q},Amp(f)$)

29: if $E_{i}\\!>\\!2*E_{\text{min}}$ then

30: $f_{m}=(f_{i}+f_{i+1})/2$

31: $\mathcal{Q}\leftarrow\mathcal{Q}/\\{[f_{i},f_{i+1}]\\}$

32: $\mathcal{Q}\leftarrow\mathcal{Q}\cup\\{[f_{i},f_{m}],[f_{m},f_{i+1}]\\}$

33: $E_{i1}\leftarrow\int_{f_{i}}^{f_{m}}Amp(f)df$

34: $E_{i2}\leftarrow\int_{f_{m}}^{f_{i+1}}Amp(f)df$

35: E_Bisect($[f_{i},f_{m}],E_{i1},E_{\text{min}},\mathcal{Q}$)

36: E_Bisect($[f_{m},f_{i+1}],E_{i2},E_{\text{min}},\mathcal{Q}$)

37: end if

38: return $\mathcal{Q}$

39:end procedure

40:for $[f_{i},f_{i+1}]\\!\in\\!\mathcal{Q}$ do

41: $E_{i}\leftarrow\int_{f_{i}}^{f_{i+1}}Amp(f)df$

42:end for

43:$\mathcal{E}\equiv\\{E_{i}\\},E_{\text{min}}\leftarrow\text{min
}\mathcal{E}$

44:for $E_{i}\\!\in\\!\mathcal{E}$ do

45: E_Bisect($[f_{i},f_{i+1}],E_{i},E_{\text{min}},\mathcal{Q}$)

46:end for

After obtaining the subband set $\mathcal{Q}$, we construct the tree-
structured network of T-WaveNet in three steps: (i). bottom-up marking; (ii).
top-down completion; and (iii). _pruning_. In the bottom-up process, given a
full binary frequency tree with some height (see the dashed tree structure in
Fig. 1(b)), we locate each subband in $\mathcal{Q}$ and set its binary gate as
“0”, which serves as the leaf nodes. Then, we set the binary gates of all
other nodes on the path from the leaves to the root as “1”. In the top-down
process, for all the nodes with gate “1”, we set the gates of their children
as “0” if the gates have not been configured in the previous process. Finally,
the nodes without gate settings are pruned from the tree. The resulted sub-
tree of the full binary tree preserves the leaf nodes of the informative
frequency subbands in $\mathcal{Q}$, as well as some closely-related frequency
components that are not in $\mathcal{Q}$ (i.e., some nodes added in the top-
down process) for robustness and tolerances.

###  3.2 Frequency Bisection Operator

We build our wavelet transform in the frequency bisection operator based on
the so-called _Lifting Scheme_ Sweldens (1998), known as the second-generation
wavelets. It is a simple and powerful approach to construct different
wavelets, e.g., _Haar_ Haar (1909). The main idea is to utilize the strong
correlation among the neighboring samples on the signal to separate the low-
frequency (approximation) and high-frequency subband (details), respectively.
The workflow of the Lifting Scheme contains three stages. Assuming the input
vector is $X=(x[0],x[1],...,x[N])$.

  * •

Splitting. The signal is split into two non-overlapping partitions. The
general partition method is dividing the signal to even part
$X_{even}=(x[0],x[2],...,x[2k])$ and odd part
$X_{odd}=(x[1],x[3],...,x[2k-1])$ , $k\in N/2$. The spliting operater is:

$(X_{even},X_{odd})=Splitting(X)$ $None$

  * •

Predictor. The intersections of two partition sets are distributed in the
original signal. Based on the signal correlation, given one of them, it is
possible to build a good predictor P for the other set. $d$ means the
difference between the given and the prediction set. This step is also called
“dual lifting”.

$d=X_{odd}-P(X_{even})$ $None$

  * •

Updator. Using the odd part to update the even part with updator U in order to
keep some consistent characteristics of the original signal, e.g., mean,
higher moments. This step is called ”primal lifting”.

$c=X_{even}+U(d)$ $None$

However, such wavelet construction framework with fixed coefficients ($P,U$)
is suboptimal and restricts the adaptability in DNNs. Therefore, we take
advantage of the invertible neural networks (INN), a bijective transformation,
which can effectively build correlations between inputs and outputs with
learnable structures. Therefore, to adjust the Lifting Scheme to an adaptive
wavelet transform, the improvements include: (i). we revise the Eq. $(3),(4)$
to an affine function as Eq. $(5),(6)$, which can enhance the transformation
ability; (ii). we realize the coefficients($\phi$, $\psi$,$\rho$ and $\eta$ )
with convolutional layers. By doing so, our adaptive wavelet transformation
can learn the wavelet coefficients from the data directly, which should be
more optimal for input signals.

$d=X_{odd}\odot exp(\phi(X_{even}))-\rho(X_{even})$ $None$ $c=X_{even}\odot
exp(\psi(d))+\eta(d)$ $None$

where $\phi$ and $\psi$ stand for scale, $\rho$ and $\eta$ stand for
translation. $\odot$ is the element-wise production. As shown in Fig. 1(c).
Note that the $exp(\cdot)$ is omitted in Fig. 1(c) around the function $\phi$
and $\psi$.

Datasets | Classes | Window size | Sensor types | Feature number | Sampling rate(Hz) | Subjects
---|---|---|---|---|---|---
OPPORTUNITY | 18 | 48 | Inertial Sensors | 77 | 30 | 4
UCI-HAR | 6 | 128 | Inertial Sensors | 9 | 50 | 30
BCICIV2a | 4 | 400 | Electrodes | 22 | 250 | 9
NinaPro DB1 | 52 | 150 | Electrodes | 10 | 100 | 27
Table 1: The overall information of the four datasets. “Window size” means the
length of the input signal used in the experiments. “Features number” denotes
the input channel in each dataset.

###  3.3 Adaptive Feature Fusion

Considering the personalized heterogeneity of the sensor data, the feature
extracted from different frequency components should have different
contributions to a specific task. Therefore, we employ an efficient feature
fusion module which is realized by self-attention mechanism Vaswani et al.
(2017). The module learns the feature dependencies across various frequency
components and enhance the more task-related features by assigning higher
weights. Following Vaswani et al. (2017), we describe our attention model as a
mapping from a query and a set of key-value pairs to an output, as illustrated
in Fig. 1(d). The sub-band features $h\in\mathbb{R}^{c\times m}$ extracted
from tree structure are first used to form the query and key-value pairs.

$q=W_{q}h+b_{q},q\in\mathbb{R}^{d\times m},$
$k=W_{k}h+b_{k},k\in\mathbb{R}^{d\times m},$ $None$
$v=W_{v}h+b_{v},v\in\mathbb{R}^{d\times m}$

Where $W_{q},W_{k},W_{v}\in\mathbb{R}^{d\times c}$ and
$b_{q},b_{k},b_{v}\in\mathbb{R}^{d}$ are the weight matrix and bias. $m$ and
$c$,$d$ denote the number of features and feature dimensions, respectively.
For each key, we then compute an attention score as

$\alpha=softmax(k^{T}\cdot q),\alpha\in\mathbb{R}^{m\times m}$ $None$

Then the output of the attention module is computed by the weighted sum of the
attention score and the values.

$H=\alpha\cdot v,H\in\mathbb{R}^{m\times d}$ $None$

Therefore, the attentive frequency sub-band representation $H$ is fed into a
standard softmax classifier:

$p_{i}=softmax(WH+b)$ $None$

where $W$ and $b$ are the weight matrix and bias, respectively. $p_{i}$ is the
predicted probability for class $i$.

###  3.4 Loss Function

In addition to employing the Cross-Entropy loss function for classification,
we also explore the utility of the regularization term to constrain the
wavelet decomposition during training. The loss function is shown in
Eqn.$(11)$, where $C$ denotes the number of classes, $y_{i}$ is the binary
indicator which equals $1$ if class label $i$ is the correct classification
for observation. The regularization term ensures that for each frequency
bisection operator, the mean value of the decomposition output $c_{j}$ is
close to that of the input $x_{j}$ Rodriguez et al. (2020). $M$ is the total
number of operators in the tree-structured network. $\lambda$ tunes the
strength of the regularization term.

$Loss=-\sum_{i}^{C}y_{i}log(p_{i})+\lambda\sum_{j}^{M}\left\|x_{j}-c_{j}\right\|_{2}$
$None$

##  4 Experimental Setup

(a) UCI-HAR

(b) BCICIV2a

(c) NinaPro DB1

Figure 2: The network structure of different signal types based on the power
spectrum analysis. The dominant frequency subbands based on formants are
highlighted with red color. (unit:Hz)

###  4.1 Datasets

We conduct experiments on four datasets, namely OPPORTUNITY dataset (OPPOR)
Sagha et al. (2011), UCI-HAR Anguita et al. (2013), the BCICIV2a Brunner et
al. (2008), and NinaPro DB1 Atzori et al. (2012). The overall statistics of
the datasets are listed in Table 1. OPPOR consists of annotated recordings
from on-body sensors when carrying out common gestures of kitchen activities
(e.g., Open Door and Close Door). We follow the settings of the sporadic
gestures task in OPPORTUNITY challenge Chavarriaga et al. (2013). UCI-HAR
collects sensor data of $6$ activities (walking, walking upstairs, walking
downstairs, sitting, standing, laying) from $30$ volunteers aged 19 to 48
years old. We follow the official dataset configuration
222https://archive.ics.uci.edu. BCICIV2a contains EEG signals from $9$ healthy
subjects performing four movement intention tasks (left hand, right hand,
feet, and tongue), which are bandpass-filtered between 0.5Hz and 100Hz. We use
the same settings as Zhang et al. (2020). NinaPro DB1 contains sparse multi-
channel sEMG recordings for hand prostheses, and we configure this dataset
following Rahimian et al. (2020).

Methods | UCI HAR | OPPOR | NinaPro | BCICIV2a
---|---|---|---|---
$F_{m}$ | $F_{w}$ | $Accuracy$ | $F_{m}$ | $F_{w}$ | $Accuracy$ | $Accuracy$
DeepConvLSTMOrdóñez and Roggen (2016) | 0.9032 | 0.9054 | 0.9080 | 0.704 | 0.915 | - | -
CNNIgnatov (2018) | - | - | 0.9235 | - | - | - | -
D2CLXi et al. (2018) | - | - | - | 0.7107 | 0.9197 | - | -
Harmonic Hu et al. (2020) | 0.9312 | 0.9292 | 0.9298 | 0.5752 | 0.8944 | - | -
FilterNetChambers and Yoder (2020) | - | - | - | 0.7430 | 0.9280 | - | -
MV-CNNWei et al. (2019a) | - | - | - | - | - | 0.8740 | -
XceptionTimeRahimian et al. (2020) | - | - | - | - | - | 0.9181 | -
EEgNetLawhern et al. (2018) | - | - | - | - | - | - | 0.5130
NG-CRAMZhang et al. (2020) | - | - | - | - | - | - | 0.6011
T-WaveNet-Haar | 0.8918 | 0.8925 | 0.8924 | 0.6437 | 0.9083 | 0.8503 | 0.4312
T-WaveNet (without feature fusion) | 0.9521 | 0.9516 | 0.9539 | 0.7473 | 0.9277 | 0.9201 | 0.6103
T-WaveNet | 0.9642 | 0.9638 | 0.9705 | 0.7633 | 0.9310 | 0.9321 | 0.6301
Table 2: Overall comparison results on the four datasets. “Haar” means the
frequency bisection operator is replaced by traditional haar wavelet.

###  4.2 T-WaveNet Configurations

Based on the power spectrum analysis, the decomposed frequency subbands and
the tree-structured network for each dataset is detailed as follows. The
configuration of OPPOR dataset is shown in Fig 1(b), which is configured based
on the spectrum of Fig. 1(a), and those for the other three datasets are
illustrated in Fig. 2. As for the power spectrum of UCI HAR, the most dominant
frequency range is below 12.5 Hz Karantonis et al. (2006). Also, due to the
subject-specific motor imagery CR et al. (2011), the dominant energy range of
BCICIV2a falls in various frequency bands, including $\delta$ ($0.1$-$3$ Hz),
$\theta$ ($4$-$7$ Hz), $\alpha$ ($8$-$13$Hz), $\beta$ ($14$-$30$ Hz) and
$\gamma$ ($31$-$100$Hz). For NinaPro DB1, the frequency bandwidth of sEMG
during muscle contraction is $20$-$2000$Hz Jamaluddin et al. (2016) and the
motion artifact noise falls mostly in $0$-$20$Hz Jamal (2012). Considering
that the sampling rate of NinaPro DB1 is only $100$Hz, we concentrate more on
signal within $20$-$50$Hz (_Nyquist_ Grenander (1959)).

###  4.3 Network Details

Frequency bisection operator. The $\phi$, $\psi$, $\rho$, and $\eta$ in Eqn.
$(4)$ and $(5)$ are modeled as DNN modules. For the T-WaveNet used to
obtaining the results in Table 2, the module structure is:
Conv1D($3,1$)$\rightarrow$LeakyRelu($\alpha=0.01$)$\rightarrow$Dropout($rate=50\%$)$\rightarrow$Conv1D($3,1$)$\rightarrow$Tanh.
The output channel of the first and the second convolutions are three times
and one times the input channel, respectively.

Feature fusion module. The dimension of key/query/value $d$ in Eqn. $(6)$ is
$32$. The input channel $c$ is different for each dataset, as listed in Table
1.

Classifer. The structure is: FC($32,1024$) $\rightarrow$ BatchNorm
$\rightarrow$ LeakyRelu($\alpha=0.01$)
$\rightarrow$FC($1024,\\#class$)$\rightarrow$Softmax.

###  4.4 Training Details

All the experiments are run on a single Nvidia GTX 1080 Ti. The batch size is
$64$ for all datasets. We use Adam optimizer Kingma and Ba (2015) with initial
learning rate $3*10^{-4}$ and the decay is $0.95$. The maximum number of
training epoch is $100$. $\lambda$ in Eqn. $(11)$ is $0.1$ for all the
experiments. The input channel $c$ and the sliding window size of each dataset
are listed in Table 1.

###  4.5 Evaluation Metrics

Following previous works, we use $Accuracy$, $weighted$ $F1$ $score$ $(F_{w})$
and $macro(mean)$ $F1$ $score$ $(F_{m})$ as our evaluation metrics. The
definitions are as follows:

$Accuracy=\frac{Number\,of\,correct\,classification}{Total\,number\,of\,test\,samples}$
$None$

$F_{w}=2\sum_{i=0}^{C-1}w_{i}\frac{precision_{i}\times
recall_{i}}{precision_{i}+recall_{i}}$ $None$
$F_{m}=\frac{2}{C}\sum_{i=0}^{C-1}\frac{precision_{i}\times
recall_{i}}{precision_{i}+recall_{i}}$ $None$

where $i$ is the class index, $w_{i}=N_{i}/\sum_{i=0}^{C-1}N_{i}$ is the
proportion of samples of the class, and $N_{i}$ is the number of samples in
$i$th class. $C$ is the total number of the class.

##  5 Experimental Results

In this section, we present detailed experimental results to demonstrate the
advantages of T-WaveNet design.

Figure 3: The $F_{m}$ results of different energy divisions in the OPPOR
dataset.

###  5.1 Comparison with the State-of-the-Arts

To evaluate the overall performance of the proposed model, we compare our
model with other methods in each dataset. The results are shown in Table 2
where scores of other methods are mainly from the original papers. Our
proposed T-WaveNet achieves the best performance on all datasets.
Specifically, our method yields relative $4.38\%$, $3.72\%$, $3.54\%$
improvements of the $accuracy$, $F_{w}$ score, $F_{m}$ score metrics over the
$2$nd best method Hu et al. (2020) on UCI-HAR, respectively. For OPPOR, we
improve over the $2$nd best method Chambers and Yoder (2020) by relative
$2.66\%$. Moreover, we achieve relative $1.52\%$ and $4.81\%$ improvements on
NinaPro and BCICIV2a, respectively.

###  5.2 Ablation Study

We study the effectiveness of each proposed components by performing a
detailed ablation study on the challenging and popular OPPOR dataset.

Effectiveness of power spectrum analysis. In Alg. 1, we first conduct a
formant-guided subband splitting process to obtaining the initial subband set
$\mathcal{Q}$, and then perform an energy-guided advanced subband splitting.
Such subband splitting scheme ensures that each resulted subband contains
roughly equivalent energy (information). To verify the effectiveness of power
spectrum analysis, we train the variants of the T-WaveNet by altering the
constraints about the formants and the energy in each decomposed frequency
subband, resulting in different levels of tree-structured networks. Fig.3
shows the network structures and the corresponding $F_{m}$ scores. The default
tree structure (Fig.3(f)) outperforms all other variants with either coarser
(shallower trees in Fig.3(a-e)) or finer energy division (deeper trees in
Fig.3(g-h)). Based on the above results, the set of frequency subbands with
evenly distributed energy is beneficial for the representation learning of
sensor data, and the proposed solution facilitates to find such configuration
effectively.

Impact of frequency bisection operator. Our frequency bisection operator is
built as an INN-based wavelet transform.

To verify the impact of such design, we first experiment with the variant, _T-
WaveNet-Haar_ , where $\phi$, $\psi$, $\rho$ and $\eta$ are traditional Haar
wavelet basis. The results in Table 2 shows that our _T-WaveNet_ achieves
$18.6\%$ improvement on $F_{m}$ score over the Haar variant. This is because
our deep wavelet basis can be learnt from data, thus being more adaptive to
various sensor signals compared with Haar wavelet.

Next, we experiment on the deep version of the Lifting Scheme, in which we
realize the Predictor $P$ and Updater $U$ in Eqn.$(3)(4)$ using the same deep
modules as $\phi$, $\psi$ ,$\rho$ and $\eta$. Besides the default
architecture, we also experiment with other structures, as shown in Table 3.
In the table, L represents the number of convolution layers. C indicates the
times of the number of output channel compared with the input channel. D is
the dilation size of the first convolution layer. As can be seen, our method
consistently outperforms the deep Lifting Scheme version across all the
network structure settings on the two evaluation metrics, indicating the
superiority of the INN-based operator.

Configuration | $P,U$ |  $\phi$,$\psi$,$\rho$,$\eta$
---|---|---
$F_{m}\%$ | $F_{w}\%$ | $F_{m}\%$ | $F_{w}\%$
default:(L-2, C-3, D-1) | 92.6 | 73.5 | 93.1 | 76.3
(L-1, C-3, D-1) | 91.2 | 69.4 | 91.9 | 71.4
(L-3, C-3, D-1) | 91.5 | 71.6 | 92.4 | 73.5
(L-2, C-1, D-1) | 92.7 | 74.5 | 92.8 | 75.5
(L-2, C-5, D-1) | 91.6 | 72.1 | 92.7 | 75.7
(L-2, C-6, D-1) | 92.3 | 73.0 | 92.4 | 74.6
(L-2, C-10, D-1) | 91.9 | 71.2 | 92.0 | 72.3
(L-2, C-3, D-2) | 91.9 | 71.9 | 92.5 | 74.4
(L-2, C-3, D-3) | 92.0 | 73.6 | 92.7 | 75.1
(L-2, C-3, D-4) | 92.0 | 72.9 | 92.7 | 75.0
(L-2, C-3, D-5) | 92.0 | 72.0 | 92.5 | 73.3
Table 3: Comparison of Lifting scheme and INN-based wavelet transform units.
L-$k$ means the module has $k$ convolution layers; C-$k$ represents the output
channel size of the first layer is $k$ times of the input; D-$k$ is the
dilation size of the first convolution layer to enlarge the receptive fields.

Feature fusion module. To demonstrate the effectiveness of the proposed
adaptive feature fusion module for handling the personalized heterogeneity of
the sensor data, we remove the fusion module from T-WaveNet and fuse the leaf
features $\\{h_{i}\\}$ with equivalent weights. As shown in Table 2, the
performance of the resulted model decrease by $2\%$-$3\%$, which indicates
that the proposed feature fusion module can make better classification for
sensor data recorded from different subjects, because it fuses the features of
various frequency subbands adaptively based on the dependencies among multiple
frequency components.

##  6 Conclusion

We propose T-WaveNet, a novel tree-structured wavelet neural network for
sensor-based time series analysis. By decomposing the input signal into
frequency subbands according to power spectrum analysis and utilizing INN-
based frequency bisection operator, T-WaveNet provides more effective
representation for sensor information over existing DNN-based feature
extraction techniques. Experimental results on four different kinds of sensor
datasets show that T-WaveNet consistently outperforms the state-of-the-art
solutions by a considerable margin.

## References

  * Ahmad Nasrul and Mohd Hanafi (2009) Norali Ahmad Nasrul and Mat Som Mohd Hanafi.  Surface electromyography signal processing and application: A review.  2009\.
  * Anguita et al. (2013) Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz.  A public domain dataset for human activity recognition using smartphones.  In Esann, volume 3, page 3, 2013.
  * Atzori et al. (2012) Manfredo Atzori, Arjan Gijsberts, Simone Heynen, Anne-Gabrielle Mittaz Hager, Olivier Deriaz, Patrick Van Der Smagt, Claudio Castellini, Barbara Caputo, and Henning Müller.  Building the ninapro database: A resource for the biorobotics community.  In IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), pages 1258–1265, 2012.
  * Atzori et al. (2014) Manfredo Atzori, Arjan Gijsberts, Claudio Castellini, Barbara Caputo, Anne-Gabrielle Mittaz Hager, Simone Elsig, Giorgio Giatsidis, Franco Bassetto, and Henning Müller.  Electromyography data for non-invasive naturally-controlled robotic hand prostheses.  Scientific data, 1(1):1–13, 2014.
  * Brunner et al. (2008) C Brunner, R Leeb, G Müller-Putz, A Schlögl, and G Pfurtscheller.  Bci competition 2008–graz data set a.  Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces), Graz University of Technology, 16, 2008.
  * Chambers and Yoder (2020) Robert D Chambers and Nathanael C Yoder.  Filternet: A many-to-many deep learning architecture for time series classification.  Sensors, 20(9):2498, 2020.
  * Chavarriaga et al. (2013) Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R Millán, and Daniel Roggen.  The opportunity challenge: A benchmark database for on-body sensor-based activity recognition.  Pattern Recognition Letters, 34(15):2033–2042, 2013.
  * Chen et al. (2016) Yuwen Chen, Kunhua Zhong, Ju Zhang, Qilong Sun, and Xueliang Zhao.  Lstm networks for mobile human activity recognition.  In 2016 International Conference on Artificial Intelligence: Technologies and Applications, 2016.
  * Chen et al. (2020) Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu.  Deep learning for sensor-based human activity recognition: overview, challenges and opportunities.  arXiv preprint arXiv:2001.07416, 2020.
  * CR et al. (2011) Hema CR, Paulraj MP, et al.  An analysis of the effect of eeg frequency bands on the classification of motor imagery signals.  International Journal of Biomedical Soft Computing and Human Sciences: the official journal of the Biomedical Fuzzy Systems Association, 16(1):121–126, 2011.
  * Duan et al. (2015) Feng Duan, Lili Dai, Wennan Chang, Zengqiang Chen, Chi Zhu, and Wei Li.  semg-based identification of hand motion commands using wavelet neural network combined with discrete wavelet transform.  IEEE Transactions on Industrial Electronics, 63(3):1923–1934, 2015\.
  * Geng et al. (2016) Weidong Geng, Yu Du, Wenguang Jin, Wentao Wei, Yu Hu, and Jiajun Li.  Gesture recognition by instantaneous surface emg images.  Scientific reports, 6:36571, 2016.
  * Grenander (1959) Ulf Grenander.  The nyquist frequency is that frequency whose period is two sampling intervals.  Probability and Statistics: The Harald Cramér Volume, page 434, 1959.
  * Guan and Plötz (2017) Yu Guan and Thomas Plötz.  Ensembles of deep lstm learners for activity recognition using wearables.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(2):1–28, 2017.
  * Haar (1909) Alfred Haar.  Zur theorie der orthogonalen funktionensysteme.  Georg-August-Universitat, Gottingen., 1909.
  * Hu et al. (2020) Yue Hu, Xiao-Qing Zhang, Li Xu, Feng Xian He, Zhao Tian, Wei She, and Wei Liu.  Harmonic loss function for sensor-based human activity recognition based on lstm recurrent neural networks.  IEEE Access, 8:135617–135627, 2020.
  * Ignatov (2018) Andrey Ignatov.  Real-time human activity recognition from accelerometer data using convolutional neural networks.  Applied Soft Computing, 62:915–922, 2018.
  * Jamal (2012) Muhammad Zahak Jamal.  Signal acquisition using surface emg and circuit design considerations for robotic prosthesis.  Computational Intelligence in Electromyography Analysis-A Perspective on Current Applications and Future Challenges, 18:427–448, 2012\.
  * Jamaluddin et al. (2016) FAUZANI N Jamaluddin, SITI A Ahmad, WZ Wan Hassan, and SAMSUL BAHARI MOHD Noor.  Filtering corner frequency using undecimated wavelet transform for surface emg.  JESTEC, pages 1–6, 2016.
  * Janidarmian et al. (2017) Majid Janidarmian, Atena Roshan Fekr, Katarzyna Radecka, and Zeljko Zilic.  A comprehensive analysis on wearable acceleration sensors in human activity recognition.  Sensors, 17(3):529, 2017.
  * Jiang and Yin (2015) Wenchao Jiang and Zhaozheng Yin.  Human activity recognition using wearable sensors by deep convolutional neural networks.  In Proceedings of the 23rd ACM international conference on Multimedia, pages 1307–1310, 2015.
  * Karantonis et al. (2006) Dean M Karantonis, Michael R Narayanan, Merryn Mathie, Nigel H Lovell, and Branko G Celler.  Implementation of a real-time human movement classifier using a triaxial accelerometer for ambulatory monitoring.  IEEE transactions on information technology in biomedicine, 10(1):156–167, 2006.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Machine Learning, 2015.
  * Labati et al. (2019) Ruggero Donida Labati, Enrique Muñoz, Vincenzo Piuri, Roberto Sassi, and Fabio Scotti.  Deep-ecg: convolutional neural networks for ecg biometric recognition.  Pattern Recognition Letters, 126:78–85, 2019.
  * Lawhern et al. (2018) Vernon J Lawhern, Amelia J Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance.  Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces.  Journal of neural engineering, 15(5):056013, 2018.
  * Lee et al. (2017) Song-Mi Lee, Sang Min Yoon, and Heeryon Cho.  Human activity recognition from accelerometer data using convolutional neural network.  In 2017 ieee international conference on big data and smart computing (bigcomp), pages 131–134. IEEE, 2017.
  * Li et al. (2017) Xiangxin Li, Oluwarotimi Williams Samuel, Xu Zhang, Hui Wang, Peng Fang, and Guanglin Li.  A motion-classification strategy based on semg-eeg signal combination for upper-limb amputees.  Journal of neuroengineering and rehabilitation, 14(1):2, 2017.
  * Ordóñez and Roggen (2016) Francisco Javier Ordóñez and Daniel Roggen.  Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition.  Sensors, 16(1):115, 2016.
  * Phinyomark et al. (2013) Angkoon Phinyomark, Franck Quaine, Sylvie Charbonnier, Christine Serviere, Franck Tarpin-Bernard, and Yann Laurillau.  Emg feature evaluation for improving myoelectric pattern recognition robustness.  Expert Systems with applications, 40(12):4832–4840, 2013.
  * Qian et al. (2019) Hangwei Qian, Sinno Jialin Pan, Bingshui Da, and Chunyan Miao.  A novel distribution-embedded neural network for sensor-based activity recognition.  IJCAI 2019, pages 5614–5620, 2019.
  * Rahimian et al. (2020) Elahe Rahimian, Soheil Zabihi, Seyed Farokh Atashzar, Amir Asif, and Arash Mohammadi.  Xceptiontime: Independent time-window xceptiontime architecture for hand gesture classification.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1304–1308, 2020.
  * Rioul and Vetterli (1991) Olivier Rioul and Martin Vetterli.  Wavelets and signal processing.  IEEE signal processing magazine, 8(4):14–38, 1991.
  * Rodriguez et al. (2020) Maria Ximena Bastidas Rodriguez, Adrien Gruson, Luisa Polania, Shin Fujieda, Flavio Prieto, Kohei Takayama, and Toshiya Hachisuka.  Deep adaptive wavelet network.  In The IEEE Winter Conference on Applications of Computer Vision, pages 3111–3119, 2020.
  * Sagha et al. (2011) Hesam Sagha, Sundara Tejaswi Digumarti, José del R Millán, Ricardo Chavarriaga, Alberto Calatroni, Daniel Roggen, and Gerhard Tröster.  Benchmarking classification techniques using the opportunity human activity dataset.  In IEEE International Conference on Systems, Man, and Cybernetics, pages 36–40, 2011.
  * Sweldens (1998) Wim Sweldens.  The lifting scheme: A construction of second generation wavelets.  SIAM journal on mathematical analysis, 29(2):511–546, 1998.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * Vepakomma et al. (2015) Praneeth Vepakomma, Debraj De, Sajal K Das, and Shekhar Bhansali.  A-wristocracy: Deep learning on wrist-worn sensing for recognition of user complex activities.  In 2015 IEEE 12th International conference on wearable and implantable body sensor networks (BSN), pages 1–6, 2015.
  * Wang et al. (2019) Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu.  Deep learning for sensor-based activity recognition: A survey.  Pattern Recognition Letters, 119:3–11, 2019.
  * Wei et al. (2019a) Wentao Wei, Qingfeng Dai, Yongkang Wong, Yu Hu, Mohan Kankanhalli, and Weidong Geng.  Surface-electromyography-based gesture recognition by multi-view deep learning.  IEEE Transactions on Biomedical Engineering, 66(10):2964–2973, 2019\.
  * Wei et al. (2019b) Wentao Wei, Yongkang Wong, Yu Du, Yu Hu, Mohan Kankanhalli, and Weidong Geng.  A multi-stream convolutional neural network for semg-based gesture recognition in muscle-computer interface.  Pattern Recognition Letters, 119:131–138, 2019.
  * Xi et al. (2018) Rui Xi, Mengshu Hou, Mingsheng Fu, Hong Qu, and Daibo Liu.  Deep dilated convolution on multimodality time series for human activity recognition.  In International Joint Conference on Neural Networks, pages 1–8, 2018.
  * Xing et al. (2019) Yang Xing, Chen Lv, Huaji Wang, Dongpu Cao, Efstathios Velenis, and Fei-Yue Wang.  Driver activity recognition for intelligent vehicles: A deep learning approach.  IEEE Transactions on Vehicular Technology, 68(6):5379–5390, 2019\.
  * Yang et al. (2015) Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiao Li Li, and Shonali Krishnaswamy.  Deep convolutional neural networks on multichannel time series for human activity recognition.  In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
  * Yao et al. (2017) Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, and Tarek Abdelzaher.  Deepsense: A unified deep learning framework for time-series mobile sensing data processing.  In Proceedings of the 26th International Conference on World Wide Web, pages 351–360, 2017.
  * Zeng et al. (2014) Ming Zeng, Le T Nguyen, Bo Yu, Ole J Mengshoel, Jiang Zhu, Pang Wu, and Joy Zhang.  Convolutional neural networks for human activity recognition using mobile sensors.  In 6th International Conference on Mobile Computing, Applications and Services, pages 197–205, 2014.
  * Zeng et al. (2020) Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, and Stephen Lin.  Srnet: Improving generalization in 3d human pose estimation with a split-and-recombine approach.  arXiv preprint arXiv:2007.09389, 2020.
  * Zhang et al. (2020) Dalin Zhang, Kaixuan Chen, Debao Jian, and Lina Yao.  Motor imagery classification via temporal attention cues of graph embedded eeg signals.  IEEE Journal of Biomedical and Health Informatics, 2020.