# Hierarchical Windowed Graph Attention Network and a Large Scale Dataset for
Isolated Indian Sign Language Recognition

Suvajit Patra
RKMVERI
Belur, India
<EMAIL_ADDRESS>Arkadip Maitra
RKMVERI
Belur, India
<EMAIL_ADDRESS>Megha Tiwari
FDMSE, RKMVERI
Coimbatore, India
<EMAIL_ADDRESS>K. Kumaran
FDMSE, RKMVERI
Coimbatore, India
<EMAIL_ADDRESS>Swathy Prabhu
RKMVC
Chennai, India
<EMAIL_ADDRESS>Swami Punyeshwarananda
RKMVERI
Belur, India
<EMAIL_ADDRESS>Soumitra Samanta
RKMVERI
Belur, India
<EMAIL_ADDRESS>
###### Abstract

Automatic Sign Language (SL) recognition is an important task in the computer
vision community. To build a robust SL recognition system, we need a
considerable amount of data which is lacking particularly in Indian sign
language (ISL). In this paper, we propose a large-scale isolated ISL dataset
and a novel SL recognition model based on skeleton graph structure. The
dataset covers 2,002 daily used common words in the deaf community recorded by
20 (10 male and 10 female) deaf adult signers (contains 40033 videos). We
propose a SL recognition model namely Hierarchical Windowed Graph Attention
Network (HWGAT) by utilizing the human upper body skeleton graph structure.
The HWGAT tries to capture distinctive motions by giving attention to
different body parts induced by the human skeleton graph structure. The
utility of the proposed dataset and the usefulness of our model are evaluated
through extensive experiments. We pre-trained the proposed model on the
proposed dataset and fine-tuned it across different sign language datasets
further boosting the performance of 1.10, 0.46, 0.78, and 6.84 percentage
points on INCLUDE [46], LSA64 [41], AUTSL [45] and WLASL [25] respectively
compared to the existing state-of-the-art skeleton-based models. The proposed
dataset and the model implementation code will be available at
https://cs.rkmvu.ac.in/~isl.

##  1 Introduction

Sign Language (SL) is a natural language with unique grammatical and
linguistic characteristics. The deaf and mute community developed this to
communicate with each other and socialize. As a visual language, SL conveys
information by adopting the articulation of human body parts with manual
characteristics such as hand shapes, body pose, and the interaction of hands
with different body parts, together with non-manual characteristics such as
facial expression and head movement [1, 42].

According to the World Health Organization (WHO), around $5\%$ ($430$ million)
people around the world suffer from hearing loss [11]. To bridge the
communication gap between signers (people with sign language as their primary
communication medium) and non-signers (people with spoken language proficiency
rather than sign language) the automatic SL recognition field has emerged and
gained popularity among computer vision and machine learning researchers [48,
54, 26]. This task contains 2 subtasks, 1) isolated SL recognition - which
maps every sign video to the corresponding gloss, and 2) continuous SL
recognition - which maps every sign video to a sequence of glosses. Here our
focus is on isolated sign language recognition. Similar to any recognition
task, to build a good SL recognition model, we need a sufficient amount of
data to get a considerable inference. Researchers have proposed different
datasets for different SL recognition tasks, for example, MS-ASL [20], WLASL
[25], ASLLVD [4] for the isolated American SL recognition task. For the
British SL recognition task, there are datasets like BOB-SL [3], and BSLDict
[31]. Also, there are other datasets [45, 41, 54] for different sign
languages.

The Indo-Pakistani sign language is the most used sign language in the world
and about $15$ million deaf signers use this in their daily
communications111https://www.ethnologue.com/. In comparison to other sign
languages Indian Sign Language (ISL) contains a higher number of composite
signs (signs made up of two or more glosses). For example, the word Wife sign
consists of the Female and Marriage signs [46]. These factors make ISL
recognition task both important and unique. For automatic ISL recognition
task, there are not many resource-rich publicly available datasets in the
literature. Some limited attempts have been made with INCLUDE [46] and CISLR
[19] containing a limited number of sign videos per word. This provides the
impetus for creating a resource-rich isolated ISL dataset. We propose a
resource-rich isolated ISL dataset containing $40033$ sign video with $20$
videos per sign word (contains $2002$ words).

In general, the sign language recognition task is a subdomain of human action
recognition from video data and it inherits all the challenges such as blurred
boundaries between classes, occlusion of body parts, human appearance,
recording environments and recording settings [34]. Also, sign language
glosses contain very subtle spatial and temporal differentiable features,
which introduces another level of complexity. This makes the sign language
(particularly ISL) recognition task more challenging than action recognition
and state-of-the-art action recognition models fall short in SL recognition
tasks.

To model the SL recognition task we consider the signer’s skeleton joint
points from sign videos and represent that as a spatio-temporal skeleton graph
structure. We learn the graph structure in natural language processing
settings. Usually, spoken language is represented by a 1-dimensional sequence
of words with well-defined syntax and semantics, and its structure is
successfully modeled by the attention mechanism [49]. However, the skeleton
graph extracted from sign videos is a 3-dimensional data with complex spatio-
temporal context, and it is non-trivial to analogically define visual word or
unit with clarified semantics. Existing graph based approaches and normal
baseline approaches face difficulty in recognition from this spatio-temporal
skeleton data. To tackle these issues, we propose a novel Hierarchical
Windowed Graph Attention Network (HWGAT). The proposed model conserves the
skeleton graph structure and attention mechanism along with input partitioning
so that specific sign expressions such as expressive hand movements can be
processed without unwanted influence from the other body parts. Through
extensive experiments, we justify different design decisions in the HWGAT
model. We evaluate the HWGAT model on the proposed dataset and other state-of-
the-art datasets and report comparative results. In this paper, our main
contributions are as follows:

  1. 1.

We propose a large-scale isolated ISL dataset consisting of over $40,000$
videos containing a rich and large vocabulary of $2,002$ daily used words in
ISL conversations. It contains unique characteristics like 20 signers, gender-
balanced and signer-independent sets (no intersection of signers in training,
validation, and testing sets). It also contains sign word analysis and their
categorizations into different groups based on the number of glosses (e.g.
atomic, composite).

  2. 2.

We propose a novel windowed graph attention-based model specifically developed
for sign language recognition on skeleton graphs.

  3. 3.

We publish our automated recording and annotations pipeline to ease such data
collection process in any sign language.

The rest of the paper is organized as follows: Section 2 provides an overview
of the existing sign language datasets and recognition techniques. The
characteristics of the proposed dataset are described in Section 3. Section 4
presents the working mechanism of our proposed framework. The experiment,
results and analysis are given in Section 5 followed by conclusion in Section
6.

##  2 Related Works

In the literature, there are limited numbers of datasets on Indian sign
language (ISL) recognition tasks. In this section, first, we review the
existing ISL datasets followed by some state-of-the-art other sign language
datasets for different languages. Then review some state-of-the-art sign
language recognition models.

Table 1: Comparison of FDMSE-ISL dataset with the state-of-the-art isolated
sign language datasets.

Dataset Language # Signs # Sign video # Signers Source # Hours (avg. per sign)
ASLLVD [4] American (ASL) 2,742 $9K$ (3) 6 lab 4 ASL-LEX 2.0 [43] American
(ASL) 2,723 $2723$(1) - lexicons, lab, web - MSASL [20] American (ASL) 1,000
$25K$(25) 222 lexicons, web 25 WLASL [25] American (ASL) 2,000 $21K$ (11) 119
lexicons, web 14 LSA64 [41] Argentinian 64 $3K$ (47) 10 - 1.9 BSLDict [31]
British (BSL) 9,283 $14K$ (1) 148 lexicons 9 DEVISIGN-L [50] Chinese (CSL)
2,000 $24K$ (12) 8 lab 13-33 SLR500 [54] Chinese (CSL) 500 $125K$ (250) 50 lab
69-139 GSL [1] Greek (GSL) 310 $40K$() 7 - 6.4 SMILE [12] Swiss German (DSGS)
100 $9K$ (90) 30 lab - BosphorusSion22k [33] Turkish (TSL) 744 $23K$ (30) 6
lab 19 AUTSL [45] Turkish (TSL) 226 $38K$ (170) 43 lab 21 INCLUDE [46] Indian
(ISL) 263 $4K$ (16) 7 lab 3 CISLR [19] Indian (ISL) 4765 $7K$ (1) 71 web 3
FDMSE-ISL Indian (ISL) 2,002 $40K$ (20) 20 lab 36

For ISL recognition, the initial datasets were primarily consisting of a few
image samples or a very limited number of sign videos. To the best of our
knowledge, the first ISL dataset proposed by Rekha et al., consists of $290$
images for $26$ alphabets [39]. One early dataset containing $600$ videos
corresponding to $22$ sign word classes [32] and another dataset with $800$
sign videos for $80$ signs [22] suffer from either a limited vocabulary size
or a low number of samples per class. These datasets are not sufficient for
robust real-world applications. The largest available ISL dataset namely
CISLR, containing $7050$ videos with over $4765$ sign words [19], suffers from
less number of per-class samples, making it inconsistent for real-world sign
language recognition tasks and is viable only for one-shot learning tasks. In
literature, the largest standard isolated ISL dataset INCLUDE [46] has a
collection of $263$ signs, recorded with $7$ signers (students) in a classroom
setting with a static background, and contains a total of $4,287$ videos. Even
this dataset contains very limited vocabulary compared to the vocabulary size
used by ISL signers in their daily communications.

(a) (b) (c) (d) (e)

Figure 1: Sample frame of the sign ”Hello” in all the five modalities ((a)
left (60fps), (b) front (60fps), (c) right (60fps), (d) Azure Kinect DK depth
(30fps) and (e) Azure Kinect DK RGB (30fps)) available in the dataset.

For other sign languages, there exists a number of large-scale sign language
datasets shown in Table 1. Athitsos et al. [4] proposed an isolated American
Sign Language (ASL) dataset called ASLLVD, consisting of $9,800$ video samples
of $3,300$ sign words, recorded with 1-6 native signers. Another isolated ASL
dataset called MS-ASL [20] proposed by Jose and Koller covering $25,000$
videos of $1,000$ sign words with $222$ signers. To make the dataset
challenging they recorded the videos in unconstrained real-life conditions. In
the recent past, Li et al. [25] proposed a word-level ASL dataset (WLASL) with
a total of $21,083$ video samples of $2,000$ signs with $119$ signers. The
videos are collected from different web sources. Momeni et al. [31] proposed a
British Sign Language (BSL) dataset known as BSLDict which contains a large
vocabulary of size $9,283$ and is created with $148$ signers. There is a sign
language dataset for Turkish Sign Language (TSL) proposed by Sincan and Keles
[45] called AUTSL, which contains $38,336$ video samples of $226$ signs
performed by $43$ different signers. The videos are recorded in both indoor
and outdoor environments. This dataset has color, depth, and skeleton
modalities. Some other isolated SL datasets include Chinese Sign Language
(CSL) SLR500 [54] contains $125,000$ videos spanning over $500$ signs. The
SMILE [12] dataset is the isolated Swiss German Sign Language (DSGS) dataset
containing $100$ signs with $9,000$ videos.

To the best of our knowledge, in terms of video count, the proposed dataset
contains the maximum number of videos compared to the other isolated sign
language datasets shown in Table 1, except the SLR500 dataset. However, the
SLR500 dataset only contains a vocabulary of $500$ signs, and each sign is
repeated five times by each signer. Regarding vocabulary size, the sign
language dictionaries ASLLVD [4], ASL-LEX 2.0 [43], WLASL [25], and BSLDict
[31] have demonstrated comparable or greater coverage of signs. However, they
have fewer videos per sign as shown in Table 1.

In general, researchers try to tackle the SL recognition task as a pattern
recognition task. It contains two subtasks namely $1)$ feature extraction:
where each sign video is represented as a fixed dimensional feature vector and
$2)$ recognition: where the represented videos are classified using a standard
classifier. For feature extraction, initially researchers [48, 33, 54, 26, 18]
have tried some hand-crafted features like Histogram of Oriented Gradient
(HOG) [9], Scale Invariant Feature Transform (SIFT) [28], Optical Flow [14].
They classify each sign using standard classifiers like HMM [35, 24], SVM
[37], and Random Forest [2].

After the prevalence of deep learning methods due to their performance
overhead over traditional methods, researchers are leveraging advanced feature
extraction and classification techniques using deep learning methods.
Particularly in the domain of SL recognition, deep learning-based methods are
broadly classified into two types. The first approach involves extracting
features from each raw RGB video using various methods such as two-dimensional
(2D) CNN [23, 38], 3D CNN [15, 25], and CNN models with Bi-LSTM decoder [10].
These features are classified into glosses using one or more fully connected
layers. In addition to RGB videos Jiang el al. [17] utilized depth, skeleton,
and motion information to enhance the recognition of Turkish sign language.
Further advancements by Zuo et al. [56] have integrated natural language
glosses during the training process, resulting in state-of-the-art performance
across multiple SL datasets. Despite these advancements, RGB-based methods are
computationally expensive, and their slow classification time poses
limitations for real-time sign language recognition.

In the second type of approach, researchers detect the skeleton keypoints of
the signer using the state-of-the-art human pose estimation methods like
MediaPipe [29], OpenPose [7], MMPose [8], Yolo-pose [30] etc. The extracted
sequential pose data is processed using various sequential data models
including GRU [25], LSTM [26, 21], and different varients of Transformers [44,
6, 5] for SL recognition task. Subsequently, researchers have represented
sequential pose data as skeleton graphs and employed Graph Convolutional
Networks (GCNs) due to their ability to capture contextual information from
graphical data effectively. Yan et al. [52] first introduced a GCN on the
sequential human keypoints data as a spatio-temporal graph using natural human
joint connectivity and called it ST-GCN. Jiang et al. adopted this model for
Turkish sign language recognition, by cascading spatial, temporal and channel
importance to the GCN block and called it SL-GCN [17]. The ST-GCN and SL-GCN
models were applied to the Indian sign language recognition task on the
INCLUDE [44] dataset. During inference, these models perform convolution
operations and give static weights to the connected neighbours of a node
irrespective of their feature representations.

Here, we propose a graph attention-based network named Hierarchical Windowed
Graph Attention Network (HWGAT), where we use an attention mechanism which
takes the node features into consideration to generate dynamic attention
weights instead of using static kernel weights. This resulted in better
performance compared to skeleton-based models across multiple SL datasets.

##  3 Proposed Dataset

This paper presents a large-scale isolated sign language dataset for SL
recognition tasks in general and in particular for Indian sign language
recognition tasks. In the creation of this dataset, we followed the FDMSE’s
[16] dictionary which was published in consultations with sign experts
throughout India. We carefully considered 2,002 common words used in daily
communications among the deaf community from that dictionary. The words are
categorised into $57$ groups _i.e._ _’family relations’, ’behaviour norms’,
’body parts’, ’household articles’_ etc. We further studied all the signs and
grouped them into two categories, namely, atomic signs or glosses (that cannot
be decomposed into other meaningful signs (eg. marriage)) and composite signs
(can be decomposed into atomic signs or glosses (eg. wife $\rightarrow$ female
\+ marriage)).

We collected the dataset with the help of $20$ native ISL signers (deaf) from
the southern part of India. To ensure that the dataset is gender unbiased, we
have considered $10$ male and $10$ female subjects. For data collection setup,
we used a static background with a green screen to facilitate the segmentation
process.

Figure 2: Sample frames from the signs ”Hello” and ”World”.

We have recorded the videos from four different viewing positions: two
frontal, approximately $30^{\circ}$ left, and $30^{\circ}$ right with respect
to the frontal view of the subject. Three Logitech BRIO $60$ fps cameras were
used for recording the videos in landscape mode with a frame size of
$1920\times 1080$. Furthermore, to capture the depth information one Azure
Kinect DK camera is used at the front position. By keeping single-view real-
world SL recognition applications in mind, this work is focused on the frontal
RGB camera data only. But all the $5$ modalities (four-RGB and one-depth
modalities) recorded will be publicly available (https://cs.rkmvu.ac.in/~isl)
and can be used for research purposes only. Furthermore, apart from the
frontal RGB modality, the other multiview modalities can also be used for
tasks such as keypoints correction, pose estimation, 3D-model generation, and
general gesture recognition. The videos were recorded in the lab settings with
required ethical clearance and a standard dress code (matt black) under the
supervision of certified ISL experts.

We built a custom tool named Word Viewer and Timeline Manager (WVTM) to manage
and automatically annotate the large corpus of videos. During recording, the
operator uses the WVTM tool first to show a sign word to the subject with a
prompt and then register the event (session start, start recording for a word,
stop recording for a word, session end etc.) timestamps in a log file. After
recording all the sessions, a Python script is used to automatically split the
videos and annotate them using the log files generated during the recording.
The complete annotation and recording pipeline will be publicly available
(https://cs.rkmvu.ac.in/~isl) to simplify the dataset collection process for
any sign language.

In total, the dataset from the frontal RGB camera contains $40,033$ videos
across $2,002$ words. The total duration of the dataset is around 36.2 hours
with 7.8 Million frames. The average duration of the sign videos is around
3.25 seconds. We crop the original videos to $1200\times 950$ resolution
keeping the signer at the centre. Table 2 summarizes the statistics of the
dataset. For each sign word, there are five different modalities (frontal both
$60$ fps and $30$ fps, two side views at $\approx$ $\pm 30$ degrees $60$ fps
and depth information) from four viewing positions. For the experiments in
this paper we used $40,033$ videos from the frontal $60$ fps RGB camera.
Figure 1 shows the sample frames from each camera view and Figure 2 shows some
frames of two signs ’Hello’ and ’World’.

The proposed dataset excels in several aspects from the existing ISL dataset
INCLUDE [46]. For example, the proposed dataset contains around a $7.6$ times
higher number of signs and a $9.3$ times higher number of videos. Furthermore,
our dataset has more qualitative diversity in terms of age, height, and skin
tone. For instance, the INCLUDE [46] dataset was recorded with young students
of similar age, whereas our signers’ approximate age ranged between $28$ and
$55$, and height between $4.5$ and $6$ feet. While our dataset is multimodal
and multicamera, the INCLUDE dataset was recorded with a single camera.

To evaluate the proposed model on the proposed dataset, we divide the dataset
in train, validation, and test partitions into $5:1:4$ ratios on randomly
chosen subjects (signers) so that the training set, validation set and test
set have no intersection of signers. In total, there are 10 subjects in
training (5 male, 5 female), 2 for validation (1 male, 1 female) and 8 for
testing (4 male, 4 female). Finally, we used 20,016, 4,003, and 16,014 videos
for training, validation, and testing respectively.

Table 2: Summary of the FDMSE-ISL dataset. Parameters | Values
---|---
Words | 2002
Videos | 40033
Groups | 57
Avg Videos per Class | $\approx 20$
Avg Video Length | $3.25\mathrm{s}$
Min Video Length | $1.5\mathrm{s}$
Max Video Length | $9.5\mathrm{s}$
Frame Rate | 60 $\mathrm{fps}$, 30 $\mathrm{fps}$
Resolution |  $1200\times 950$, $512\times 512$
Modalities | 4 $RGB$, 1 $depth$
Figure 3: The proposed Hierarchical Windowed Graph Attention Network (HWGAT)
takes the spatio-temporal graph structure as input and divides this graph into
multiple spatial windows based on distinct body parts as represented in Figure
4. Next, multiple part attention layers are applied on this windowed graph
structure to extract features and a fully connected layer is used to get the
sign word.

##  4 Proposed Approach

The objective of this work is to create a sign language recognition system
that works on extracted keypoints from any sign video. According to ISL
experts, the influence of different body parts on any particular sign is
essential, especially hands. The body keypoints and the edges connecting these
keypoints form a graph structure. Classification of these graph structures are
generally modeled by Graph Convolutional Networks (GCN) [52, 17]. In general,
GCN models are homogenous to CNNs where the kernel in CNN is similar to the
learned adjacency matrix in GCN. But, even after the training process, the
elements of the adjacency matrix are static. We propose the Hierarchical
Windowed Graph Attention Network (HWGAT) where we use the attention mechanism
which takes the node features into consideration to generate dynamic attention
weight instead of fixed kernel weight. Therefore giving importance to the
neighbourhood nodes based on their similarity during information sharing. Also
to incorporate the importance of body parts for any sign word recognition our
attention mechanism is designed to be restricted according to the spatio-
temporal skeleton graph. An overview of the proposed HWGAT is presented in
Figure 3. This model consists of two major parts namely spatio-temporal input
representation and part attention layer. We represent the input graph
structure as follows:

Figure 4: Grouping of spatial keypoints into the $5$ body parts. The part
marked as $1$ contains the selected right-hand keypoints, the part marked as
$2$ has the right arm keypoints, the part marked as $3$ has the selected face
keypoints, the part marked as $4$ is the left-arm and $5$ marks the left-hand
keypoints respectively. These part combinations are used to create the $4$
spatial windows.

###  4.1 Spatio-temporal Input Representation:

The spatio-temporal input representation consists of 2 parts: skeleton graph
representation and spatial window substructures. We select 27 keypoints from
each frame of a sign video, encompassing 3 facial keypoints (nose, 2 eyes), 2
shoulders, 2 elbows, and 10 keypoints from each hand to represent the spatial
skeleton graph input for our model as shown in Figure 5. These keypoints are
selected based on recommendations from SL experts and inspired by [18, 44].
The keypoint graph generated from an entire video consists of $F$ frames and
each frame contains $K$ keypoints. Each keypoint is represented as a 2D vector
with $x$ and $y$ coordinates, forming nodes in this generated spatio-temporal
keypoint graph. The edge set $E(G)$ for the entire graph is constructed as
follows:

$E(G)=\begin{cases}e_{i,j}^{t}=1,&\text{if $v_{i},v_{j}$ are connected
spatially}\\\ e_{i}^{t,t+1}=1,&\text{temporal connection}\\\
e_{i}^{t}=0,&\text{otherwise}\end{cases}$ (1)

where, $t\in\\{1,2,...,(F-1)\\}$ denotes the frame index, $v$ is a node with
indices $i,j\in\\{1,2,...,K\\}$ and $e$ represents a particular edge in the
graph. Figure 6 illustrates the visual representation of the whole spatio-
temporal skeleton structure of a sign video. This spatio-temporal skeleton
structure is split into multiple subsets named spatial window substructures
Figure 4.

Figure 5: Visual representation of 27 keypoints used as input (10 per hands
and 7 pose points with connections).
Figure 6: The spatio-temporal skeleton graph obtained from a sign language
video. Figure 7: The temporal frames are divided into temporal blocks where
each temporal block contains multiple frames. Each spatial window with its
respective temporal block creates a spatio-temporal block.

A distinctive feature of our model, in contrast to GCNs, is the partitioning
of the keypoint graph into spatial windows rather than utilizing the entire
graph as input. In the single-view pose estimation methods, inconsistencies in
keypoint generation are omnipresent, especially for hand keypoints. These
discrepancies often arise from factors such as motion blur, occlusion by body
parts, and low video resolution, leading to misplacement of hand keypoints. To
mitigate these issues we divide the whole graph into multiple spatial windows
by stacking the part keypoints combinations. This spatial division restricts
the flow of information from one window to another. With this approach, for
one-handed signs, a window only contains the signing hand, thereby eliminating
the potential influence of the non-signing hand’s motion. The creation of
these spatial windows is formalized as follows:

For each frame, the input keypoints $K=27$ are divided into $5$ spatial
subsets corresponding to body parts: face, left arm, left hand, right arm, and
right hand, as illustrated in the Figure 4. For a spatial window $j$ in frame
$f$, the stack of face (3 keypoints), any one arm (3 keypoints) and any one
hand (10 keypoints) are taken to obtain $W_{j}^{f}$ keypoints subset with
$w_{k}=16$ keypoints. If there are $S$ spatial windows per frame then they are
stacked to generate the output frame with $K^{\prime}=S\times w_{k}$
keypoints. For a skeleton input sign video $I\in\mathbb{R}^{F\times K\times
d}$ with $F$ frames and $K$ keypoints per frame in $d(=2)$ dimensions, output
$I^{\prime}\in\mathbb{R}^{F\times K^{\prime}\times d}$ is our spatial window
representation.

It was demonstrated that the Fourier feature mapping allows a model to learn
high-frequency functions more effectively [47]. This process involves
embedding the low-dimensional input coordinates into a higher-dimensional
space. Specifically, our low $d$ dimensional node input representation is
embedded into a higher dimension $d^{\prime}$ using the Fourier feature
mapping. Finally, we get our spatio-temporal input representation
$I_{stir}\in\mathbb{R}^{F\times K^{\prime}\times d^{\prime}}$.

We incorporated the frame position as a sequential marker into the input
embeddings using a fixed positional encoding scheme similar to the one
proposed by Vaswani et al. [49].

###  4.2 Part Attention Layer

The individual signs depend on the motion of different parts and we want to
capture that using spatio-temporal part attention layer. To learn different
part attention, we design a part attention layer consisting of two key
components: part attention block, and temporal merge shown in Figure 3.

####  4.2.1 Part Attention Block

The part attention block is the fundamental component of each part attention
layer. The different contextual features are accumulated through information
propagation between nodes by utilizing multiple part attention blocks within
each part attention layer. The part attention block has three key components,
temporal blocks, temporal shift, and graph attention block shown in Figure 3.

Figure 8: The Graph Attention Block consists of the multi-head graph attention
module along with normalization and feed-forward layers. The multi-head graph
attention module with scaled dot product graph attention is the basic
attention block which uses the graph adjacency matrix.

Temporal Blocks: Temporally consecutive frames tend to have very subtle
changes in spatial graphs, and these changes can be effectively captured by
grouping frames within a block. This grouping approach is inspired by the
hierarchical divide-and-conquer paradigm. The attention mechanism is applied
within these blocks to capture local context and focus on granular movement
changes. Subsequently, these blocks are merged to capture the entire temporal
context. To implement this, the embedded spatio-temporal input representation
$I_{stir}$ (input sequence) is divided into multiple temporal blocks, each
consisting of consecutive frames. Let each temporal block contain $T$
consecutive frames which divide $F$ temporal frames into $F^{\prime}$
consecutive blocks such that $F=F^{\prime}\times T$. Let
$B_{ij}\in\mathbb{R}^{T\times K^{\prime}\times d^{\prime}}$ be the spatio-
temporal block consists of $i^{th}$ spatial window and $j^{th}$ temporal block
shown in Figure 7.

Temporal Shift: In our proposed network, the restriction of attention within
each spatio-temporal block limits the sharing of information between
consecutive temporal blocks. To address this issue, we have used a shifting
mechanism inspired by [27] that shifts the temporal window through the input
sequence resulting in the propagation of information between consecutive
temporal blocks. In this mechanism, frames that fall outside the first window
are rolled and appended at the end of the temporal sequence. These rolled
frames are masked during attention operation to ensure they do not influence
the graph attention values in the last temporal block. Let
$f_{i}\in\mathbb{R}^{K^{\prime}\times d^{\prime}}$ be the $i^{th}$ frame of
our spatio-temporal input representation
$I_{stir}=\\{f_{1},f_{2},...,f_{F}\\}$. Then after $s$ shift we get
$I_{stir}^{s}=\\{f_{s+1},f_{s+2},...,f_{F},f_{1},f_{2},...,f_{s}\\}$.

Graph Attention Block:

The graph attention block, illustrated in Figure 8, is designed to operate on
spatio-temporal block $B_{ij}$. This module accumulates information from the
neighboring nodes using the attention mechanism that utilizes node feature
similarity, rather than using a convolution kernel. To preserve our skeleton
structure the attention mechanism is restricted to the neighboring nodes
connected by edges and belong to the same spatio-temporal block. We used an
attention mask called edge bias following the human joint-bone connectivity as
an inductive bias to restrict the attention mechanism. We followed the spatio-
temporal edge connectivity described in the input representation Section 4.1.
In every spatio-temporal block $B_{ij}$ for $i^{th}$ spatial window and
$j^{th}$ temporal block, the connectivity is further restricted according to
the corresponding edge set $e_{pq}\in E(G)$. The edge set $e_{pq}$ is used to
create a Graph Adjacency Matrix $A_{pq}$ for the spatio-temporal block
$B_{ij}$.

Let $n$ be the total number of keypoints within the $(i,j)^{th}$ spatio-
temporal block, then $B_{ij}\in\mathbb{R}^{n\times d^{\prime}}$ such that
$n=T\times K^{\prime}$. We define our graph attention ($GAttn$) and other
modules shown in Figure 8 as:

$\begin{split}GAttn&=Softmax(\frac{QK^{T}}{\sqrt{h}}\odot A_{pq})\\\
Q&=Norm(B_{ij})\times W^{q}\\\ K&=Norm(B_{ij})\times W^{k}\\\
V&=Norm(B_{ij})\times W^{v}\\\ B^{\prime}_{ij}&=B_{ij}+GAttn\times V\\\
O_{ij}&=B^{\prime}_{ij}+FF(Norm(B^{\prime}_{ij}))\\\ \end{split}$ (2)

where $W^{q},W^{k},W^{v}$ are the model parameters similar to the multi head
attention in [49] and $\odot,\times$ represent the element-wise multiplication
and matrix multiplication respectively. Here we use layer normalization as
$Norm$ to normalize the input, $FF$ is a feed-forward layer, and the output
$O_{ij}$ represents the embedding of $B_{ij}$ which exhibits the local spatio-
temporal contextual information.

Through our observations, we found that the attention weights given to similar
adjacent nodes are high due to their feature similarity (eg. nose keypoint
along temporal connections in edge connectivity defined in (1)). The high
attention values among similar nodes can sometimes reduce the attention
weights of other nodes, leading to a loss of important information. To
overcome this issue, we propose an attention dropout mechanism inspired by
[53]. But, instead of randomly nullifying attention weights, our proposed
regularization technique is more likely to drop attention between two nodes
with higher attention value. First a random variable $\gamma$ is uniformly
sampled from $(0,1)$. Then the attention matrix $GAttn$ is masked with value
$0$ where $GAttn$ is greater than $\gamma$.

####  4.2.2 Temporal Merge

The attention mechanism in the first Part Attention Layer captures the local
spatio-temporal features. To obtain global context, we use a hierarchical
temporal merging technique. In each Part Attention Layer, following the $M$
graph attention operations, the number of temporal frames is reduced by
merging frames within each temporal block, resulting in upper-level contextual
attention in deeper Part Attention Layers. For a spatio-temporal input
$I_{stir}\in\mathbb{R}^{F\times K^{\prime}\times d^{\prime}}$ and temporal
block size $T$, after first merging we get a spatio-temporal contextual
embedding in $\mathbb{R}^{(F/T)\times K^{\prime}\times(d^{\prime}\times T)}$.

After $N$ Part Attention Layer we use an average pooling over all the spatial
windows to represent the final spatio-temporal contextual embedding. Finally,
a fully connected layer utilizes this embedded feature to classify each sign.

##  5 Experimental Evaluation

In this section, first, we evaluate our proposed dataset compared to other
existing isolated Indian sign language datasets. Then we evaluated our
proposed model in isolated ISL datasets as well as in other isolated SL
datasets. We conducted extensive ablation studies to evaluate the
effectiveness of various design choices of our model development. Here we
start with a brief description of all the datasets used in our experiment,
then experimental settings, and data pre-processing followed by discussions of
all the results.

Table 3: Comaprison of dataset frame statistics. Frame Statistics | FDMSE-ISL | FDMSE-ISL400 | WLASL | AUTSL | LSA64 | INCLUDE
---|---|---|---|---|---|---
# Videos | 40033 | 8000 | 21095 | 36302 | 3200 | 4284
Mean | 195.40 | 195.07 | 60.55 | 60.97 | 82.75 | 64.27
Standard Deviation | 40.04 | 39.98 | 23.56 | 10.96 | 27.58 | 14.96
Minimum | 30.00 | 90.00 | 9.00 | 7.00 | 14.00 | 33.00
1st Quartile | 180.00 | 180.00 | 43.00 | 54.00 | 65.00 | 54.00
Median | 180.00 | 180.00 | 58.00 | 60.00 | 78.00 | 62.00
3rd Quartile | 210.00 | 210.00 | 74.00 | 67.00 | 100.00 | 71.25
Maximum | 570.00 | 570.00 | 282.00 | 156.00 | 201.00 | 154.00

###  5.1 Datasets

To evaluate our proposed FDMSE-ISL dataset, we considered a state-of-the-art
ISL dataset called INCLUDE [46]. It contains a vocabulary size of $262$ words
with $4285$ video samples in $1920\times 1080$ resolution. This dataset is
created in lab settings and has imbalanced samples in classes. A detailed
description of the FDMSE-ISL dataset is presented in Section 3.

In addition to our dataset and INCLUDE, we evaluated our proposed model on $3$
other isolated SL datasets namely AUTSL [45], LSA64 [41] and WLASL [25]. The
WLASL [25] is an isolated American sign language dataset with $2000$ glosses
and $21083$ samples. The videos were collected from web sources and were
recorded in unconstrained real-life settings. AUTSL [45] is a Turkish sign
language dataset comprising $38336$ samples throughout $226$ glosses. The
videos are recorded in real-life conditions with multiple backgrounds. LSA64
[41] is an Argentinian sign language dataset having $3200$ videos over $64$
signs. The videos were recorded in lab settings with static backgrounds.

For all the datasets we follow the author-given data partitions for all the
evaluations. Since there was no validation set available in the INCLUDE
dataset we made a validation set which contains $10\%$ data from the train
set.

For our extensive ablation study, we considered a $20\%$ subset of our
proposed dataset due to our experimental resource limitations. We call this
subset as FDMSE-ISL400. It contains $400$ classes with $8000$ video samples
and these $400$ classes were chosen from $2002$ classes with Simple Random
Sampling without replacement (SRSWOR) sampling procedure and tried to preserve
the frame statistics. A comparison (with respect to the number of signs,
number of videos, number of unique signers, video source, and total duration)
of all these datasets summary is shown in Table 1 and frame statistics are in
Table 3.

###  5.2 Experimental Settings

In the keypoint based approaches, first, we have to detect the body keypoints
effectively. There are many keypoints detection or pose estimation methods
available in the literature, some notables are OpenPose [7], MediaPipe
holistic [29], HR-net [51], and others [55, 36, 40]. Here we have considered
MediaPipe holistic [29] for our pose estimations due to its balance in both
accuracy and processing speed compared to others. The MediaPipe holistic gives
a total of $543$, 3D keypoints ($468$ for face, $33$ for pose, and $21$ per
hand) per video frame. Out of these, we considered $27$ keypoints (3 facial
keypoints (nose, $2$ eyes), $2$ shoulders, $2$ elbows, and $10$ keypoints from
each hand) with $x$ and $y$ coordinates for our skeleton representation.

We implemented all the models in Python 222https://www.python.org/ using the
PyTorch toolbox 333https://pytorch.org/ and trained on a system running on
Ubuntu 22.04 OS, 192 GB RAM, Xeon gold 32-core CPU and NVIDIA A100 80G GPU. To
train the models AdamW Optimizer was used with an initial learning rate of
$1e^{-4}$ with the Cosine Anealing scheduler having a patience of $20$ epochs.
The max training epochs were set to $4000$ with early stopping on validation
loss, set to the patience of $400$ epochs. The objective function used for
training is label smoothed cross entropy loss [13]. For all our experiments,
we report top-1 and top-5 per instance accuracy as the performance measure.

###  5.3 Data Pre-processing and Augmentations

Various data pre-processing techniques are used to normalize the data and data
augmentation techniques are employed during training to increase the model
robustness.

Instead of the normalized coordinates of the keypoints generated by the
Mediapipe holistic ($[0,1]$), we used the keypoint coordinates with respect to
the image size by multiplying the height and width (frame size) of the sign
video.

In real-life video recording scenarios, subjects can appear anywhere within
the image plane and with different scales. We normalise the keypoint
coordinates to achieve location and scale invariance for keypoints using
dynamic bounding boxes. Additionally, we apply shear and rotation
transformations during training to the keypoints to introduce subtle random
variations, inspired by [44].

After using mediapipe holistic we observed occasional failures in detecting
hand keypoints. Instead of seeing this as a limitation, we integrated this
occurrence into our framework by intentionally masking hand keypoints in
certain frames. We sample some frames with hyperparameter sampling probability
$\beta$ following the SRSWOR sampling procedure and mask all the hand
keypoints. Next, these frames are filled with approximated hand keypoints
using spherical linear interpolation technique. This missing data-filling
technique ensures a stable learning process for our graph networks.

According to the ISL experts, signing speed differs from signer to signer. To
incorporate this speed variability we introduce the temporal augmentation
technique by randomly reducing or increasing the number of frames through
SRSWOR or Simple Random Sampling with replacement (SRSWR) sampling procedure.

The video clips can have any number of frames while the models require a fixed
number of frames. To combat this inconsistent number of frames, temporal
sampling is introduced. If a video clip contains fewer frames than needed, we
pad it with a random offset at the beginning and accordingly adjust at the
end. Conversely, if the video contains more frames than required, we uniformly
sample the necessary number of frames from the clip.

###  5.4 Results

####  5.4.1 Ablation Study

We have conducted an ablation study for different hyperparameters (number of
spatial windows, temporal blocks, temporal shift, edge bias, regularizer) of
our proposed HWGAT model. For this study, we have used a subset called FDMSE-
ISL400 of our proposed dataset and report top-1 and top-5 per instance
accuracy as performance measurements.

Figure 9: Inter class similarity in signs ”Low” vs. ”Short/Young” and ”Seat”
vs. ”Bench”.

Number of spatial windows: The division of each skeleton frame into spatial
windows is a distinctive feature of our proposed model’s input representation.
We have experimented with three different numbers of spatial windows $(1,2,4)$
defined as follows:

  * •

In the case of $1$ spatial window, there is no division of the skeleton
spatial structure.

  * •

For $2$ spatial window, we divide the skeleton spatial structure into $2$
substructures: substructure-1 (contains head, left arm, and left hand) and
substructure-2 (contains head, right arm, and right hand).

  * •

In the case of $4$ spatial window, we divide the skeleton spatial structure
into $4$ substructures: substructure-1 (contains head, left arm, and left
hand), substructure-2 (contains head, left arm, and right hand),
substructure-3 (contains head, right arm, and left hand) and substructure-4
(contains head, right arm, and right hand).

The results of these three spatial windows divisions are shown in Table 4 and
the $4$ spatial windows give the highest performance of top-1 $95.94\%$ and
top-5 $99.44\%$ accuracy. Also, these results indicate that the proposed
substructure-based model performs relatively better than the whole graph input
representation ($1$ spatial window).

Table 4: Impact of number of spatial windows on FDMSE-ISL400 dataset. # Spatial Windows | Test Accuracy
---|---
Top-1 | Top-5
1 | 95.37 | 99.22
2 | 94.59 | 99.22
4 | 95.94 | 99.44
Table 5: Impact of length of temporal blocks on FDMSE-ISL400 dataset. Temporal Block Length | Test Accuracy
---|---
Top-1 | Top-5
2 | 95.94 | 99.44
4 | 94.97 | 99.31
Table 6: Impact of temporal shift on FDMSE-ISL400 dataset. Shifting Window | Test Accuracy
---|---
Top-1 | Top-5
Without Shift | 94.97 | 99.31
With Shift | 95.94 | 99.44
Table 7: Impact of edge bias on FDMSE-ISL400 dataset. Edge Bias Type | Test Accuracy
---|---
Top-1 | Top-5
Learnable Edge Bias | 95.53 | 99.41
Without Edge Bias | 95.16 | 99.00
With Edge Bias | 95.94 | 99.44
Table 8: Impact of regularizer on FDMSE-ISL400 dataset. Presence of Regularizer | Test Accuracy
---|---
Top-1 | Top-5
No Regularizer | 95.94 | 99.44
Regularizer | 96.63 | 99.47

Temporal block length: The accuracy reaches the top when the temporal block
length is set to $2$ as demonstrated in Table 5. The temporal block length
being $2$ enforces the model to capture the temporal dynamics in consecutive
frames.

Effect of temporal shift: Table 6 shows the results of the temporal shift
mechanism as described in Section 8. The temporal window shifting allows the
information propagation between consecutive temporal blocks resulting in
better top-1 and top-5 accuracy compared to without shifting.

Effect of edge bias: The proposed edge bias described in Section 4.2.1 has a
relatively better impact on the final classification task as shown in Table 7.
Our proposed edge bias gives $0.78$ percentage points top-1 accuracy compared
to the without edge bias mainly due to the attention mechanism being
restricted to each spatio-temporal block according to the skeleton structure.

Impact of regularizer: The proposed attention dropout regularization technique
described in Section 8 gives better performance ($0.69$ percentage point in
top-1 and $0.03$ percentage point in top-5) compared to without regularizer.

####  5.4.2 Evaluation of proposed dataset

To evaluate our proposed dataset, we consider our proposed model, a baseline
transformer-based model, and $two$ state-of-the-art skeleton-based models ST-
GCN [18] and SL-GCN [18]. We compared the results with an isolated ISL dataset
INCLUDE [46]. Table 9 shows the top-1 and top-5 performance of all the methods
on the proposed and INCLUDE datasets. From Table 9 we see that on the proposed
dataset all the models perform less compared to the INCLUDE datasets. This
surmise that the proposed dataset is more challenging, may be due to higher
number of classes and atomic signs being the subsets of composite signs
classes. Also, there is a lot of inter-class similarity between two different
signs like _’Low’ Vs. ’Short/Young’, ’Seat’ Vs. ’Bench’_ etc. shown in Figure
9. To see the effect of knowledge transfer, we pre-trained our proposed model
on the proposed dataset and fine-tuned ($200$ epochs) it on the INCLUDE
dataset. We achieved top-1 with $97.79\%$ and top-5 with $99.26\%$ as shown in
the last row of Table 9. This result shows the utility of the proposed dataset
for the ISL recognition task as a pre-training dataset on relatively smaller
isolated ISL datasets similar to INCLUDE.

Table 9: Comparing of results for Indian sign language dataset. Method | INCLUDE | FDMSE-ISL
---|---|---
Top-1 | Top-5 | Top-1 | Top-5
Transformer | 94.85 | 99.14 | 89.71 | 97.95
ST-GCN [44] | 96.69 | 99.14 | 93.57 | 99.01
SL-GCN [44] | 96.57 | 99.26 | 93.39 | 98.98
HWGAT | 97.67 | 99.26 | 93.86 | 99.19
HWGAT (Finetuned) | 97.79 | 99.26 | - | -
Table 10: Comparison of results for other sign language datasets. Methods | LSA64 | AUTSL | WLASL
---|---|---|---
Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5
Pose-GRU [25] | - | - | - | - | 22.54 | 49.81
Pose-TGCN [25] | - | - | - | - | 23.65 | 51.75
Transformer | 90.00 | 98.12 | 90.19 | 98.61 | 23.20 | -
ST-GCN [44] | 92.81 | 98.43 | 90.67 | 98.66 | 34.40 | 66.57
SL-GCN [18] | 98.13 | 100.00 | 95.02 | - | 41.65 | 74.68
HWGAT | 97.81 | 99.84 | 95.43 | 99.17 | 43.28 | 74.92
HWGAT (Finetuned) | 98.59 | 99.84 | 95.80 | 99.49 | 48.49 | 80.86

####  5.4.3 Evaluation of proposed model

To evaluate the proposed model, we used our proposed dataset as well as four
other SL datasets: INCLUDE (Indian sign language), LSA64 (Argentinian sign
language), AUTSL (Turkish sign language), and WLASL (American sign language)
as described in Section 5.1. Table 9 summarizes a comparative analysis with
state-of-the-art skeleton-based models on ISL datasets, highlighting that our
proposed model either performs better or exhibits similar results compared to
other models. Table 10 demonstrates the ability of the proposed model across
different other sign language datasets. The proposed model achieves the
highest performance in both top-1 ($95.43\%$, $43.28\%$) and top-5 ($99.17\%$,
$74.92\%$) for AUTSL and WLASL respectively. For the LSA64 dataset, the
proposed model gives comparable results in both top-1 and top-5 performance
measure. When we fine-tune (pre-trained on the proposed dataset) our model, we
achieve the highest top-1 performance for all the datasets as shown in the
last row of Table 10.

##  6 Conclusion

In conclusion, we presented a new large-scale isolated Indian sign language
dataset and a novel sign language recognition model based on the skeleton
graph structure. The proposed dataset is not only large-scale but also
contains characteristics such as gender-balanced, class-balanced, multi-modal
and multi-view. The proposed model tries to capture the distinctive
characteristics of different signs by restricting the attention between
interacting body parts in the skeleton graph structure. Comparative analysis
of the proposed dataset with the state-of-the-art isolated ISL dataset using
different state-of-the-art skeleton-based models shows the importance of our
proposed dataset. The proposed model has been evaluated on different sign
language (Indian, American, Argentinian, and Turkish) datasets giving better
or comparable results against the state-of-the-art skeleton-based models.
Also, the utility of the proposed dataset is shown by the knowledge transfer
(or fine-tuning) experiment using the proposed model throughout different sign
language datasets. We believe that the proposed dataset (with different
modalities) will accelerate the ISL recognition task and be useful for other
SL recognition tasks. In addition, other multiview modalities would be useful
for keypoints correction, pose estimation, 3D-model generation, general
gesture recognition and other related tasks. Furthermore, our word grouping
based on the number of glosses can be used in classrooms for regular ISL
teaching and learning purposes.

## Acknowledgement

This work is partially funded by VECC, Kolkata. We thank Mr. Subhankar Nag for
the data processing and model discussions.

## References

  * [1] Nikolas Adaloglou, Theocharis Chatzis, Ilias Papastratis, Andreas Stergioulas, Georgios Th. Papadopoulos, Vassia Zacharopoulou, George J. Xydopoulos, Klimnis Atzakas, Dimitris Papazachariou, and Petros Daras.  A comprehensive study on deep learning-based methods for sign language recognition, 2021.
  * [2] S Ajay, Ajith Potluri, Sara Mohan George, R Gaurav, and S Anusri.  Indian sign language recognition using random forest classifier.  In 2021 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT), pages 1–6. IEEE, 2021.
  * [3] Samuel Albanie, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, et al.  Bbc-oxford british sign language dataset.  arXiv preprint arXiv:2111.03635, 2021.
  * [4] Vassilis Athitsos, Carol Neidle, Stan Sclaroff, Joan Nash, Alexandra Stefan, Quan Yuan, and Ashwin Thangali.  The american sign language lexicon video dataset.  In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 1–8. IEEE, 2008.
  * [5] Matyáš Boháček and Marek Hrúz.  Sign pose-based transformer for word-level sign language recognition.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 182–191, 2022.
  * [6] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden.  Sign language transformers: Joint end-to-end sign language recognition and translation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [7] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017.
  * [8] MMPose Contributors.  Openmmlab pose estimation toolbox and benchmark.  https://github.com/open-mmlab/mmpose, 2020.
  * [9] Navneet Dalal and Bill Triggs.  Histograms of oriented gradients for human detection.  In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), volume 1, pages 886–893. Ieee, 2005.
  * [10] Soumen Das, Saroj Kr. Biswas, and Biswajit Purkayastha.  Occlusion robust sign language recognition system for indian sign language using cnn and pose features.  Multimedia Tools and Applications, 2024.
  * [11] Deafness and hearing loss.  https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss.  Accessed: 2024-04-10.
  * [12] Sarah Ebling, Necati Cihan Camgöz, Penny Boyes Braem, Katja Tissi, Sandra Sidler-Miserez, Stephanie Stoll, Simon Hadfield, Tobias Haug, Richard Bowden, Sandrine Tornay, et al.  Smile swiss german sign language dataset.  In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC) 2018. The European Language Resources Association (ELRA), 2018.
  * [13] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li.  Bag of tricks for image classification with convolutional neural networks.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 558–567, 2019.
  * [14] Berthold KP Horn and Brian G Schunck.  Determining optical flow.  Artificial intelligence, 17(1-3):185–203, 1981.
  * [15] Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, and Weiping Li.  Video-based sign language recognition without temporal segmentation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [16] Indian sign language dictionary.  https://indiansignlanguage.org/.  Accessed: 2024-04-10.
  * [17] Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li, and Yun Fu.  Sign language recognition via skeleton-aware multi-model ensemble.  arXiv preprint arXiv:2110.06161, 2021.
  * [18] Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li, and Yun Fu.  Skeleton aware multi-modal sign language recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3413–3423, 2021.
  * [19] Abhinav Joshi, Ashwani Bhat, Pradeep S, Priya Gole, Shashwat Gupta, Shreyansh Agarwal, and Ashutosh Modi.  CISLR: Corpus for Indian Sign Language recognition.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10357–10366. Association for Computational Linguistics, 2022.
  * [20] Hamid Reza Vaezi Joze and Oscar Koller.  Ms-asl: A large-scale data set and benchmark for understanding american sign language.  arXiv preprint arXiv:1812.01053, 2018.
  * [21] G. Khartheesvar, Mohit Kumar, Arun Kumar Yadav, and Divakar Yadav.  Automatic indian sign language recognition using mediapipe holistic and lstm network.  Multimedia Tools and Applications, 2023.
  * [22] P.V.V. Kishore and Panakala rajesh kumar.  A video based indian sign language recognition system (inslr) using wavelet transform and fuzzy logic.  International Journal of Engineering and Technology, 4:537–542, 01 2012.
  * [23] Oscar Koller, Hermann Ney, and Richard Bowden.  Deep hand: How to train a cnn on 1 million hand images when your data is continuous and weakly labelled.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3793–3802, 2016.
  * [24] Oscar Koller, O Zargaran, Hermann Ney, and Richard Bowden.  Deep sign: Hybrid cnn-hmm for continuous sign language recognition.  In Proceedings of the British Machine Vision Conference 2016, 2016.
  * [25] Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li.  Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1459–1469, 2020.
  * [26] Tao Liu, Wengang Zhou, and Houqiang Li.  Sign language recognition with long short-term memory.  In 2016 IEEE international conference on image processing (ICIP), pages 2871–2875. IEEE, 2016.
  * [27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows, 2021.
  * [28] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60:91–110, 2004.
  * [29] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann.  Mediapipe: A framework for building perception pipelines, 2019.
  * [30] Debapriya Maji, Soyeb Nagori, Manu Mathew, and Deepak Poddar.  Yolo-pose: Enhancing yolo for multi person pose estimation using object keypoint similarity loss.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2637–2646, 2022.
  * [31] Liliane Momeni, Gul Varol, Samuel Albanie, Triantafyllos Afouras, and Andrew Zisserman.  Watch, read and lookup: learning to spot signs from multiple supervisors.  In Proceedings of the Asian Conference on Computer Vision, 2020.
  * [32] Anup Nandy, Jay Shankar Prasad, Soumik Mondal, Pavan Chakraborty, and G. C. Nandi.  Recognition of isolated indian sign language gesture in real time.  In Information Processing and Management, pages 102–107, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg.
  * [33] Oğulcan Özdemir and Kındıroğlu.  Bosphorussign22k sign language recognition dataset.  arXiv preprint arXiv:2004.01283, 2020.
  * [34] Ronald Poppe.  Poppe, r.: A survey on vision-based human action recognition. image and vision computing 28(6), 976-990.  Image Vision Comput., 28:976–990, 06 2010.
  * [35] Lawrence Rabiner and Biinghwang Juang.  An introduction to hidden markov models.  ieee assp magazine, 3(1):4–16, 1986.
  * [36] Umer Rafi, Bastian Leibe, Juergen Gall, and Ilya Kostrikov.  An efficient convolutional network for human pose estimation.  In BMVC, volume 1, page 2, 2016.
  * [37] JL Raheja, Anand Mishra, and Ankit Chaudhary.  Indian sign language recognition using svm.  Pattern Recognition and Image Analysis, 26:434–441, 2016.
  * [38] G. Anantha Rao, K. Syamala, P. V. V. Kishore, and A. S. C. S. Sastry.  Deep convolutional neural networks for sign language recognition.  In 2018 Conference on Signal Processing And Communication Engineering Systems (SPACES), pages 194–197, 2018.
  * [39] J. Rekha, J. Bhattacharya, and S. Majumder.  Shape, texture and local movement hand gesture features for indian sign language recognition.  In 3rd International Conference on Trendz in Information Sciences & Computing (TISC2011), pages 30–35, 2011.
  * [40] Pengfei Ren, Haifeng Sun, Qi Qi, Jingyu Wang, and Weiting Huang.  Srn: Stacked regression network for real-time 3d hand pose estimation.  In BMVC, page 112, 2019.
  * [41] Franco Ronchetti, Facundo Quiroga, Cesar Estrebou, Laura Lanzarini, and Alejandro Rosete.  Lsa64: A dataset of argentinian sign language.  XX II Congreso Argentino de Ciencias de la Computación (CACIC), 2016.
  * [42] Wendy Sandler and Diane Lillo-Martin.  Sign Language and Linguistic Universals.  Cambridge University Press, Cambridge, United Kingdom, 2006.
  * [43] Zed Sevcikova Sehyr, Naomi Caselli, Ariel M Cohen-Goldberg, and Karen Emmorey.  The asl-lex 2.0 project: A database of lexical and phonological properties for 2,723 signs in american sign language.  The Journal of Deaf Studies and Deaf Education, 26(2):263–277, 2021.
  * [44] Prem Selvaraj, Gokul NC, Pratyush Kumar, and Mitesh Khapra.  Openhands: Making sign language recognition accessible with pose-based pretrained models across languages, 2021.
  * [45] Ozge Mercanoglu Sincan and Hacer Yalim Keles.  Autsl: A large scale multi-modal turkish sign language dataset and baseline methods.  IEEE Access, 8:181340–181355, 2020.
  * [46] Advaith Sridhar, Rohith Gandhi Ganesan, Pratyush Kumar, and Mitesh Khapra.  Include: A large scale dataset for indian sign language recognition.  In Proceedings of the 28th ACM international conference on multimedia, pages 1366–1375, 2020.
  * [47] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
  * [48] Alaa Tharwat, Tarek Gaber, Aboul Ella Hassanien, Mohamed K Shahin, and Basma Refaat.  Sift-based arabic sign language recognition system.  In Afro-European Conference for Industrial Advancement: Proceedings of the First International Afro-European Conference for Industrial Advancement AECIA 2014, pages 359–370. Springer, 2015.
  * [49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [50] Hanjie Wang, Xiujuan Chai, Xiaopeng Hong, Guoying Zhao, and Xilin Chen.  Isolated sign language recognition with grassmann covariance matrices.  ACM Transactions on Accessible Computing (TACCESS), 8(4):1–21, 2016.
  * [51] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al.  Deep high-resolution representation learning for visual recognition.  IEEE transactions on pattern analysis and machine intelligence, 43(10):3349–3364, 2020.
  * [52] Sijie Yan, Yuanjun Xiong, and Dahua Lin.  Spatial temporal graph convolutional networks for skeleton-based action recognition.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [53] Lin Zehui, Pengfei Liu, Luyao Huang, Junkun Chen, Xipeng Qiu, and Xuanjing Huang.  Dropattention: A regularization method for fully-connected self-attention networks, 2019.
  * [54] Jihai Zhang, Wengang Zhou, Chao Xie, Junfu Pu, and Houqiang Li.  Chinese sign language recognition with adaptive hmm.  In 2016 IEEE international conference on multimedia and expo (ICME), pages 1–6. IEEE, 2016.
  * [55] Zhiming Zou, Kenkun Liu, Le Wang 0003, and Wei Tang.  High-order graph convolutional networks for 3d human pose estimation.  In BMVC, 2020.
  * [56] Ronglai Zuo, Fangyun Wei, and Brian Mak.  Natural language-assisted sign language recognition, 2023.


# Anomaly Detection of Wind Turbine Time Series using Variational Recurrent
Autoencoders

Alan Preciado-Grijalva  Hochschule Bonn-Rhein-Sieg
Fraunhofer Center for Machine Learning and SCAI
<EMAIL_ADDRESS>Victor Rodrigo Iza-Teran  Fraunhofer Center for
Machine Learning and SCAI
<EMAIL_ADDRESS>
###### Abstract

Ice accumulation in the blades of wind turbines can cause them to describe
anomalous rotations or no rotations at all, thus affecting the generation of
electricity and power output. In this work, we investigate the problem of ice
accumulation in wind turbines by framing it as anomaly detection of multi-
variate time series. Our approach focuses on two main parts: first, learning
low-dimensional representations of time series using a Variational Recurrent
Autoencoder (VRAE), and second, using unsupervised clustering algorithms to
classify the learned representations as normal (no ice accumulated) or
abnormal (ice accumulated). We have evaluated our approach on a custom wind
turbine time series dataset, for the two-classes problem (one normal versus
one abnormal class), we obtained a classification accuracy of up to 96% on
test data. For the multiple-class problem (one normal versus multiple abnormal
classes), we present a qualitative analysis of the low-dimensional learned
latent space, providing insights into the capacities of our approach to tackle
such problem. The code to reproduce this work can be found here
https://github.com/agrija9/Wind-Turbines-VRAE-Paper.

###### Index Terms:

anomaly detection, dimensionality reduction, unsupervised learning,
unsupervised clustering, variational recurrent autoencoder, wind turbines time
series

##  I Introduction

In engineering, it is of particular interest to study the freezing of the
blades of wind turbines caused by harsh weather conditions. In order to ensure
the optimal operation of such systems, we can analyze and gain meaningful
insights from the data they generate. For this, certain data manipulation
tasks like compression, clustering, and reconstruction are helpful.

Data clustering, for example, attempts to visually identify data patterns in
low dimensional spaces. Numerous physical spatio-temporal insights can be
derived from this, since clustering in low-dimensional spaces indicates data
has (un)correlated attributes.

With more data available and novel machine learning models being developed, it
is reasonable to explore their potential to tackle the problem of ice
accumulation in wind turbines. Suitable models for these tasks are Variational
Autoencoders (VAEs)[7] due to their capacities to compress, generate and
project data into a low-dimensional space. VAEs have achieved state-of-the-art
results on image generation [5], clustering and anomaly detection [9][10][15],
and data reconstruction [14].

In this work, we report on the implementation of a fully unsupervised learning
pipeline for anomaly detection in the blades of wind turbine time series
simulations data. Our pipeline consists of two main steps: (1) learning
abstract low-dimensional time series data representations with a Variational
Recurrent Autoencoder (VRAE) [3] and (2) classifying these representations
using clustering algorithms (KMeans++, Hierarchical Clustering and DBSCAN).
Great emphasis has been put on the first part, since fine-tuning a VRAE neural
network model requires extensive experimentation.

##  II Related Work

Dimension reduction methods like PCA and kernel-PCA [12] have been effective
in many use cases and straightforward to implement [8]. However, a numerous
amount of datasets contain non-linearities that can not be captured by these
methods, thus having to resort to other non-linear techniques like t-SNE [13].
Just like PCA, t-SNE is also used for visualizing large datasets, it uses a
random walk on neighborhood graphs to reveal structure at different scales.
These methods of dimension reduction are a key tool to gain insights into any
clustering behavior datasets may present. Furthermore, highly non-linear
neural network approaches have shown promising results, in the case of time
series dimensionality reduction and clustering, VAEs have achieved state-of-
the art results on anomaly detection [9][15]. [9] demonstrated state-of-the-
art accuracy for the detection of anomalies in the ECG500 dataset with an
accuracy exceeding 90%.

Moreover, [15] extracted the relationship between time series variables
obtained from the monitoring of wind turbine systems. This group worked with
an autoencoder network based on Restricted Boltzmann Machines, successfully
implementing an early warning of faulty components and deducing the physical
location of such components.

##  III Proposed model for anomaly detection

The proposed model for ice detection in the rotor blades consists of two
fundamental steps: unsupervised representation learning and anomaly detection.

###  III-A Unsupervised representation learning

The model that we use for this task is the Variational Recurrent Autoencoder
(VRAE). More formally, let $\chi=[\textbf{x}^{(n)}]^{N}_{n=1}$ be the time
series dataset composed of $N$ sequences, with each sequence having a length
$T$, $x^{(n)}=[x^{(n)}_{1},x^{(n)}_{2},...,x^{(n)}_{T}]$, and each data point
$x_{t}^{(n)}$ is a $d_{x}$ dimensional vector (number of features).

The encoder of the VRAE takes each time series $x^{(n)}$ and it is
parametrized by a long short-term memory (LSTM) layer that at each time step
computes a hidden state $h_{t}^{enc}$. The last hidden state $h_{T}^{enc}$ is
thus an abstract representation that represents the whole given sequence x.
Similarly to [9][10], the prior distribution $p(\textbf{z})$ is a multi-
variate normal distribution $\mathcal{N}$(0, I). The parameters that
approximate the posterior distribution $q_{\psi}(z|x)$, $\mu_{z}$ and
$\Sigma_{z}$, are obtained by taking mean and standard deviation from this
last hidden state by using two fully connected layers with a SoftPlus
activation. According to [25], using a SoftPlus activation ensures that
variance is non-negative . The latent variables z are sampled from the
parametrized posterior $q_{\psi}(z|x)$ via $\mu_{z}$ and $\Sigma_{z}$ by using
the re-parametrization trick by doing

$z=\mu_{z}+\sigma_{z}\odot\epsilon$ (1)

Where $\epsilon\sim\mathcal{N}(\textbf{0},\textbf{I})$ is Gaussian noise and
$\odot$ corresponds to element-wise product.

The decoder of the VRAE is another LSTM network that takes as input the latent
vector z from the approximate posterior and outputs at each time step $t$ the
parameters that reconstruct the input variable x. Similar to the encoding
distribution, the decoding distribution $p_{\phi}(x|z)$ is defined as a multi-
variate Gaussian distribution. The loss function is the VAE loss function
introduced in [7] and the training procedure follows subsequently following
gradient update and stochastic gradient descent.

###  III-B Anomaly detection

We perform anomaly scoring using the learned low-dimensional time series
representations provided by the VRAE model. Following the procedure of [9],
the model is mapping sequences x into a lower-dimensional space and we then
project them into two dimensions using PCA and t-SNE in order to evaluate
grouping in specific regions. This makes it more feasible for a clustering
method to detect normal vs anomalous (abnormal) cases. Anomaly detection
consists, therefore, in detecting if a latent representations is normal or
abnormal. In this work, we have implemented this detection using clustering
algorithms.

Clustering algorithms give a numerical label to each latent representation,
framing the problem as a two-class or multiple-class classification problem.
We are taking this approach based on the fact that the model is capable of
learning representations that tend to group in lower-dimensions given a
balanced normal and abnormal percentage of training data, also assuming there
is statistical difference between cases. We have applied three different
clustering methods in the representations: k-means ++ [1], density-based
spatial clustering (DBSCAN) [2] and hierarchical clustering [11]. These
methods are set to find two (or more) clusters given the number of classes
(normal and abnormal). The output is then matched to the ground truth labels
given corresponding to the actual class each representation belongs to. With
this, we can compute a classification accuracy.

##  IV Experiments and Results

In this section we describe the wind turbine dataset used in our experiments,
relevant data preprocessing steps, and the results obtained for the case of
two and multiple classes representation learning and anomaly scoring.

###  IV-A Dataset Description

Our dataset is composed of wind turbine multivariate time series simulated
with an open source whole-turbine simulator called FAST. We model and collect
time series data of a wind turbine operating with/without ice on the rotor
blades. We categorize ice accumulation in one blade into three zones: the
first zone covers the first half and the two other zones divide the second
half of the blade into two again. For each simulation, each one of these three
zones contains a particular ice mass. The convention to refer to the region
and amount of ice mass is $x-y-z$. For example, a configuration $0.4-0.6-0.8$
implies 0.4 kg of ice in zone one, 0.6 kg in zone two and 0.8 kg in zone
three.

As an initial approach, we have filtered the dataset to contain only
simulations with ice mass in one zone at a time (i.e. no ice in two other
zones). This is to investigate as a first approximation the capacity of our
model to cluster time series in lower dimensions based on ice mass
configuration in one zone at a time.

To setup an anomaly detection problem, we define specific classes of times
series. A normal time series corresponds to the configuration $0-0-0$ (no ice
in any zone), an abnormal time series corresponds to any configuration in eq.
(2).

$\begin{split}zone1\rightarrow x_{mass}-0-0\\\ zone2\rightarrow
0-y_{mass}-0\\\ zone3\rightarrow 0-0-z_{mass}\end{split}$ (2)

###  IV-B Data Preprocessing

We have balanced the dataset to have approximately the same number of normal
and abnormal time series. It is composed of 14 normal and 11 abnormal
simulations. Each simulation corresponds to a time series consisting of 10,000
time steps and 27 sensor features in total.

First, we MinMax scale the data to a range [-1 to 1], this is a crucial
preprocessing step since the learning performance of the model can be affected
by the various sensor amplitudes recorded. Second, we select only 6 out of the
the initial 27 features. These filtered features are the accelerations in
flap-wise and edge-wise components for three blades (see table I). We have
chosen these features based on the idea that this physical information is
sufficient for the model to be able to cluster time series in lower dimensions
efficiently. Fig.1 illustrates a sample simulation after the scaling and
feature filtering steps.

TABLE I: Filtered features from the wind turbine simulations. Parameter |  Description
---|---
Spn1ALxb1 |  Blade 1 flapwise acceleration (absolute) span station 1
Spn1ALyb1 |  Blade 1 edgewise acceleration (absolute) span station 1
Spn1ALxb2 |  Blade 2 flapwise acceleration (absolute) span station 1
Spn1ALyb2 |  Blade 2 edgewise acceleration (absolute) span station 1
Spn1ALxb3 |  Blade 3 flapwise acceleration (absolute) span station 1
Spn1ALyb3 |  Blade 3 edgewise acceleration (absolute) span station 1

Next, we reshape the data to the format (samples, timesteps, features) since
this is the required input format for LSTM blocks that compose the VRAE.
Finally, we slice each time series into smaller segments of 200 to 1000 time
steps. This is because our simulations are set up such that we have roughly 12
rotations per minute, this means there is one rotation every 5 seconds, this
corresponds to a length of approximately 1000 time steps, on top of this,
smaller time steps avoid vanishing gradients. In summary, our procedure
generates a total of 1250 time series.

Figure 1: A simulated time series from our dataset. Each color line
corresponds to a specific physical variable recorded.

###  IV-C Anomaly detection with two classes

The first problem we tackled is the detection of anomalies given two classes.
We have taken normal time series ($0-0-0$) and one abnormal time series (zone
1: $x_{mass}-0-0$).

Following the proposed model for ice detection mentioned in Section III, we
trained a VRAE using a single hidden LSTM layer with 90 units (one hidden
layer gave better results than multiple hidden layers). We used the Adam
optimizer [6], gradient clipping (to avoid gradient explosion) and dropout in
the hidden LSTM layer with rate of $0.2$. A bottleneck layer (variational
layer) maps these $90$-dimensional vectors into $20$-dimensional vectors (i.e.
latent space dimensions). We used a learning rate of $l=0.0005$ and momentum
$0.9$. The data was split in 70% training and 30% validations sets with time
series split into chunks of 200 time steps and the 6 features described in
table I. We loaded data into memory in batches of 64 and performed training
for 2000 epochs. The reconstruction term in the loss objective is the mean
squared error (MSE loss) given that we assume that the parametrized posterior
probability $p(x|z)$ is a normal distribution. The total training time was
approximately 30 minutes using an NVIDIA Tesla v100 GPU with 32GB of RAM.

####  IV-C1 Analysis of latent space

Before training the VRAE, we ran a sanity check by applying PCA directly to
the test set to verify if this method can linearly separate the data. We
observed that pure PCA is not good enough to cluster normal and abnormal time
series in distinctive regions, indicating that such techniques can not capture
non-linearities that exist in our data.

Fig.2 shows the results obtained on the test set after training the VRAE. The
data points correspond to a projection of the 20-dimensional latent vectors
into 2-dimensional vectors using PCA (first and second components), t-SNE and
Spectral Embedding. We note that in most cases, the data points are properly
clustered according to normal (red points) and abnormal (blue points) classes
in specific regions of the 2D plane. The location of the 2D data points
depends on each projection method, but the underlying clustering behavior is
quite evident in each case, thus proving that the model has learned to
identify when a time series contains ice or no ice. From here, we highlight
the importance of learning good abstract representations of time series, since
having such distinctive grouping behavior can make unsupervised clustering
algorithms perform better. We have colored the 2D data points to have a visual
reference since we know the ground truth labels for each of them, however, the
training and projections are fully unsupervised.

(a) PCA

(b) t-SNE

(c) Spectral Embedding

Figure 2: 2D projections of the 20-dimensional representations obtained with
the VRAE.

In order to further analyze the learned representations by our model, we took
15 samples of normal and abnormal 20-dimensional latent vectors and plot each
dimension horizontally (without projecting them into 2D).

In Fig.3, the red lines correspond to normal latent vectors and the blue lines
to abnormal ones. Here, the x-axis corresponds to each entry of the
20-dimensional vectors and the y-axis are the normalized amplitudes provided
directly by the model. Remarkably, we can see that even in their
20-dimensional representation, red lines tend to describe different maxima and
minima compared to blue lines. Note for instance, that the dimensions 1, 5, 7,
13 and 19 are the ones where normal and abnormal classes have the less
correlation, knowing what dimensions provide the highest contrast between our
classes can in principle allow us to further reduce the dimensions of the
latent vectors from 20 to 5 or 6.

The next step in our pipeline consists in applying clustering algorithms on
top of the projected 2D data points. These models classify data based on a
predefined number of clusters thus providing a final anomaly scoring.

Figure 3: VRAE 20-dimensional latent vectors plotted as horizontal lines. Red
lines are normal classes, blue lines are abnormal classes.

####  IV-C2 Unsupervised clustering and anomaly scoring

Fig.4 shows the clustering results of KMeans++ and Hierarchical clustering on
our 2D data points (obtained with PCA). Both algorithms have been set to
identify data points between two clusters. In general, we see that both
methods perform efficient clustering when compared to the groundtruth data
labels (figure on the left). Furthermore, the advantage of these methods is
that they are relatively fast and straightforward to implement.

(a) Groundtruth

(b) KMeans++

(c) Hierarchical Clustering

Figure 4: Visual evaluation of clustering performance of KMeans++ and
Hierarchical Clustering. Both methods label most data points correctly when
compared to the groundtruth labels.

In order to provide a quantitative evaluation for anomaly scoring, we have
computed several metrics like classification accuracy, area under the curve
(AUC) and precision. We have added a third clustering method (DBSCAN) for a
broader evaluation. Table II summarizes the obtained results.

TABLE II: Anomaly scoring results using 3 unsupervised clustering algorithms.
Anomaly scoring results
---
Metric |  KMeans++ |  Hierarchical |  DBSCAN
Accuracy |  0.9667 |  0.9639 |  0.6315
AUC |  0.9619 |  0.9605 |  0.5507
Precision |  0.9674 |  0.9641 |  0.9890
Recall |  0.9667 |  0.9639 |  0.6315
F1-score |  0.9666 |  0.9638 |  0.7616

The results from table II indicate that DBSCAN has a lower performance
overall. The other two methods on the contrary, achieved classification
accuracies of up to 96%. These classification results indicate that our
anomaly detection proposed framework is being able to classify between normal
and abnormal time series up to 96% of the time.

###  IV-D Unsupervised time series representation learning for multiple
classes

The next problem we tackled is anomaly detection for multiple classes. Here,
we have taken into account all three zones to compose the training dataset
(i.e. weights in zone 1, zone 2 and zone 3). We trained the VRAE with a single
hidden LSTM layer, in this case, we modified this layer to 128 units. The
bottleneck layer projects these 128 dimensional representations into latent
vectors of 5 dimensions (we changed from 20 to 5 after tuning this parameter).
Adam optimizer is used, gradient clipping (to avoid gradient explosion) and
dropout in the hidden layer with rate of 0.2. The learning rate is $l=0.0005$
and momentum $0.9$. The data is split in 70% training and 30% validations sets
with time series split into chunks of 200 time steps and the 6 features
described in the previous chapter. We load batches of 64 into memory and
perform training for 2000 epochs. The reconstruction term in the loss
objective is the MSE loss. The training time is approximately 1.5 hours.

####  IV-D1 2D latent space of time series for multiple classes

Fig.5 shows the 2D projections of the 5-dimensional latent vectors of t-SNE
and spectral embedding after training the model with a balanced number of
classes (around 600 time series per class). In this case, we obtained that the
model is capable of clustering normal versus abnormal cases successfully (red
dots versus the rest), however, it is not capable of clustering abnormal
classes depending on their zone (i.e. blue, green, black dots are not
clustered). To investigate this behavior, we have plotted a few 5-dimensional
vectors from all three abnormal samples (similar to the analysis in Fig.3), we
have seen that there is not a lot of variability between lines of abnormal
zones themselves, they rather tend to describe the same maxima and minima. We
assume that this occurs due to the fact that training with a fixed balanced
number of samples per class prevents the model to regularize the latent space
properly.

We ran a series of experiments to overcome the problem of clustering abnormal
zones correctly: First, we implemented a VRAE using bi-directional LSTMs and
increased the number of timesteps of each time series from 200 to 500.
However, for the bi-directional LSTM we obtained similar results as the ones
shown in Fig.5 (with uni-directional LSTM), in the case of longer time steps,
we obtained even worse results, this can have to do with the fact that longer
sequences implies less available training samples.

As a second experiment, we increased the number of abnormal samples for
training (in previous experiments, the training set was balanced, but since
there are more abnormal samples available, we have trained the model with all
the available data), in addition to this, we implemented cyclical annealing to
mitigate vanishing of the KL divergence [4]. After training the model we
projected the test set into 5-dimensional latent vectors and then into 2D
using t-SNE and Kernel PCA (radial basis function), the results are shown in
Fig.6. From these projections, we can see a more successful formation of
clusters. We note in both projections that time series from abnormal zone 1
(blue dots) form a very distinctive cluster, zones 2 and 3 (green and black
dots) have some overlapping but they still present clustering. These results
indicate that the model is capable of clustering time series from all abnormal
zones and anomaly scoring can be applied next.

(a) t-SNE projections

(b) spectral embedding projections

Figure 5: Test set 2D projections of 5-dimensional latent vectors after
training with all 3 classes (balanced dataset).

(a) t-SNE projections

(b) Kernel PCA

Figure 6: Test set 2D projections of 5-dimensional latent vectors after
training with full dataset.

##  V Conclusions

In this work, we have introduced an end-to-end unsupervised learning pipeline
for wind turbine anomaly detection; first, we have built a dataset composed of
wind turbine multi-variate time series and preprocessed this data to train a
Variational Recurrent Autoencoder, secondly, we have used this model to learn
low-dimensional latent representations and implemented clustering algorithms
on top of these latent representations to predict class correspondences. In
the case of anomaly detection for two classes, we have obtained a
classification accuracy of 96% on the test set, furthermore, our approach
allows us to draw intuitive conclusions and improves interpretability due to
the fact that our analysis is two dimensional. In the case of anomaly
detection for multiple time series classes, we have observed that the amount
of training data plays an important role; we showed that our model is capable
of clustering time series better when more data samples are available, these
results take us one step further for efficient representation learning for
anomaly scoring in the case of multiple classes. From our insights, we
conclude that learning high-quality data representations with neural network
is a key aspect of our approach.

## References

  * Arthur and Vassilvitskii [2007] David Arthur and Sergei Vassilvitskii.  K-means++: The advantages of careful seeding.  In _Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms_ , SODA 07, pages 1027–1035, USA, 2007. Society for Industrial and Applied Mathematics.  ISBN 9780898716245.
  * Ester et al. [1996] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu.  A density-based algorithm for discovering clusters in large spatial databases with noise.  In _Proceedings of the Second International Conference on Knowledge Discovery and Data Mining_ , KDD 96, pages 226–231. AAAI Press, 1996\.
  * Fabius and van Amersfoort [2014] Otto Fabius and Joost R. van Amersfoort.  Variational Recurrent Auto-Encoders.  _arXiv e-prints_ , art. arXiv:1412.6581, 2014.
  * Fu et al. [2019] Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin.  Cyclical annealing schedule: A simple approach to mitigating kl vanishing, 2019.
  * Jaskari and Kivinen [2018] Joel Jaskari and Jyri J. Kivinen.  A Novel Variational Autoencoder with Applications to Generative Modelling, Classification, and Ordinal Regression.  _arXiv e-prints_ , art. arXiv:1812.07352, Dec 2018.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-Encoding Variational Bayes.  _arXiv e-prints_ , art. arXiv:1312.6114, Dec 2013.
  * Nguyen and Holmes [2019] Lan Huong Nguyen and Susan Holmes.  Ten quick tips for effective dimensionality reduction.  _PLOS Computational Biology_ , 15(6):1–19, Jun 2019.  doi: 10.1371/journal.pcbi.1006907.  URL https://doi.org/10.1371/journal.pcbi.1006907.
  * Pereira and Silveira [2019] J. Pereira and M. Silveira.  Learning representations from healthcare time series data for unsupervised anomaly detection.  In _2019 IEEE International Conference on Big Data and Smart Computing (BigComp)_ , pages 1–7, Feb 2019.  doi: 10.1109/BIGCOMP.2019.8679157.
  * Pereira and Silveira [2019] João Pereira and Margarida Silveira.  Unsupervised representation learning and anomaly detection in ecg sequences.  _International Journal of Data Mining and Bioinformatics_ , 22:389–407, Aug 2019.  doi: 10.1504/IJDMB.2019.101395.
  * Reynolds et al. [2006] Alan P Reynolds, Graeme Richards, Beatriz de la Iglesia, and Victor J Rayward-Smith.  Clustering rules: a comparison of partitioning and hierarchical clustering algorithms.  _Journal of Mathematical Modelling and Algorithms_ , 5(4):475–504, 2006.
  * Schölkopf et al. [1997] Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller.  Kernel principal component analysis.  In Wulfram Gerstner, Alain Germond, Martin Hasler, and Jean-Daniel Nicoud, editors, _Artificial Neural Networks — ICANN’97_ , pages 583–588, Berlin, Heidelberg, 1997. Springer Berlin Heidelberg.  ISBN 978-3-540-69620-9.
  * van der Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ , 9:2579–2605, 2008\.  URL http://www.jmlr.org/papers/v9/vandermaaten08a.html.
  * Wetzel [2017] Sebastian J. Wetzel.  Unsupervised learning of phase transitions: From principal component analysis to variational autoencoders.  _Physical Review E_ , 96:022140, Aug 2017.  doi: 10.1103/PhysRevE.96.022140.  URL https://link.aps.org/doi/10.1103/PhysRevE.96.022140.
  * Zhao et al. [2018] Hongshan Zhao, Huihai Liu, Wenjing Hu, and Xihui Yan.  Anomaly detection and fault analysis of wind turbine components based on deep learning network.  _Renewable Energy_ , 127:825 – 834, 2018.  ISSN 0960-1481.  doi: https://doi.org/10.1016/j.renene.2018.05.024.  URL http://www.sciencedirect.com/science/article/pii/S0960148118305457.