# Towards the extraction of robust sign embeddings for low resource sign
language recognition

Mathieu De Coster
IDLab-AIRO – Ghent University – imec
<EMAIL_ADDRESS>Ellen Rushe
Complex Software Lab – Trinity College Dublin
<EMAIL_ADDRESS>Ruth Holmes
Complex Software Lab – Trinity College Dublin
<EMAIL_ADDRESS>Anthony Ventresque
Complex Software Lab – Trinity College Dublin
<EMAIL_ADDRESS>Joni Dambre
IDLab-AIRO – Ghent University – imec
<EMAIL_ADDRESS>
###### Abstract

Isolated Sign Language Recognition (SLR) has mostly been applied on relatively
large datasets containing signs executed slowly and clearly by a limited group
of signers. In real-world scenarios, however, we are met with challenging
visual conditions, coarticulated signing, small datasets, and the need for
signer independent models. To tackle this difficult problem, we require a
robust feature extractor to process the sign language videos. One could expect
human pose estimators to be ideal candidates. However, due to a domain
mismatch with their training sets and challenging poses in sign language, they
lack robustness on sign language data and image based models often still
outperform keypoint based models. Furthermore, whereas the common practice of
transfer learning with image based models yields even higher accuracy,
keypoint based models are typically trained from scratch on every SLR dataset.
These factors limit their usefulness for SLR. From the existing literature, it
is also not clear which, if any, pose estimator performs best for SLR. We
compare the three most popular pose estimators for SLR: OpenPose, MMPose and
MediaPipe. We show that through keypoint normalization, missing keypoint
imputation, and learning a pose embedding, we can obtain significantly better
results and enable transfer learning. We show that keypoint-based embeddings
contain cross-lingual features: they can transfer between sign languages and
achieve competitive performance even when fine-tuning only the classifier
layer of an SLR model on a target sign language. We furthermore achieve better
performance using fine-tuned transferred embeddings than models trained only
on the target sign language. The application of these embeddings could prove
particularly useful for low resource sign languages in the future.

##  1 Introduction

Sign language recognition (SLR) has various applications, including automatic
sign language corpus annotation [27], sign language information retrieval [15]
and Sign Language Translation (SLT) [6]. Within these applications, the task
of the SLR model is to extract syntactic and semantic information from video
data that can be used in the downstream task. In order to generalize to as
many people and surroundings as possible, an SLR model should be invariant to
the camera’s position and intrinsics, to environments and lighting, and to the
specific characteristics of the person that is signing. These characteristics
not only include age, gender, and ethnicity, but also clothing, accessories,
individual body morphology, signing speed, and any other features that do not
contribute to the linguistic content. Sign language datasets are typically
limited in size, which means that SLR models trained on these datasets are
prone to exacerbating bias and not having robust feature extractors.

In recent years, several open source human pose estimation tools have been
released. Among the most popular are OpenPose [7], MMPose [8], and BlazePose
[3], which is used in the MediaPipe toolkit [17]. These tools extract
landmarks, also known as landmarks, from image data: these are estimates of
the 2D or 3D positions of joints in the human body and other important
landmarks of the face and head. Considering that these models tend to be
trained on a large variety of images of human poses, they are supposedly
robust against the factors that may be confusing for SLR models [28]. Recent
work has shown that models trained with pose estimator features are more
signer independent111Within the field of SLR, a distinction is often made
between signer dependent and signer independent SLR. For the former, the same
signers are present in the training, validation, and test sets. This boils
down to information leakage and exaggerated accuracy figures. sign language
recognizers than image based models, especially in low resource scenarios
[19]. This is important as in out-in-the-wild applications, individuals will
differ from those in the training set.

In the current scientific literature, keypoint based SLR appears to be limited
by three factors. First, any errors made by the pose estimator will be carried
over to any downstream SLR model. Second, pose estimators are often trained on
general data (i.e., not on sign language data). This can result in sub-optimal
performance when applied to sign language data that often involves fast and
precise hand movements. For instance, MediaPipe Holistic may fail to predict
hand keypoints in certain cases of inter-manual or manual-facial interaction
[28], and OpenPose may produce noisy keypoints [11, 30]. A third factor is the
way keypoints are used in SLR models. The predicted keypoint coordinates are
often used as raw inputs to sequential models such as LSTMs or Transformers
[11, 30]. Clearly, however, the exact positions of keypoints are not as
informative to models as the _relations_ between keypoints.

These limitations can be tackled with a more tailored approach to pre-
processing of raw keypoint predictions. For the first two, we propose a novel
post-processing method that normalizes the predicted keypoints and imputes any
missing values. For the final limitation we introduce a “pose embedding”, a
non-linear transformation that can be fine-tuned to specific datasets.

The contributions of this paper are as follows. We compare the three most
popular pose estimators (OpenPose, MMPose/HRNet, and MediaPipe Holistic) for
SLR on the same dataset. This dataset is representative of real-world
spontaneous signing. We analyze their failure cases across a number of
different dimensions. We also introduce a standardized preprocessing procedure
that can be used across all three of these pose-estimation frameworks and show
that this procedure can mitigate much of the impact of these failure cases. We
present a pose embedding which improves SLR performance and allows for
transfer learning for keypoint based models, similar to how this is possible
for image based models. This work is an important step towards making SLR and
SLT more robust.

##  2 Related work

###  2.1 Human pose estimation for SLR

Human pose estimation is the task of predicting the positions of specific
keypoints in the human body. These keypoints are often related to joints
(e.g., the shoulders, elbows and wrists), but also include other pose
landmarks such as the positions of the eyes and ears. A multitude of human
pose estimation models exist; here we give an overview of toolkits that are
commonly used in SLR research.

We compare two approaches towards pose estimation: top-down and bottom-up. A
top-down pose estimator first detects the individual(s) in an image, and then
detects and predicts keypoints for the constituent body parts. A bottom-up
pose estimator detects and predicts keypoints for body parts and assigns them
to individuals afterwards.

In 2017, the first open source real time full body multi person human pose
estimation model OpenPose was released [7]. It uses a bottom-up approach,
detecting body parts and aligning them to individuals using Part Affinity
Fields (PAFs). First, body part heatmaps are predicted along with the PAFs
that indicate the direction from one body part to another. A greedy algorithm
is used to match body parts to the individuals in the image. Finally, the
keypoints are derived from the body part heatmaps.

MediaPipe Holistic is a top-down pose estimator that combines several neural
networks into a unified pipeline for body, face, and hand pose recognition.
BlazePose [3] is used to detect and track the pose throughout video data. The
pose keypoints are used to crop hand and face images that are passed on to
specialized pipelines. The hand model first detects the presence of the palm
to refine the region of interest. Then, coordinate regression is performed to
obtain the hand keypoint coordinates. Finally, all landmarks are merged into a
single result.

MMPose is a software library that includes several computer vision algorithms
related to pose estimation [8]. For example, it is often used as a top-down
pose estimator. Faster R-CNN [34] detects and crops the individual in the
video, and HRNet [41] predicts keypoints. HRNet uses heatmap regression to
predict keypoint coordinates and confidence values for the full body.

Given a single (i.e., monocular) input image, OpenPose and HRNet predict 2D
keypoint coordinates. Even though it is an ill-posed problem, techniques exist
to “lift” these 2D coordinates into 3D [25]. Alternative approaches directly
predict depth from monocular data, as is the case in BlazePose which regresses
keypoints from images [3] (as part of the MediaPipe software package).

OpenPose, MediaPipe, and MMPose have all been applied in domains such as
action recognition [38, 47], gesture recognition [33, 46], and SLR [22, 10,
24, 11, 20, 21, 23, 12, 28]. To the best of our knowledge, there exists no
comparative analysis of the accuracy or run-time performance of these three
toolkits on sign language data.

###  2.2 Keypoint based SLR

SLR is the process of extracting syntactic and/or semantic information from
video data containing signing for use in downstream tasks. _Isolated_ SLR is
one way to perform this extraction. It is a video classification task: given a
video clip of a single sign, the goal is to predict which sign is performed.
Individual signs are distinct meaningful elements in sign language. They map
to concepts such as nouns and verbs, and in some cases to clauses [35].
Isolated SLR models can be used for the automatic annotation of sign language
corpora [10, 29, 27]. These models can also be used in a transfer learning
set-up, for example as feature extractors for SLT models [1, 37].

SLR has evolved similarly to other subfields of computer vision, in that early
works used handcrafted features [48], but in the past decade the field has
moved towards automatic feature extraction with deep learning [32]. Even more
recently, SLR research has moved towards using only monocular RGB data,
instead of RGB-D data extracted with sensors such as the Microsoft Kinect. The
results of an SLR challenge held at CVPR 2021 show that RGB models can achieve
accuracy on par with RGB-D models [39].

These RGB videos are typically processed using Convolutional Neural Networks
(CNNs). A first approach factorizes the spatial and temporal processing. De
Coster et al. use a ResNet to extract a feature vector for every frame in the
video clip, and then process the sequence of feature vectors with an LSTM or
transformer [11]. Li et al. use VGG16 as a feature extractor, and a GRU to
process the sequences [24]. In contrast, 3D CNNs allow for spatio-temporal
processing. I3D is a popular 3D CNN architecture for SLR [1, 36, 43]. Mino et
al. have shown that both kinds of models (factorized and spatio-temporal)
reach similar levels of accuracy on the same data [26]. Yet, the optimal model
is of course dataset specific, especially in low resource settings.

Due to the low resource nature of many SLR datasets, human pose estimators (as
pre-trained models) are a popular alternative for CNNs (trained on sign
language data). Li et al. use graph convolutional networks for SLR on the
WLASL dataset [24]. The state of the art pose based SLR on the AUTSL dataset
[40] is reached by Vazquez et al., who use “Multi-Scale Graph Convolutional
Networks” [45], and Jiang et al., who propose a similar “Sign Language Graph
Convolutional Network” [21]. Moryossef et al. compare OpenPose and MediaPipe
Holistic, and conclude that they perform similarly on the AUTSL dataset, but
that both have failure cases that negatively impact the performance of their
downstream SLR models [28]. De Coster et al. extract motion features from
OpenPose keypoints to augment visual representations, but do not evaluate the
performance of these features on their own [12]. Konstantinidis et al. use
OpenPose keypoints as inputs to an SLR model and compare the performance to
image based models on the LSA64 dataset [23]. De Coster et al. also perform
such a comparison on the Corpus VGT [42], and, like Konstantinidis et al.,
find that their image based models outperform their pose based models [11].
This is somewhat unexpected, especially with smaller datasets, as extracting
pose keypoints should alleviate some of the difficulty with learning signs
from RGB data and remove much of the background noise associated with signing
in-the-wild. Ko et al. use OpenPose keypoints in combination with a GRU
network and SVM classifier on the KETI dataset [22].

##  3 Keypoint based SLR

###  3.1 Dataset

Table 1: Subset statistics for the VGT dataset. Subset | Number of examples | Number of signers
---|---|---
Train | 19267 | 88
Validation | 2702 | 12
Test | 2998 | 11
Figure 1: The class distribution of the entire VGT dataset shows a large class
imbalance.

We use the Flemish sign language corpus (Corpus VGT) [42] for our experiments.
This corpus is used for linguistics research and contains gloss level
annotations as well as, to a lesser extent, translations into written Dutch.
We select all annotations for which the corresponding gloss (a transcription
of a sign which is used as a label in SLR) has at least 20 occurrences, and
only keep glosses belonging to the established lexicon222Signs in the
established lexicon have a known form, in contrast to productive signs which
are created “on-the-fly”.. We obtain 26,639 examples belonging to 348 classes.
We then perform a stratified (on label) and grouped (on signer identity)
split. Because the vocabulary distributions vary between individual signers,
we now have several glosses that do not occur in our training, validation,
_and_ test set. We remove those glosses and are left with 24,968 examples
belonging to 292 classes. The resulting dataset has a long-tailed class
distribution, that is shown in Figure 1, similar to the true sign distribution
in the corpus [5]. The head of the distribution consists of pointing signs,
more specifically WG-1 (“I/me”), WG-2 (“you”) and WG-3 (“he/she/they”). The
dataset details are shown in Table 1.

We remark that, despite every video corresponding to a single sign, the signs
in this dataset are still influenced by coarticulation, which represents
changes in the articulation of the sign due to the presence of other signs
around it (e.g., the transitions between signs). This is an important
difference with datasets such as AUTSL where signs are also _performed_ in
isolation. Coarticulation makes the isolated SLR problem more challenging.
Compared to working with standard datasets such as AUTSL or WLASL, the
challenge is further compounded by the class imbalance, the low number of
training examples, and the signer independence requirement. Moreover, in our
dataset, there are different camera angles, whereas in many other SLR
datasets, the camera is positioned in front of the signer.

We process every video with three keypoint extractors. For every frame in each
video, we extract keypoints with OpenPose, MMPose, and MediaPipe Holistic. We
keep only the upper body and hand keypoints. In the case of MediaPipe
Holistic, we also remove the face mesh, which is high-dimensional and contains
a large number of redundancies. In future work, keypoint selection from the
face mesh could allow us to model facial expressions and mouthings, which are
disregarded in our current pipeline. For OpenPose, we obtain 54 2D keypoints,
53 2D keypoints for MMPose, and 67 3D keypoints for MediaPipe.

###  3.2 Post-processing keypoints

We post-process the keypoint data before using them as inputs to the SLR
model. For MediaPipe, this post-processing is applied to the 3D keypoints.
When we choose to use 2D keypoints only, we drop the depth _after_ post-
processing. Our post-processing pipeline has two stages. We first perform
missing keypoint imputation (only for MediaPipe and OpenPose) and then
normalize the keypoints.

#### Imputation

The imputation consists of linear interpolation, extrapolation, and zero-based
imputation. For MediaPipe, we do this separately for the right hand, the left
hand, and the pose keypoints (for each of these subsets, MediaPipe will either
return estimations for the entire subset, or return none at all, i.e., these
are the smallest subsets of keypoints that MediaPipe provides). For OpenPose,
we perform imputation on a per-keypoint level, because individual keypoints
can be missing.

Interpolation is applied to impute frames with missing data that are
positioned between at least two other frames with predicted keypoints. Every
missing frame is imputed by linearly interpolating between the nearest non-
missing previous and subsequent frames.

Extrapolation is applied for missing frames that occur in the beginning or at
the end of a sequence. When no previous or subsequent frame exists,
interpolation cannot be performed and we simply select the first and last non-
missing frame respectively and copy it over the missing frames. There can also
be samples where no predictions were available for a subset of the keypoints
for the entire sequence. In these cases, we impute with zeros.

Figure 2: For cases such as these (the right hand is missing in the second
frame but present in the first and third), linear interpolation can impute
missing keypoints.

Figure 2 shows an example of a case where linear interpolation can impute
missing keypoints. Despite minor visual differences, the pose estimator has
failed to predict keypoints in the middle frame. We correct this using the
above approach.

#### Normalization

For normalization, we account for differences in translation and scale of the
keypoints. We first translate the pose to be centered on the chest, and then
re-scale all keypoints by dividing them by the Euclidean distance between the
shoulders. We do the same for each hand, centering on the wrist and scaling by
the Euclidean distance between the wrist and the knuckle of the middle finger.
This normalization step has a significant impact on the performance of
keypoint based models [4].

#### Limitations

The selected pose estimation pipeline only allows imputing keypoints that were
not predicted. It is not possible to correct errors in predicted keypoints.
This is left for future research. To implement keypoint correction, confidence
values (which are provided by OpenPose and MMPose but not by MediaPipe) are of
great importance.

Furthermore, our imputation algorithm is limited. Extrapolation naively copies
frames. Linear interpolation fails to capture changes in acceleration and may
impute intermediate hand shapes and body poses incorrectly for longer missing
sub-sequences. Interpolation is performed in the keypoint space (Cartesian
coordinates). Linearly interpolating between these coordinates may result in
physically impossible poses. Context-aware imputation should be investigated
in future work.

Our normalization approach does not account for the rotation of the pose.
Despite MediaPipe providing us with 3D keypoint data, giving us the ability to
fully standardize the poses by rotating every pose to face the camera, we find
that the depth predictions are not sufficiently robust to do this in a
reliable way.

###  3.3 Model architecture

Figure 3: Generic three-stage model architecture for isolated SLR.

Our models follow a three stage architecture that is illustrated in Figure 3.
They consist of a frame embedder, a sequence embedder, and a softmax
classifier.

The frame embedder varies based on the model being utilized. For keypoint
based models, we employ a pose estimator followed by post-processing and
finally by a dense pose embedding (see below). In the case of image based
models (to which we compare our keypoint based models), we utilize ResNet [18]
pre-trained on ImageNet [13]. Both the ResNet and pose embedding are applied
to each individual frame in the sequence, resulting in a new sequence that is
equally long as the original input.

The sequence embedder is a self-attention [44] network, which outperforms
recurrent neural networks for isolated SLR [11]. This self-attention network
accommodates sequences of variable length by zero padding shorter sequences in
a batch and masking the padded frames in the attention computations. We
prepend to every sequence a learned vector, similar to the `[CLS]` token in
BERT [14]. The corresponding transformer output vector serves as the input to
the classifier.

Unlike previous research using keypoint data as inputs for SLR models [11, 28,
23, 22], we propose learning an embedding on top of the keypoints within the
architecture described above. The purpose is to learn the non-linear relations
between keypoints. Furthermore, we hypothesize that this embedding can be
transferred between languages and tasks, which is useful for pre-training
purposes or for downstream translation models. The pose embedding is
implemented as a dense network of four blocks. The first three blocks consist
of a linear layer, normalized using layer normalization, followed by a ReLU
activation, and finally dropout ($p=0.125$). The final block has a hyperbolic
tangent activation function instead of the ReLU and dropout. We apply L1
regularization to the input layer of the pose embedding ($\lambda=0.002$) to
focus on select keypoints. We finally linearly transform the keypoints into
the same dimension as the embedding, and add the resulting vector element-wise
to the embedding as a residual connection.

###  3.4 Experiments

Table 2: Accuracy values for the VTN and PTN models. Model | Pose estimator | Train | Validation | Test
---|---|---|---|---
VTN | N/A | 52.42% | 39.93% | 38.93%
PTN | MediaPipe (2D) | 76.61% | 48.15% | 45.70%
PTN | MediaPipe (3D) | 80.77% | 45.93% | 45.26%
PTN | OpenPose | 77.90% | 39.27% | 35.86%
PTN | MMPose | 62.72% | 36.49% | 36.00%

We use a Video Transformer Network (VTN)—not to be confused with Video Vision
Transformers [2]—as a baseline classifier. We use the architecture and
hyperparameters proposed by De Coster et al. [11]. As feature extractor we use
ResNet-34 pre-trained on ImageNet [13], which produces a 512 dimensional frame
embedding. The sequence of frame embeddings is processed using self-attention:
in particular 4 transformer layers with 8 self-attention heads. We classify
the output of the transformer (i.e., the `[CLS]` token embedding) using a
softmax classifier.

The Pose Transformer Network (PTN) is not directly adapted from De Coster et
al. [11]. Instead, we prepend a pose embedding network to the self-attention
network. This novel network is applied to every element of the sequence,
producing a 128-dimensional embedding per frame. Then, this embedding is
processed similarly to the way it is in the VTN. We compare MediaPipe Holistic
(with 3D and 2D keypoints), OpenPose, and MMPose by using their post-processed
predicted keypoints as input to the PTN.

From Table 2, it is clear that MediaPipe outperforms OpenPose and MMPose.
Interestingly, adding depth predictions from MediaPipe results in worse
performance. Visual analysis informs us that these are not robust333The
MediaPipe documentation states: “$z$: Should be discarded as currently the
model is not fully trained to predict depth.” See
https://google.github.io/mediapipe/solutions/holistic.html. and they may
simply introduce noise instead of additional information. Considering that the
test accuracies between MediaPipe with and without depth are more similar, it
may be that the depth predictions are worse for certain individuals in the
validation set.

##  4 Analysis of pose estimators for SLR

###  4.1 Comparison

We compare MediaPipe Holistic, OpenPose, and MMPose on several aspects: ease
of use, runtime performance, and differences in outputs. For some of these
analyses, we use the dataset introduced in Section 3.1.

MediaPipe is the easiest software package to install and the most Pythonic to
use. Due to space constraints, we discuss the installation and usage of the
three pose estimators in Supplementary Material 1.

Figure 4: Runtime performance for the three pose estimators.

We measure the runtime performance of MediaPipe, OpenPose and MMPose as
follows: we select a subset of 100 clips from our dataset. We then run all
three pose estimators on these clips and measure the execution time. We divide
the execution time by the number of frames in the video. We process a single
video at a time to simulate runtime performance at inference time. These
experiments were performed on an Intel Xeon Silver 4216 CPU (2.10GHz) and an
NVIDIA GeForce RTX 3090. In these experiments, MediaPipe runs on the CPU,
whereas OpenPose and MMPose leverage the GPU. The results are shown in Figure
4. MediaPipe Holistic processes 4.8 frames per second (FPS), OpenPose 1.1 FPS
and MMPose 1.3 FPS.

OpenPose and MMPose output confidence values per keypoint. MediaPipe Holistic
does not. Instead, if a hand could not be detected or the pose estimation has
failed, no values are returned for the entire hand. The same is true for the
body pose. In the Corpus VGT (see Section 3.1), we find that the body
keypoints are missing in 0.005% of the frames, the left hand keypoints in 11%
of the frames and the right hand keypoints in 8% of the frames. We explore the
failure cases of MediaPipe later in this section.

Figure 5: Distribution of confidence values returned by OpenPose. Figure 6:
Distribution of confidence values returned by MMPose.

We observe in Figure 5 that the hand keypoints predicted by OpenPose have
lower confidence values than the body keypoints. This aligns with previous
research that stated that OpenPose body keypoints are more robust than hand
keypoints [12]. For the hands and the body, we observe peaks at 0 confidence.
These are keypoints that are mapped to the value $(0,0,0)$ as a fallback; they
are essentially “missing values”.

Figure 6 shows that MMPose is more confident in its predictions than OpenPose
and there are no peaks at zero confidence. Because in some cases there will be
keypoints that MMPose should not be able to predict (e.g., when a hand is out
of the frame), this suggests that MMPose may be more confident in its mistakes
than OpenPose. For the purpose of SLR, where precise keypoint predictions are
important, we argue that having missing values that can be imputed or
corrected, is more useful than having the pose estimator hallucinate keypoint
coordinates.

Figure 7: Keypoints predicted by OpenPose, MediaPipe, and HRNet. For OpenPose,
the left elbow, wrist, and hand keypoints are missing.

The three pose estimators were trained on different data and have different
architectures. Hence, their predictions are different as well. Figure 7 is
illustrative of the difference between keypoints predicted by OpenPose,
MediaPipe Holistic, and MMPose. OpenPose fails to predict any keypoints for
the left arm and hand. The hand keypoints in particular show noise and
variable predictions. Especially the right hand keypoints, which are the
distinguishing keypoints for this sign, are accurately predicted only by
MediaPipe. MediaPipe Holistic has a dedicated hand pose estimation model,
whereas OpenPose and MMPose use full-body models. As a result, the hand
keypoints are typically better in MediaPipe predictions, as is also visible in
the figure. These differences in keypoint predictions also show in the
downstream performance: see Table 2. Since MediaPipe Holistic yields the best
SLR results and because of its ease of use and superior runtime performance,
we further investigate some properties of this pose estimator.

###  4.2 MediaPipe analysis

#### Video resolution

Not all sign language datasets are recorded in high resolution. For example,
the recordings in the Corpus VGT were made between 2012 and 2015 and have a
resolution of 960 by 540 pixels (540p). Moreover, sign language applications
may be designed to run on mobile devices that may have varying
characteristics.

Table 3: Proportion of missing keypoints (MediaPipe) for different resolutions. The aspect ratio (16:9) is maintained. 540p is the original resolution of the video data. Resolution | Pose | Left hand | Right hand
---|---|---|---
$426\times 240$ | 0% | 16.38% | 10.17%
$640\times 360$ | 0% | 14.97% | 10.17%
$854\times 480$ | 0% | 14.97% | 9.61%
$960\times 540$ | 0% | 12.99% | 9.32%

We analyze the impact of lowering the video resolution on the number of
keypoints that MediaPipe Holistic predicts. We find that the resolution has
only little impact on the output (we show results for a random subset of 100
videos in Table 3). This is not surprising, considering that MediaPipe resizes
and crops images to $256\times 256$ as a pre-processing step. The impact is
limited to the hands. When MediaPipe crops the hands out of the image, a
higher original resolution leads to higher quality crops and hence better
predictions.

#### Lack of temporal consistency

Figure 8: An example of a MediaPipe failure due to motion blur.

Motion blur, that occurs due to fast movements of the hands and fingers, makes
predicting accurate finger keypoints more challenging. Figure 8 shows an
example of how motion blur causes MediaPipe to fail. Our post-processing
pipeline is able to mitigate this by interpolating keypoints between non-
blurry frames. However, a pose estimator that is robust to video distortions
such as motion blur would be preferable.

MediaPipe, like OpenPose and MMPose, predicts keypoints on a per-image basis.
As a result, it lacks temporal consistency in its outputs. We illustrate this
with an example in Figure 2. The frames are perceptually similar, yet
MediaPipe fails to predict the keypoints for the right hand in the middle
image. It is known that the predictions of deep neural networks can change
drastically for minor input changes (see Figure 10 in [31]). It is possible
that this failure case also happens due to imperceptible differences in the
input image that are nevertheless picked up by the neural network. Our post-
processing pipeline can again correct for this particular failure case, but
the pose estimator itself should be robust against minor perturbations in the
input. This could be achieved by leveraging the temporal structure of the
data. For example, in Figure 2, it is highly unlikely that parts of the hand
have moved to a drastically different spatial position, or indeed disappeared,
in the milliseconds between frames.

##  5 Analysis

###  5.1 Pose post-processing

Table 4: Ablation study results (validation accuracy). Pose estimator | Norm. | Imputation | Accuracy
---|---|---|---
MediaPipe (2D) | ✓ | ✓ | 48.15%
MediaPipe (2D) | ✓ | ✗ | 47.08%
MediaPipe (2D) | ✗ | ✗ | 40.01%
OpenPose | ✓ | ✓ | 39.23%
OpenPose | ✓ | ✗ | 39.27%
OpenPose | ✗ | ✗ | 35.53%
MMPose | ✓ | N/A | 36.49%
MMPose | ✗ | N/A | 21.47%

We analyze the impact of our pose post-processing steps by performing an
ablation study, removing imputation (for MediaPipe and OpenPose) and
normalization from the pipeline. The results are shown in Table 4.
Normalization is clearly a crucial step in the pipeline. Missing data
imputation is important for MediaPipe, but less so for OpenPose. Whereas in
OpenPose, individual keypoints can be missing, in MediaPipe it is always an
entire hand or more. Therefore, the impact of missing data is larger for
MediaPipe.

Figure 9: Example of a failure case for linear interpolation. The transparent
hand is the ground truth, and the opaque hand is interpolated.

Interpolating shorter sequences works better than interpolating longer
sequences. The longer the sequence, the higher the probability that the
interpolation is incorrect. Figure 9 shows an example: an intermediate hand
pose was missed due to fast movements. At the same time, when we only need to
interpolate a short sequence, this missing data may not have as large of an
impact. This is reflected in the small difference in score when using
imputation and when not using imputation.

###  5.2 Transfer learning with pose embeddings

#### NGT to VGT

We investigate the generalization ability of the pose embedding by first
training a network on a different dataset (the NGT corpus [9]), and then
transferring the learned weights of the SLR model (except the classifier
weights) to the VGT model. We also do this for the image based VTN. This
dataset is constructed in a similar manner to the VGT dataset (Section 3.1);
it contains 68,854 samples for 458 classes in Dutch sign language (NGT). We
only investigate transfer learning with the existing architecture and do not
optimize the architecture for transfer learning.

We investigate the generalization of our frame and sequence embeddings with
three experiments. First, we freeze the entire network except the
classification layer, which we train on VGT. Second, we do the same, but after
the model has converged on VGT, we fine-tune the sequence embedding and
classifier until early stopping. In a third experiment, we fine-tune the
entire model after convergence. This illustrates the potential gains in
accuracy with transfer learning. Table 5 shows the results.

Table 5: Transfer learning validation accuracy. $\Delta$ is the difference in accuracy between the model trained with transfer learning (NGT to VGT/ASL) and the model trained only on VGT/ASL data. FT FE: Fine-tune frame embedding. FT SE: Fine-tune sequence embedding. Baselines are italicized. Model | FT FE | FT SE | NGT | VGT | $\Delta$
---|---|---|---|---|---
PTN | N/A | N/A | N/A | 48.15% | N/A
PTN | ✗ | ✗ | 44.19% | 46.19% | -1.96%
PTN | ✗ | ✓ | 44.19% | 47.93% | -0.22%
PTN | ✓ | ✓ | 44.19% | 49.30% | +1.15%
VTN | N/A | N/A | N/A | 39.93% | N/A
VTN | ✗ | ✗ | 38.81% | 39.05% | -0.88%
VTN | ✗ | ✓ | 38.81% | 39.64% | -0.29%
VTN | ✓ | ✓ | 38.81% | 46.78% | +6.85%
Model | FT FE | FT SE | NGT | ASL | $\Delta$
PTN | N/A | N/A | N/A | 53.29% | N/A
PTN | ✗ | ✗ | 44.19% | 30.40% | -22.89%
PTN | ✗ | ✓ | 44.19% | 41.24% | -12.05%
PTN | ✓ | ✓ | 44.19% | 55.05% | +1.76%

The frame and sequence embeddings can be transferred from NGT to VGT. Even
when not fine-tuning the embeddings, we achieve competitive (but slightly
lower) accuracy values. This means that transfer learning to languages without
labeled data or with very limited amounts of labeled data is possible. When
fine-tuning the sequence embedding alone, and the sequence embedding and frame
embedding together, we obtain better results. The best results are obtained
when fine-tuning the entire network after first fine-tuning the classifier
layer. We obtain a larger increase in accuracy when fine-tuning the VTN
(compared to the PTN). This is to be expected, because the ResNet has more
trainable parameters and thus can benefit significantly more from pre-training
than our pose embedding. Note that the VTN also needs to adapt to the
different visual characteristics of the VGT dataset, unlike the PTN.
Nevertheless, this score (46.78%) is still below that of a PTN trained only on
VGT data (48.15%). For low resource sign languages, transfer learning of
keypoint based models specifically will prove the most useful.

#### NGT to ASL

The NGT and VGT datasets, though they contain real-world signing, are recorded
in studio conditions. If our sign embeddings are robust, then they should
transfer also to other visual conditions. Therefore, we also investigate
transfer learning from NGT to a recently released dataset for a Kaggle
competition [16]. This dataset contains 94,477 instances for 250 American Sign
Language (ASL) signs executed in a one handed manner by individuals holding
their smartphone in their other hand to record themselves with the front
camera. Keypoints extracted with MediaPipe Holistic are provided as input
data. We use the data from four signers as the validation set, keeping the
other individuals in the training set. This paper is, to the best of our
knowledge, the first to report results on this dataset. Without transfer
learning, we obtain 53.29% validation accuracy. Our transfer learning results
are shown in Table 5.

Here, fine-tuning the pose embedding is crucial for successful transfer
learning: whereas the upstream dataset (NGT) contains two handed signing, the
downstream dataset (ASL) contains one handed signing. Nevertheless, we are
able to obtain an improvement of 1.76% validation set accuracy with transfer
learning.

##  6 Conclusion

This paper tackles low resource spontaneous sign language recognition (SLR).
SLR models can be used for various tasks, such as automatic sign language
corpus annotation, sign language information retrieval, and sign language
translation. This paper investigates the extraction of robust sign embeddings,
a crucial step for these tasks.

With little data available for training, recognizing signs in fluent signing
is a challenging problem. We leverage human pose estimation tools as feature
extractors. Comparing the three most popular pose estimators (OpenPose,
MMPose, and MediaPipe), we perform post-processing on the keypoints in the
form of normalization and missing value imputation and show the importance of
these steps. This paper furthermore introduces a pose embedding that enables
transfer learning for keypoint models across datasets and even sign languages.
By applying these steps, we outperform image based models that were previously
more powerful than keypoint based models for SLR.

Future research still needs to tackle the large class imbalance in SLR
datasets and handle larger vocabularies. Our transfer learning results across
languages suggest that training multilingual recognition models could be
beneficial. Even though we are able to drastically improve performance using
our pipeline, more research is still required into pose estimation
specifically for sign language data and how to optimally use the resulting
keypoint data.

## Acknowledgements

The images of signers in Figures 2, 3, 7, and 8 are reproduced from the Corpus
VGT project [42] (CC BY-NC-SA).

Mathieu De Coster’s research is funded by the Research Foundation Flanders
(FWO Vlaanderen): file number 77410. This work has been conducted within the
SignON project. This project has received funding from the European Union’s
Horizon 2020 research and innovation programme under grant agreement No
101017255.

## References

  * [1] Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman.  Bsl-1k: Scaling up co-articulated sign language recognition using mouthing cues.  In European conference on computer vision, pages 35–53. Springer, 2020.
  * [2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846, 2021.
  * [3] Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann.  Blazepose: On-device real-time body pose tracking.  arXiv preprint arXiv:2006.10204, 2020.
  * [4] Matyáš Boháček and Marek Hrúz.  Sign pose-based transformer for word-level sign language recognition.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 182–191, 2022.
  * [5] Michiel Bruynseraede.  Lexicale Frequentie In de Vlaamse Gebarentaal.  PhD thesis.
  * [6] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden.  Neural sign language translation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7784–7793, 2018.
  * [7] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In CVPR, 2017.
  * [8] MMPose Contributors.  OpenMMLab pose estimation toolbox and benchmark.  https://github.com/open-mmlab/mmpose, 2020.
  * [9] Onno Crasborn and Inge Zwitserlood.  The corpus ngt: an online corpus for professionals and laymen.  Construction and Exploitation of Sign Language Corpora. 3rd Workshop on the Representation and Processing of Sign Languages, 01 2008.
  * [10] Mathieu De Coster, Mieke Van Herreweghe, and Joni Dambre.  Towards automatic sign language corpus annotation using deep learning.  In 6th workshop on sign language translation and avatar technology, 2019.
  * [11] Mathieu De Coster, Mieke Van Herreweghe, and Joni Dambre.  Sign language recognition with transformer networks.  In 12th international conference on language resources and evaluation, pages 6018–6024. European Language Resources Association (ELRA), 2020.
  * [12] Mathieu De Coster, Mieke Van Herreweghe, and Joni Dambre.  Isolated sign recognition from rgb video using pose flow and self-attention.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3441–3450, 2021.
  * [13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [15] Amanda Duarte, Samuel Albanie, Xavier Giró-i Nieto, and Gül Varol.  Sign language video retrieval with free-form textual queries.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14094–14104, 2022.
  * [16] Google.  Google - Isolated Sign Language Recognition.  https://www.kaggle.com/competitions/asl-signs.  Accessed: 2023-03-07.
  * [17] Google.  MediaPipe Holistic.  https://google.github.io/mediapipe/solutions/holistic.html.  Accessed: 2023-02-28.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [19] Ruth Holmes, Ellen Rushe, Frank Fowley, and Anthony Ventresque.  Improving signer independent sign language recognition for low resource languages.  In Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology: The Junction of the Visual and the Textual: Challenges and Perspectives, pages 45–52, 2022.
  * [20] Marek Hrúz, Ivan Gruber, Jakub Kanis, Matyáš Boháček, Miroslav Hlaváč, and Zdeněk Krňoul.  One model is not enough: Ensembles for isolated sign language recognition.  Sensors, 22(13):5043, 2022.
  * [21] Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li, and Yun Fu.  Skeleton aware multi-modal sign language recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3413–3423, 2021.
  * [22] Sang-Ki Ko, Jae Gi Son, and Hyedong Jung.  Sign language recognition with recurrent neural network using human keypoint detection.  In Proceedings of the 2018 conference on research in adaptive and convergent systems, pages 326–328, 2018.
  * [23] Dimitrios Konstantinidis, Kosmas Dimitropoulos, and Petros Daras.  A deep learning approach for analyzing video and skeletal features in sign language recognition.  In 2018 IEEE international conference on imaging systems and techniques (IST), pages 1–6. IEEE, 2018.
  * [24] Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li.  Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1459–1469, 2020.
  * [25] Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little.  A simple yet effective baseline for 3d human pose estimation.  In Proceedings of the IEEE international conference on computer vision, pages 2640–2649, 2017.
  * [26] Ajkel Mino, Mirela Popa, and Alexia Briassouli.  The effect of spatial and temporal occlusion on word level sign language recognition.  In 2022 IEEE International Conference on Image Processing (ICIP), pages 2686–2690. IEEE, 2022.
  * [27] Liliane Momeni, Hannah Bull, KR Prajwal, Samuel Albanie, Gül Varol, and Andrew Zisserman.  Automatic dense annotation of large-vocabulary sign language videos.  In European Conference on Computer Vision, pages 671–690. Springer, 2022.
  * [28] Amit Moryossef, Ioannis Tsochantaridis, Joe Dinn, Necati Cihan Camgoz, Richard Bowden, Tao Jiang, Annette Rios, Mathias Muller, and Sarah Ebling.  Evaluating the immediate applicability of pose estimation for sign language recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3434–3440, 2021.
  * [29] Medet Mukushev, Arman Sabyrov, Madina Sultanova, Vadim Kimmelman, and Anara Sandygulova.  Towards semi-automatic sign language annotation tool: Slan-tool.  In sign-lang@ LREC 2022, pages 159–164. European Language Resources Association (ELRA), 2022.
  * [30] Alptekin Orbay and Lale Akarun.  Neural sign language translation by learning tokenization.  In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 222–228. IEEE, 2020.
  * [31] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami.  The limitations of deep learning in adversarial settings.  In 2016 IEEE European symposium on security and privacy (EuroS&P), pages 372–387. IEEE, 2016.
  * [32] Lionel Pigou, Sander Dieleman, Pieter-Jan Kindermans, and Benjamin Schrauwen.  Sign language recognition using convolutional neural networks.  In European conference on computer vision, pages 572–578. Springer, 2014.
  * [33] Sen Qiao, Yilin Wang, and Jian Li.  Real-time human gesture grading based on openpose.  In 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pages 1–6. IEEE, 2017.
  * [34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster R-CNN: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [35] Wendy Sandler and Diane Lillo-Martin.  Natural sign languages.  Handbook of linguistics, pages 533–562, 2001.
  * [36] Noha Sarhan and Simone Frintrop.  Transfer learning for videos: from action recognition to sign language recognition.  In 2020 IEEE International Conference on Image Processing (ICIP), pages 1811–1815. IEEE, 2020.
  * [37] Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu.  Open-domain sign language translation learned from online video.  arXiv preprint arXiv:2205.12870, 2022.
  * [38] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu.  Skeleton-based action recognition with directed graph neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7912–7921, 2019.
  * [39] Ozge Mercanoglu Sincan, Julio Junior, CS Jacques, Sergio Escalera, and Hacer Yalim Keles.  Chalearn LAP large scale signer independent isolated sign language recognition challenge: Design, results and future research.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3472–3481, 2021.
  * [40] Ozge Mercanoglu Sincan and Hacer Yalim Keles.  AUTSL: A large scale multi-modal turkish sign language dataset and baseline methods.  IEEE Access, 8:181340–181355, 2020.
  * [41] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693–5703, 2019.
  * [42] Mieke Van Herreweghe, Myriam Vermeerbergen, Eline Demey, Hannes De Durpel, Hilde Nyffels, and Sam Verstraete.  Het Corpus VGT. Een digitaal open access corpus van videos and annotaties van Vlaamse Gebarentaal, ontwikkeld aan de Universiteit Gent ism KU Leuven.  www.corpusvgt.be, 2015.
  * [43] Gul Varol, Liliane Momeni, Samuel Albanie, Triantafyllos Afouras, and Andrew Zisserman.  Read and attend: Temporal localisation in sign language videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16857–16866, 2021.
  * [44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [45] Manuel Vázquez-Enríquez, Jose L Alba-Castro, Laura Docío-Fernández, and Eduardo Rodríguez-Banga.  Isolated sign language recognition with multi-scale spatial-temporal graph convolutional networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3462–3471, 2021.
  * [46] Santosh Kumar Yadav, Amitojdeep Singh, Abhishek Gupta, and Jagdish Lal Raheja.  Real-time yoga recognition using deep learning.  Neural Computing and Applications, 31(12):9349–9361, 2019.
  * [47] Hang Yan, Beichen Hu, Gang Chen, and E Zhengyuan.  Real-time continuous human rehabilitation action recognition using openpose and fcn.  In 2020 3rd International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE), pages 239–242. IEEE, 2020.
  * [48] Mahmoud M Zaki and Samir I Shaheen.  Sign language recognition using a combination of new vision based features.  Pattern Recognition Letters, 32(4):572–577, 2011.