11institutetext: 1Warsaw University of Technology 2CERN
3Jagiellonian University 4Tooploox
5IDEAS NCBR
11email<EMAIL_ADDRESS>
# Machine Learning methods for simulating particle response in the Zero Degree
Calorimeter at the ALICE experiment, CERN

Jan Dubiński 11 Kamil Deja 11 Sandro Wenzel 22
Przemysław Rokita 11 Tomasz Trzcinski 11334455

###### Abstract

Currently, over 50% of the computing power at CERN’s GRID is used to run High
Energy Physics simulations. The recent updates at the Large Hadron Collider
(LHC) create the need for developing more efficient simulation methods. In
particular, there exist a demand for a fast simulation of the neutron Zero
Degree Calorimeter, where existing Monte Carlo-based methods impose a
significant computational burden. We propose an alternative approach to the
problem that leverages machine learning. Our solution utilises neural network
classifiers and generative models to directly simulate the response of the
calorimeter. In particular, we examine the performance of variational
autoencoders and generative adversarial networks, expanding the GAN
architecture by an additional regularisation network and a simple, yet
effective postprocessing step. Our approach increases the simulation speed by
2 orders of magnitude while maintaining the high fidelity of the simulation.

##  1 Introduction

At the European Organisation for Nuclear Research (CERN) located near Geneva,
Swizterland physicists and engineers study the fundamental properties of
matter through High Energy Physics (HEP) experiments. Inside the Large Hadron
Collider (LHC), two particle beams are being accelerated nearly to the speed
of light and brought to collide in order to recreate the extreme conditions of
the early universe just after the Big Bang.

Understanding what happens during these collisions requires complex
simulations that generate the expected response of the detectors inside the
LHC. The currently used methods are based on statistical Monte Carlo
simulations of physical interactions of particles. The high-fidelity results
they provide come at a price of high computational cost. Currently, standard
simulation procedures occupy the majority of CERN’s computing grid system
(over 500 000 CPUs in 170 centres). To address the shortcomings of this
approach, an alternative solution for simulation in high-energy physics
experiments that leverages generative machine learning techniques has been
proposed recently. [13, 4, 6]

In this work, we examine the performance of machine learning models on the
task of simulating the data from the neutron Zero Degree Calorimeter (ZDC)
from the ALICE experiment, CERN. We apply a variational autoencoder and
generative adversarial networks to the problem treating the results as
baselines. Moreover, we expand the GAN architecture with an additional
regularisation network and a simple, yet effective postprocessing step. Our
solution uses a neural network classifier to filter inputs that do not cause
any response of the calorimeter before passing the data to the generative
model.

The proposed models are able to generate the end data directly, without
simulating the effect of every physical law and interaction between particles
and the experiment’s matter separately. Therefore, this approach greatly
reduces the demand for computational power. Our approach increases the
simulation speed by 2 orders of magnitude while maintaining the high fidelity
of the simulation.

##  2 Related work

The need for simulating complex processes exists across many scientific
domains. In recent years, solutions based on generative machine learning
models have been proposed as an alternative to existing methods in cosmology
[16] and genetics [15]. However, one of the most profound applications for
generative simulations is in the field of High Energy Physics, where machine
learning models can be used as a resource-efficient alternative to classic
Monte Carlo-based[10] approaches.

Recent attempts to simulate High Energy Physics experiments [13, 11, 7]
leverage solutions based on Generative Adversarial Networks [8] or Variational
Autoencoders [12]. To the best of our knowledge [13] is the first attempt to
simulate a CERN calorimeter with generative machine learning models. The
authors combine three parallel GAN processing streams and an attention
mechanism to simulate the response of an electromagnetic calorimeter. The
authors of [3] use Wasserstein GAN to simulate the response of another
electromagnetic calorimeter. Similarly to our method, the authors embed a
regressor pretrained on predicting input particle parameters in the model.
This network extension allows them to overcome the difficulties with
conditioning on continuous values. In [2] the authors investigate the use of a
network architecture dubbed Bounded Information Bottleneck Autoencoder to
simulate an electromagnetic calorimeter. Their approach employs multiple
additional regularization networks. Additionally, this work utilizes a post-
processing network which must be trained jointly with the remaining network
components. Our method employs a similar post-processing step, however, it
does not require the training of an additional neural network.

##  3 Zero Degree Calorimeter simulation

The neutron Zero Degree CalorimeterN is a quartz-fiber spaghetti calorimeter,
which will measure the energy of the spectator neutrons in heavy ion
collisions at the CERN LHC. Its principle of operation is based on the
detection of the Cherenkov light produced by the charged particles of the
shower in silica optical fibres, embedded in a W-alloy absorber. [1]. One out
of every two fibres is sent to a photomultiplier (PMTc), while the remaining
fibers are collected in bundles and sent to four different photomultipliers
(PMT1 to PMT4) forming four independent towers. This segmentation allows to
check the relative behaviour of the different towers and to give a rough
localization of the spectator neutron’s spot on the front face of the
calorimeter. The information coming from the PMTc provides a complementary
measurement of the shower’s energy, in particular, useful for calibration
purposes. Since the number of photons collected by each tower (further
referred to as channels) is directly used in the further analysis of the
calorimeter’s output, we aim to achieve the best possible agreement, measured
between the distributions of channel values for the original and fast
simulation.

Simulating the response of the Zero Degree Calorimeter (ZDC) offers a
challenging benchmark for generative models. The dataset consists of over 8
million samples obtained from the GEANT4 [10] simulation tool. Each response
is created by a single particle described with 9 attributes (mass, energy,
charge, momenta, primary vertex).

During the simulation process, the particle is propagated through the detector
for over 100 meters while simulation tools must account for all of its
interactions with the detector’s matter. The end result of the simulation is
the energy deposited in the calorimeter’s fibres, which are arranged in a grid
with 44 × 44 size. We treat the calorimeter’s response as a 1-channel image
with 44 × 44 pixels, where pixel values are the number of photons deposited in
a given fibre. The schema of the simulation is depicted in Fig.1.

Figure 1: Fast simulation of the Zero Degree Calorimeter

To create the dataset the simulation was run multiple times for the same input
particles. For that reason, multiple possible outcomes correspond to the same
particle properties. We refer to this dataset as HEP.

Importantly, over 95% of input particles does not produce any response of the
calorimeter. For that reason, there are only 300 thousand non-zero ZDC
responses in the dataset. We randomly split the dataset into training (80%)
and validation (20% subsets.

##  4 Method

The proposed method for simulating the response of the ZDC consists of two
main parts. First, we pass the input particle parameters to a binary
classifier that assigns one of two possible class labels to the particle -
zero or non-zero. If a particle produces an empty ZDC response, the simulation
returns a 44x44 matrix of zeros. If a particle is labelled as producing a non-
empty response, then we pass its parameter to a generative model. The
generative model synthesises a calorimeter response from the input particle
parameters and a random noise vector. The schema of our method is visible on
Fig. 2.

Figure 2: Simulation pipeline.

### Zero vs non-zero response classifier

We use a binary neural network classifier to filter particles that do not
produce any response of the ZDC. The model is a densely connected 3-layer
network trained to distinguish between input parameters that produce results
belonging to one of 2 classes - zero vs non-zero response. The first two
layers of the network consist of 124 and 64 neurons respectively with ReLU
activation functions. The last layer consists of a single neuron with a
sigmoid activation function. The model was trained using binary cross-entropy
loss.

### Variatonal Autoencoder

As a baseline generative model, we apply a Variatonal Auteoncder [12] to the
problem of simulating data from the ZDC. The network consists of 2 parts: the
encoder and the decoder. During training, the encoder compresses the data into
a multivariate normal distribution of the latent variables. The decoder
attempts to decompress the data and reconstruct the input. We use a
conditional variant of the model, providing a conditional particle data vector
for both the encoder and the decor, as presented in Fig.3. For inference, only
the encoder model is used, generating data from random normal noise and a
conditional vector representing particle properties.

We use the following architecture for the encoder network:

  * •

3 convolutional layers with 32, 64 and 128 4x4 filters respectively, with
stride = 2 and a LeakyReLU activation function

  * •

a flattening layer to the output of which we concatenate corresponding
conditional particle data vector

  * •

a fully connected layer with 32 neurons and a LeakyReLU activation function

  * •

two layers with 10 neurons for encoding mean and deviation of the latent
variables

The decoder model has the following architecture:

  * •

an input layer of size 10 for latent variables to which we concatenate
corresponding conditional particle data vector

  * •

a fully connected layer with 4608 neurons, reshaped to a 6$\times$6$\times$128
shape

  * •

3 blocks each consisting of an upsampling layer, a convolutional layer
(128/64/32 4$\times$4 filters), a batch normalization layer and a LeakyReLU
activation function

  * •

an output convolutional layer with one 5$\times$5 filter and a ReLU activation
function

Figure 3:  Variational autoencoder.

### Deep Convolutional Generative Adversarial Network

We adopt the Deep Convolutional Generative Adversarial Network architecture
introduced in [14] as another baseline approach for the generative model. The
GAN architecture consists of 2 networks: the generator and the discriminator.
The generator learns to transform random noise into realistic data samples,
while the discriminator learns to distinguish real and generated data. The two
networks compete with each other during training. This process leads to a
generator that can be used to generate new realistic data samples. We employ a
conditional variant of the model, providing a conditional particle data vector
for both the generator and the discriminator, as presented in Fig.4.

The generator model has the following architecture:

  * •

an input layer of size 10 for random normal noise to which we concatenate
corresponding conditional particle data vector

  * •

2 fully connected layers with 256 and 21632 neurons, dropout=0.2 and LeakyReLU
activation, reshaped to a 13$\times$13$\times$128 shape

  * •

2 blocks each consisting of an upsampling layer, a convolutional layer (128/64
3$\times$3 filters), a batch normalization layer and a LeakyReLU activation
function

  * •

an output convolutional layer with one 3$\times$3 filter and a ReLU activation
function

For the discriminator we use the following network:

  * •

2 convolutional layers with 32, 16 4x4 filters respectively, with stride = 2
and LeakyReLU activation function

  * •

a flattening layer to the output of which we concatenate corresponding
conditional particle data vector

  * •

2 fully connected layer with 128 and 64 neurons with LeakyReLU activation
function and dropout = 0.2

  * •

an output layer with a single neuron with a sigmoid activation

### Auxiliary regressor

To improve how well the GAN model reflects the geometric properties of the
real data we expand the standard GAN architecture by an auxiliary regressor.
This additional network was pretrained to output the position coordinates of
the maximum number of photons in the input image. The regressor provides an
additional source of loss to the generator by comparing the coordinates of the
maximum of the generated examples with the maximum coordinates of the
corresponding sample in the training set. During the training of the GAN model
the loss coming from the auxiliary regressor is added to the loss of the
generator. The auxiliary regressor has the same architecture as the GAN
discriminator, apart from the last layer with has 2 neurons with a ReLU
activation function. The network was pre-trained on the same dataset used for
training the generative model. The target variables corresponding to the
coordinates of the maximum number of photons were calculated before the
training as a data preprocessing step. The model was pretrained with mean
squared error loss.

Figure 4:  DC-GAN with auxiliary regressor.

### Postprocessing

Contrary to many generative machine learning applications simulating High
Energy Physics experiments provides an objective way to directly measure the
quality of the generated samples. Motivated by this observation we introduce a
simple, yet effective postprocessing step.

To improve the alignment of the distribution of generated samples and the
original simulations we multiply the output of the generative model by a
constant c. Next, we calculate the Wasserstein distance between channels of
the original and generative simulation on the training subset as shown in Sec.
Results. We find the optimal value of c by searching the parameter space
between 0.9 to 1.1. We obtained the lowest Wasserstein distance for c == 0.96
for GAN with the auxiliary regressor and c == 1.03 for the standard GAN.

Additionally, we find a value for the standard deviation of the input noise
vector that minimizes the Wasserstein distance between the original and the
generative simulation. In our setup increasing the randomness of the input
noise vectors helps to smooth the distribution of the generated results. We
achieve the lowest Wasserstein distance for N($\mu=0,\sigma=3$).

##  5 Results

To evaluate the performance of the particle classifier we use well-established
classification metrics - precision, recall, accuracy and F1- score. For
generative models. the most common method for evaluation utilizes Frechet
Inception Distance (FID) [9]. However, to measure the quality of the
simulation we propose a domain-specific evaluation scheme, as introduced in
Sec. Zero Degree Calorimeter simulation. Following the calorimeter’s
specification [5] we base our evaluation procedure on 5 channels calculated
from the pixels of generated images that correspond to the 5 optic fibre
towers described in Sec. Zero Degree Calorimeter simulation. To measure the
quality of the simulation we compare the distribution of channels for the
original and generated data using Wasserstein distance [17]. Those channels
represent the physical properties of the simulated particle showers and are
used in the analysis of the calorimeter’s output.

Table 1: Results for the binary particle classifier. We achieve high values for all evaluation metrics despite using a relatively simple neural network architecture. | precision $\uparrow$ | recall $\uparrow$ | F1 score $\uparrow$ | accuracy $\uparrow$
---|---|---|---|---
zero | 0.96 | 0.95 | 0.96 | 0.95
non-zero | 0.93 | 0.95 | 0.94 | 0.95
Table 2: : Results comparison for the HEP datasets. The VAE model performs better than GAN networks, even after introducing an auxiliary regressor to the GAN network. Applying the postprocessing step has a major impact on lowering the Wasserstein distance and allows GAN to outperform VAE. The GAN model with the auxiliary regressor and added postprocessing step achieves the lowest Wasserstein distance between channels calculated from the original and generated data. | Wasserstein $\downarrow$
---|---
Real | -
VAE | 6.45
DC-GAN | 8.25
DC-GAN + auxreg | 7.20
DC-GAN + postproc | 5.71
DC-GAN + auxreg+ postproc | 5.16

We present the results of our experiments using both qualitative and
quantitative comparisons. In Tab. 1 we demonstrate that even using a
relatively simple neural network we can successfully filter particles causing
empty ZDC responses. In Fig. 5 we show examples of simulated calorimeter
responses for different generative models. As demonstrated in Tab. 2, our
approach outperforms other solutions on the ZDC dataset. Expanding the GAN
architecture by an auxiliary regressor and introducing the postprocessing
steps allows the GAN model to outperform VAE while producing visually sound
calorimeter responses. The positive impact of this approach on the
distribution of the generated samples is further confirmed by Fig. 6 where we
compare channel distribution for all competing approaches for 2 selected
channels. Applying the postprocessing increases the fidelity of the simulation
by smoothing the distribution of generated responses and covering the whole
range of possible outputs.

real data |
---|---
VAE |
DC-GAN |
DC-GAN \+ auxreg |
DC-GAN \+ auxreg +postproc |
Figure 5: Examples of calorimeter response simulations with different methods.
Samples generated from VAE reproduce the exact locations of the particle
showers, but they appear to be blurred. Although the results from DC-GAN are
visually sound with the original data, the model was not able to properly
capture relations from conditional values. Adding an auxiliary regressor to
the GAN architecture improves the methods in terms of reconstructing the
position of the shower’s centre.

##  6 Conclusions

In this work, we apply generative machine learning models to the task of
simulating the response of the Zero Degree Calorimeter from the ALICE
experiment, CERN. We evaluate basic VAE and DC-GAN as baselines and apply two
improvements to the GAN architecture. Simulating HEP experiments requires the
generative models to follow the strict physical properties of the modelled
process. However, this domain also offers new possibilities to improve the
performance of the generative models and provides objective evaluation
metrics. The developed generative machine learning models offer a cost-
efficient alternative to Monte-Carlo-based simulations achieving a speed-up of
2 orders of magnitude.

## Acknowledgments

This research was funded by National Science Centre, Poland grant no
2020/39/O/ST6/01478, grant no 2018/31/N /ST6/02374 and grant no
2020/39/B/ST6/01511.

| Channel 1 | Channel 2
---|---|---
VAE |  |
DC-GAN |  |
DC-GAN \+ auxreg |  |
DC-GAN \+ postproc |  |
DC-GAN \+ auxreg \+ postproc |  |
Figure 6:  Comparison of channel values distribution for selected channels.
VAE achieves good agreement between the two distributions. The GAN model is
not able to cover the whole distribution of possible channel values and has
trouble producing responses with high energy. Introducing the auxiliary does
not fully mitigate this issue. However, adding the postprocessing step
decreases the differences between the distribution of original and generated
data and smooths the distribution of the synthesised results.

## References

  * [1] Arnaldi, R., Chiavassa, E., Cicalò, C., Cortese, P., De Falco, A., Dellacasa, G., De Marco, N., Ferretti, A., Gallio, M., Gemme, R., Masoni, A., Mereu, P., Musso, A., Oppedisano, C., Piccotti, A., Poggio, F., Puddu, G., Scomparin, E., Serci, S., Siddi, E., Travaglia, G., Usai, G., Vercellin, E.: The neutron zero degree calorimeter for the alice experiment. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 564(1), 235–242 (2006). https://doi.org/https://doi.org/10.1016/j.nima.2006.03.044, https://www.sciencedirect.com/science/article/pii/S0168900206005961
  * [2] Buhmann, E., Diefenbacher, S., Eren, E., Gaede, F., Kasieczka, G., Korol, A., Krüger, K.: Getting high: High fidelity simulation of high granularity calorimeters with high speed. Computing and Software for Big Science 5(1) (may 2021). https://doi.org/10.1007/s41781-021-00056-0
  * [3] Chekalina, V., Orlova, E., Ratnikov, F., Ulyanov, D., Ustyuzhanin, A., Zakharov, E.: Generative models for fast calorimeter simulation: the lhcb case¿. EPJ Web of Conferences 214, 02034 (2019)
  * [4] Deja, K., Dubiński, J., Nowak, P., Wenzel, S., Spurek, P., Trzcinski, T.: End-to-end sinkhorn autoencoder with noise generator. IEEE Access 9, 7211–7219 (2021). https://doi.org/10.1109/ACCESS.2020.3048622
  * [5] Dellacasa, G., Zhu, X., Wahn, M., Staley, F., Danielian, V., Karavicheva, T., Mikhalev, D., Carrer, N., Gheata, M., Stefanek, G., et al.: Alice technical design report of the zero degree calorimeter (zdc). Tech. rep., ALICE (1999)
  * [6] Dubiński, J., Deja, K., Wenzel, S., Rokita, P., Trzciński, T.: Selectively increasing the diversity of gan-generated samples (2022). https://doi.org/10.48550ARXIV.2207.01561
  * [7] Erdmann, M., Glombitza, J., Quast, T.: Precise simulation of electromagnetic calorimeter showers using a wasserstein generative adversarial network. Computing and Software for Big Science 3(1) (Jan 2019)
  * [8] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)
  * [9] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in neural information processing systems. pp. 6626–6637 (2017)
  * [10] Incerti, S., Kyriakou, I., Bernal, M., Bordage, M., Francis, Z., Guatelli, S., Ivanchenko, V., Karamitros, M., Lampe, N., Lee, S.B., et al.: Geant4-dna example applications for track structure simulations in liquid water: A report from the geant4-dna project. Medical physics 45(8), e722–e739 (2018)
  * [11] Kansal, R., Duarte, J., Su, H., Orzari, B., Tomei, T., Pierini, M., Touranakou, M., Vlimant, J.R., Gunopulos, D.: Particle Cloud Generation with Message Passing Generative Adversarial Networks. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2021)
  * [12] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. CoRR abs/1312.6114 (2013)
  * [13] Paganini, M., de Oliveira, L., Nachman, B.: Calogan: Simulating 3d high energy particle showers in multi-layer electromagnetic calorimeters with generative adversarial networks. CoRR abs/1705.02355 (2017)
  * [14] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
  * [15] Rodríguez, A.C., Kacprzak, T., Lucchi, A., Amara, A., Sgier, R., Fluri, J., Hofmann, T., Réfrégier, A.: Fast cosmic web simulations with generative adversarial networks. Computational Astrophysics and Cosmology 5(1) (Nov 2018), http://dx.doi.org/10.1186/s40668-018-0026-4
  * [16] Rong, R., Jiang, S., Xu, L., Xiao, G., Xie, Y., Liu, D.J., Li, Q., Zhan, X.: MB-GAN: Microbiome Simulation via Generative Adversarial Network. GigaScience 10(2) (02 2021), https://doi.org/10.1093/gigascience/giab005
  * [17] Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558 (2017)


# PenSLR: Persian end-to-end Sign Language Recognition Using Ensembling

Amirparsa Salmankhah<EMAIL_ADDRESS>0009-0002-4195-5164 Amirkabir
University of Technology (Tehran Polytechnic)P.O.Box: 15875-4413Department of
Computer EngineeringTehranIran15875-4413 ,  Amirreza Rajabi
0009-0001-6298-6667<EMAIL_ADDRESS>Amirkabir University of Technology
(Tehran Polytechnic)P.O.Box: 15875-4413Department of Computer
EngineeringTehranIran15875-4413 ,  Negin Kheirmand<EMAIL_ADDRESS>0009-0003-7538-3819 Amirkabir University of Technology (Tehran
Polytechnic)P.O.Box: 15875-4413Department of Computer
EngineeringTehranIran15875-4413 ,  Ali Fadaeimanesh  0009-0005-3442-3292
<EMAIL_ADDRESS>Amirkabir University of Technology (Tehran
Polytechnic)P.O.Box: 15875-4413Department of Computer
EngineeringTehranIran15875-4413 ,  Amirreza Tarabkhah  0009-0002-6732-8441
<EMAIL_ADDRESS>Amirkabir University of Technology (Tehran
Polytechnic)P.O.Box: 15875-4413Department of Computer
EngineeringTehranIran15875-4413 ,  Amirreza Kazemzadeh  0009-0007-9390-2247
<EMAIL_ADDRESS>Amirkabir University of Technology (Tehran
Polytechnic)P.O.Box: 15875-4413Department of Computer
EngineeringTehranIran15875-4413 and  Hamed Farbeh  0000-0002-4204-9131
<EMAIL_ADDRESS>Amirkabir University of Technology (Tehran
Polytechnic)P.O.Box: 15875-4413Department of Computer
EngineeringTehranIran15875-4413

(2024)

###### Abstract.

Sign Language Recognition (SLR) is a fast-growing field that aims to fill the
communication gaps between the hearing-impaired and people without hearing
loss. Existing solutions for Persian Sign Language (PSL) are limited to word-
level interpretations, underscoring the need for more advanced and
comprehensive solutions. Moreover, previous work on other languages mainly
focuses on manipulating the neural network architectures or hardware
configurations instead of benefiting from the aggregated results of multiple
models. In this paper, we introduce PenSLR, a glove-based sign language system
consisting of an Inertial Measurement Unit (IMU) and five flexible sensors
powered by a deep learning framework capable of predicting variable-length
sequences. We achieve this in an end-to-end manner by leveraging the
Connectionist Temporal Classification (CTC) loss function, eliminating the
need for segmentation of input signals. To further enhance its capabilities,
we propose a novel ensembling technique by leveraging a multiple sequence
alignment algorithm known as Star Alignment. Furthermore, we introduce a new
PSL dataset, including 16 PSL signs with more than 3000 time-series samples in
total. We utilize this dataset to evaluate the performance of our system based
on four word-level and sentence-level metrics. Our evaluations show that
PenSLR achieves a remarkable word accuracy of 94.58% and 96.70% in subject-
independent and subject-dependent setups, respectively. These achievements are
attributable to our ensembling algorithm, which not only boosts the word-level
performance by 0.51% and 1.32% in the respective scenarios but also yields
significant enhancements of 1.46% and 4.00%, respectively, in sentence-level
accuracy.

Sign Language Recognition, Gesture Recognition, Ensemble Methods, Multiple
Sequence Alignment

††copyright: acmlicensed††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††journalyear: 2024††doi: XXXXXXX.XXXXXXX††journal: TOMM††isbn:
978-1-4503-XXXX-X/18/06††ccs: Human-centered computing Ubiquitous and mobile
devices††ccs: Computing methodologies Ensemble methods††ccs: Hardware Signal
processing systems

##  1\. Introduction

There are more than 72 million users worldwide who make use of sign language
on a daily basis. In Iran, the number of deaf and hard-of-hearing individuals
is estimated to be over 3 million as of 2019 (Sanjabi et al., 2016). These
people face communication problems, leading to social isolation, which can
impact their quality of life. The two main solutions for communication between
sign language users and the rest of the population are either through
handwriting or an interpreter, which can sometimes be neither feasible in
everyday scenarios nor fast or interactive. Moreover, sign languages can be
diverse, with multiple variations existing in a single region or country. This
could lead to difficulties in communication, even between sign language users.

Sign Language Recognition (SLR) has emerged as a rapidly developing field
within the research community. It focuses on addressing the problem of
recognizing sentence-level sign language glosses. SLR can help bridge
communication gaps between deaf or hard-of-hearing individuals who use sign
language and those who do not, enabling more inclusive interactions in various
settings. In understanding sign languages, Persian Sign Language (PSL), like
others, consists of a combination of intricate finger positions, hand
movements, and face mimics that collectively define its unique
characteristics. These gestures fall into two categories: manual and non-
manual markers. The former indicates the finger positions and trajectory of
the hand throughout the gesture, and the latter refers to facial expressions
or head motion. Given the critical role of manual and non-manual markers in
SLR, any effective SLR system must incorporate as many of these features into
its recognition process as possible.

A number of approaches exist to tackle the SLR problem, each with its own set
of pros and cons. The main methods are vision-based and wearable-based SLR
systems. The vision-based approach consists of analyzing visual signals
through the use of cameras or other visual sensors, and it has the advantage
of considering both manual and non-manual markers (Karami et al., 2011; Zhang
et al., 2023; Zuo and Mak, 2024; Hu et al., 2021; Mittal et al., 2019).
However, this approach also comes with its set of drawbacks, such as its
undermining of privacy, complexity of the data gathering process, and
sensitivity to lighting conditions. The second approach involves wearable-
based systems that depend on a glove or any other type of wearable device
attached to different parts of the hands or head. The embedded sensors provide
the data that will be processed and translated into sign language glosses. A
number of these methods make use of traditional machine learning approaches
(Wu et al., 2016; Khomami and Shamekhi, 2021; Zhou et al., 2020; Lu et al.,
2023; Li et al., 2020) while others leverage the power of deep learning
(Basiri et al., 2023; Wen et al., 2021; Liu et al., 2023; DelPreto et al.,
2022; Zhang et al., 2019; Sharma et al., 2023; Zhang et al., 2022; Wang et
al., 2020) to recognize and translate sign language. Although these systems
mostly come with the disadvantage of not taking into account the non-manual
markers and depend on individual anatomy, their privacy-preserving nature,
portability, and affordability make them suitable choices for SLR. Recent
research in this field has also introduced a new approach based on wireless
sensing (Jin et al., 2023, 2021; Santhalingam et al., 2020; Shang and Wu,
2017). These types of SLR systems depend on the analysis of acoustic or non-
acoustic waves and usually have the advantage of less computational complexity
and portability. However, these solutions may experience interference from
external waves and underperform in environments where obstacles hinder wave
propagation.

Researchers may face several challenges while developing an SLR system.
Firstly, the input data is dependent on both spatial and temporal features.
The former is due to the spatial nature of this task, while the latter
originates from the complexity of glosses as well as the variations in sign
language users’ speed. Secondly, It can be challenging to distinguish between
some sign language gestures due to their similarities in hand movements,
finger positions, or facial expressions. Moreover, due to variations in the
way that sign language users perform gestures, the trained model must
generalize training data to maintain accuracy when input from a new user is
fed into the model. This also must be taken into account when predicting a
previously unseen sequence from a user in the training set. Lastly, the model
must be capable of detecting the transitions between the glosses to achieve
high accuracy when facing new sentences with much higher lengths than the
training samples.

In this paper, we propose a glove-based SLR system to detect variable-length
sentences from PSL using an end-to-end deep learning framework. We leverage a
customized sign language glove with a low-cost Inertial Measurement Unit (IMU)
attached to the back of the hand and five flexible sensors mounted on fingers.
We collect two short-length and long-length datasets to prove the ability of
our framework to generalize in longer unseen sentences. Our deep learning
framework consists of two main parts. Firstly, we design a Convolutional
Recurrent Neural Network (CRNN) to extract the spatio-temporal features from
the given input sequence and predict the corresponding label. We achieve this
by utilizing the widely-used Connectionist Temporal Classification (CTC) loss
function (Graves et al., 2006), which allows us to optimize the alignments
between the input sequences and the ground truth without any prior knowledge
about the alignments or any segmentation scheme. Secondly, we propose a novel
approach for ensembling that makes use of multiple trained models and performs
a voting process to obtain the final result. Our ensembling algorithm utilizes
Star Alignment, a popular multiple sequence alignment algorithm, to align the
predicted sequences of the models, ensuring they are of equal length. It will
then take a majority vote between the aligned sequences at each position to
generate the final prediction. Our framework leads to benefits regarding
unseen sentences, specifically longer ones. Furthermore, this scheme improves
the system’s robustness in adapting to new individuals. In addition, since the
ensembling method does not rely on any particular characteristic of PSL,
future work in this domain could take advantage of it to enhance their result.
Ultimately, its applicability extends to other domains like gesture
recognition, given its independence from linguistic attributes.

In order to assess the effectiveness of our system, we gathered a dataset
comprising over 3000 samples from 16 commonly used PSL glosses. The dataset
was collected with the help of five volunteers and includes sentences of up to
three words from the selected glosses. Another dataset, containing 4 to 8-word
sentences, was gathered with the aim of evaluating the model’s ability to
recognize longer sentences. Since it is not a realistic scenario to train a
new model for each new user, we conduct a subject-independent analysis to
guarantee the practicality of our system. We compare the two best models
obtained by an ablation study in terms of four word-level and sentence-level
evaluation metrics. The results depict that our system achieves almost 94%
word-level accuracy in both datasets, showcasing its proficiency in detecting
both short and long sequences. The model is capable of predicting the length
of the sequences in approximately 95% of samples, and a sentence-level exact
match ratio of almost 88% and 80% is achieved for the short-length and long-
length datasets, respectively.

To sum up, our main contributions are summarized as follows:

  * •

We design a low-cost sign language glove using an IMU and five flexible
sensors capable of capturing a wide range of finger bendings and hand
movements.

  * •

We collect two datasets containing short-length and long-length time-series
data from PSL sentences containing 16 widely-used PSL glosses. We make our
dataset publicly available111https://github.com/Persian-Sign-Language/PenSLR-
dataset in order to make it accessible to researchers in the field of SLR,
especially the ones working on PSL.

  * •

We develop a CRNN architecture with the ability to process variable-length
signals and predict complete sign language sentences in an end-to-end fashion.

  * •

We propose a brand new ensembling scheme using Star Alignment as its backbone,
which is adaptable to other SLR or sequence-to-sequence tasks.

The rest of the paper is organized as follows. Section 2 presents the previous
work in the field of SLR and provides details about multiple sequence
alignment algorithms related to our ensembling method. Section 3 describes our
dataset as well as the data gathering process. Section 4 deeply investigates
different parts of PenSLR, including the designed glove, the architecture of
the model, and our ensembling scheme. In Section 5, the performance of the
system is discussed using multiple experiments and evaluation metrics. Then,
in Section 6 we investigate the limitations of our system and explore possible
avenues for future research. Finally, the conclusion of the paper is drawn in
Section 7.

##  2\. Related Work

In this section, we explore the related works in the field of sign language
recognition and provide a brief explanation of multiple sequence alignment
algorithms.

###  2.1. Sign Language Recognition

Two leading solutions exist to tackle the problem of sign language recognition
(SLR) based on sensing technologies: vision-based and mobile/wearable
solutions. Also, a recently emerging approach is based on wireless sensing.

####  2.1.1. Vision-based Methods

Vision-based methods usually utilize camera setup and visual signals, thus
have the advantage of considering non-manual markers. An early work (Karami et
al., 2011) in this area for PSL consists of the use of Discrete Wavelet
Transform (DWT) for feature extraction and a Multi-layer Perceptron (MLP)
neural network for categorizing sign language gestures. In another study, C2ST
(Zhang et al., 2023) takes into account the linguistic features of gloss
sequences using a linguistic model. Later research in the vision field ensures
spatial attention consistency using a keypoint-guided spatial attention module
(Zuo and Mak, 2024). Another work in this field also takes into account non-
manual markers using a custom-developed Global-local enhancement network (GLE-
Net) architecture (Hu et al., 2021). Moreover, (Mittal et al., 2019) suggests
using a Leap Motion sensor to extract the coordinates of different parts of
the hands. It also propose a modified version of Long Short Term Memory
(LSTM), introducing a new reset gate that resets the memory of LSTM whenever a
non-active situation is detected.

####  2.1.2. Wireless sensing-based Methods

The following methods introduce wireless sensing-based SLR systems that use
electromagnetic or acoustic waves to detect body movements. In (Shang and Wu,
2017), a Wi-Fi receiver and two Wi-fi transmitters are used to collect data,
and then a Kernel-based Support Vector Machine (SVM) model is used to classify
gestures on different systems. Another work, mmASL (Santhalingam et al.,
2020), takes advantage of millimeter waves and a multi-task deep learning
model to recognize American Sign Language (ASL) gestures. Additionally,
SonicASL (Jin et al., 2021), a real-time gesture recognition system,
identifies sign language through the use of earphones with built-in
microphones and speakers, achieving high accuracy rates in both word and
sentence recognition. In a later work, SmartASL (Jin et al., 2023) not only
utilizes earbud signals but also employs IMU sensors to detect both manual and
non-manual markers. The last two mentioned works employ CRNN architecture
along with CTC loss function to perform sign language recognition in real-
time.

####  2.1.3. Wearable-based Methods

Mobile/Wearable systems designed to recognize hand or body gestures are
another approach to the SLR task. Although these solutions do not consider
non-manual markers, they can help solve the range of failure points that the
previously mentioned vision-based and wireless sensing-based solutions have.

An early work in this field used customized bending sensors, accelerometers,
and Hall effect sensors in order to detect gestures of digits (0 to 9) through
logistic regression (Chouhan et al., 2014). Furthermore, authors in
(Muralidharan et al., 2022) suggest a 5-bit representation of the words or
sentences and assign a bit to the output of each flexible sensor on fingers to
predict the gestures based on the perceived values. Another study involves
using a combination of IMU and sEMG sensors to gather data (Wu et al., 2016).
The sEMG sensors make it possible for them to automatically split the input
signals into segments that are fed to an SVM classifier for the final
prediction. Moreover, a study on PSL suggests using a similar sensor setting
along with a KNN classifier and K-fold cross-validation to detect 20 commonly
used isolated sign gestures (Khomami and Shamekhi, 2021). However, it did not
provide a solution for recognizing PSL sentences. In (Zhou et al., 2020), a
yarn-based stretchable sensor was introduced, which is not only capable of
capturing hand gestures but can also be attached to eyebrows and mouth. Then,
an SVM model was used to perform the classification task. Authors in (Lu et
al., 2023) took a unique approach by storing predefined sentences in a
database while harnessing a customized glove with an IMU and flexible sensors.
During the testing phase, a novel DTW distance was proposed to find the
nearest data in the database and predict the label of the new input signal. In
(Li et al., 2020), the authors suggested an approach to calculate the movement
trajectories of gestures and took advantage of another customized DTW distance
to classify the signals.

Later works leverage the power of deep learning to achieve better results and
accomplish more complex tasks. A work on PSL (Basiri et al., 2023) collected a
time-series dataset of 15 words with 600 samples in total. The collected data
were virtually augmented to 30000 images using the State-Image approach, and a
CNN network was proposed to train on these images. Similar to (Khomami and
Shamekhi, 2021), this study lacked the ability to predict continuous
sentences. Another study suggests using a sliding-window approach to segment
the input signals and construct the predicted sentence by feeding the segments
to a CNN-based neural network (Wen et al., 2021). This enabled the model to
achieve high results in detecting 50 words and 20 predefined sentences, as
well as new sentences that could be constructed by different combinations of
those words. Authors in (Liu et al., 2023) utilized a similar CNN-based
network with a customized glove containing an IMU and strain sensors to detect
48 Chinese Sign Language (CSL) gestures. They took advantage of multiple
sliding windows with different lengths and aggregated their results for
sentence-level prediction. In another study, a glove was designed using
conductive knit fabric and an accelerometer (DelPreto et al., 2022). To
generate the output, an LSTM network was proposed to train on a dataset
containing 12 distinct classes from ASL using a sliding-window approach.

The most recent approaches involve using CRNN models along with CTC loss
function or an extension of them involving attention-based networks. Myosign
(Zhang et al., 2019) suggests an armband capable of collecting data from
different modalities, i.e., accelerometer, gyroscope, orientation, and EMG.
They propose a CRNN network consisting of multimodal CNN and Bidirectional
LSTM (BiLSTM) layers to build an end-to-end system for predicting ASL
sentences. They achieve this through the use of CTC loss, which enables them
to perform SLR without any prior knowledge of the alignments between input and
output. In (Sharma et al., 2023), a transfer learning scheme was employed on a
similar architecture as Myosign, enabling the system to converge faster while
maintaining accuracy. Their wearable system consisted of multiple IMU sensors
on both forearms. Building on Myosign, Wearsign (Zhang et al., 2022)
introduces an encoder-decoder framework that takes advantage of the attention
mechanism to translate the sign language glosses to the spoken text. Moreover,
they utilized the back-translation technique to augment and extend their ASL
dataset. Similarly, Hear Sign (Wang et al., 2020) proposes an attention-based
encoder-decoder approach with a multi-channel CNN to train on the data
collected by two MYO armbands effectively.

###  2.2. Multiple Sequence Alignment

Due to the increase of data in today’s era, especially sequence data, various
algorithms have been designed to align multiple sequences together. The
application of these algorithms is mainly in the field of bioinformatics. For
example, understanding the functional significance of genetic variations
across diverse species counts as one of their primary applications. The task
of aligning two sequences, pairwise sequence alignment, serves as the
foundation for aligning multiple sequences. Pairwise alignment is primarily
categorized into local alignment and global alignment. Local alignment focuses
on identifying and aligning similar local regions, while global alignment
involves aligning sequences in an end-to-end manner (Chao et al., 2022). In
this section, our primary focus relies on global alignment because it is the
approach we use in our proposed algorithm.

The Needleman-Wunsch (NW) algorithm (Needleman and Wunsch, 1970) is a widely
employed global alignment algorithm that leverages a dynamic programming
approach to calculate pairwise sequence alignment, taking advantage of the
optimal substructure of this problem. However, using this algorithm for
multiple sequence alignment is not feasible due to its time complexity. In
order to tackle this problem, two main strategies are suggested: star
alignment and progressive alignment. Star alignment works by automatically
selecting one of the sequences as the center and aggregating the results of
pairwise alignment between the center and the other sequences to obtain the
final alignment. This approach considerably reduces the time needed to compute
the final alignments, thus making it a suitable choice for real-time tasks. On
the other hand, progressive alignment algorithms, such as ClustalW (Thompson
et al., 1994), are based on a guide tree that determines the order by which
the sequences should get aligned. There are two main drawbacks to the
progressive alignment algorithms. First, the final result is highly dependent
on the quality of the guide tree. Second, they need more time to execute as
hierarchical clustering is needed to construct the guide tree, and internal
nodes of the tree require more complex operations for computing the alignments
(Chao et al., 2022).

##  3\. Dataset

We collected our own dataset for two reasons. First, as mentioned in Section
2.1.3, there was only one public PSL dataset (Basiri et al., 2023) available,
which was neither large enough nor contained sentence-level data. Second, none
of the previous work on PSL used our hardware settings, forcing us to collect
new data to ensure the compatibility of our glove-based design with our
dataset. Unlike many conducted studies in the area of SLR, which used
separated word-level and sentence-level datasets to train the model, we
collected a unified dataset containing word combinations (sentences) of up to
three words. Sentences are random permutations of the selected PSL signs and
do not necessarily convey meaning. Moreover, the reason for limiting the
number of words in sentences was to show the capability of our Seq2Seq model
to predict sentences containing more than three words without being trained on
them.

To demonstrate the effectiveness of our model, we cherry-picked 16 words from
PSL, categorizing them into five similarity groups. Table 1 demonstrates the
groups and the relationship between the similar words. The words within the
same tuple in groups 1 and 2 mutually share the characteristics of the group.
That is, they share similarities in at least one aspect, either through hand
movements or finger positions, making them indistinguishable without the
concurrent use of both sensors. On the other hand, groups 3 to 5 feature the
words that have fixed, dynamic, or rotational hand movements, respectively. As
a result, achieving high accuracy in this setup could showcase the model’s
proficiency in detecting and distinguishing minor variations between similar
gestures. For clearer understanding, Figure 1 depicts the execution of two
pairs of our PSL words distinguished by different colors. The red group shows
a pair of gestures (”Blue” and ”Year”) with rotational hand movements on
different axes. The blue group illustrates gestures (”is” and ”Very”), which
share finger positions but have minor differences in hand movement.

Figure 1. Illustration of step-by-step execution of two pairs of PSL glosses
(”Blue”, ”Year”) and (”is”, ”Very”) belonging to two distinct similarity
groups. The blue glosses have similar finger positions but different hand
movements, while the red ones are examples of rotational gestures.

ID |  Group Characteristic |  Members
---|---|---
1 |  Same finger position but different hand movement |  (Agreement, Disagreement) - (Yesterday, Father, Luck, Year) - (is, Very, Hopeful)
2 |  Same hand movement but different finger position |  (Luck, Summer)
3 |  Fixed hand movement and finger position |  Good - Agreement - Disgreement
4 |  Dynamic hand movement and finger position |  Day - Forget - Mother
5 |  Rotational movements |  Blue - Green - Year

Table 1. Our selected PSL words categorized into five groups

We asked five volunteers to perform sign language gestures. Although recording
hand-picked sentences multiple times by different volunteers is a common way
to build a sign language dataset, we randomly generated the sentences in our
dataset, ensuring no human bias is involved in the selection process. For our
primary dataset (Dataset1-3), each volunteer recorded, on average, 100 one-
word sentences, 300 two-word sentences, and 200 three-word sentences.
Moreover, we asked each volunteer to record 20 extra sentences containing 4 to
8 words to evaluate the ability of our model in longer unseen sentences
(Dataset4-8). Additionally, a GUI application was designed and implemented to
increase the speed and ease of the recording process.

Each data in the dataset is a sequence of values received from the sensors at
a rate of 100 Hz. IMU features include total acceleration, linear
acceleration, gyroscope, and gravity acceleration in the X, Y, and Z axes.
Combining these features with five features returned from flexible sensors
forms 17 distinct features per data point. Therefore, every sample in the
dataset is a time series data that can have different lengths depending on the
execution time of each gesture.

Table 2 depicts the average number of data points derived from sequences
containing one to three words across different subjects. According to the
table, firstly, the average amount of time to execute sentences with the same
length is different for different subjects. Secondly, subjects show different
behaviors for different sentence lengths and do not have a constant speed
while performing gestures. For example, on average, subject 4 has performed
single-word sentences in the least amount of time but exhibits the slowest
pace for 3-word sentences. These variations pose a challenge for the model in
detecting word transitions within sentences.

|  Subject 1 |  Subject 2 |  Subject 3 |  Subject 4 |  Subject 5 |  Average
---|---|---|---|---|---|---
Length 1 |  13.0 |  11.7 |  11.6 |  9.7 |  10.7 |  11.34
Length 2 |  19.0 |  19.1 |  18.2 |  19.8 |  20.2 |  19.26
Length 3 |  25.4 |  28.0 |  27.3 |  30.2 |  26.5 |  27.48

Table 2. The average number of sample lengths in Dataset1-3 grouped by
different subjects. The slowest and the fastest subjects in each row are
denoted by green and red, respectively.

The dataset is intended to facilitate the research toward PSL or any other
sign language variation. To ensure the reproducibility and accessibility of
our work, we have made the dataset publicly
available222https://github.com/Persian-Sign-Language/PenSLR-dataset. We
encourage other researchers, especially the ones working on PSL, to use our
dataset to become familiar with the challenges that may arise while working
with sign language datasets or to propose new architectures that could achieve
better results than our model.

##  4\. Proposed Method

In this section, we describe different parts of PenSLR. This includes the
design of the sign language glove, the data collection process, preprocessing
techniques, the deep learning framework, and the proposed ensembling method.

###  4.1. Glove Design

Figure 2. Our sign language glove equipped with an Adafruit BNO055 IMU mounted
on the back of the hand and five flexible sensors on each finger.

As depicted in Figure 2, our designed glove incorporates two types of sensors:
an IMU sensor mounted on the back of the hand and five flexible sensors
attached to each finger. We utilized Arduino Mega to collect the sensors’
data. The IMU sensor sends its data to the microcontroller via I2C protocol,
whereas the flexible sensors achieve this by analog input pins.

One of the main goals of our research was to design a low-cost but accurate
glove. As a result, we used the affordable Adafruit BNO055 IMU, which can
deliver a variety of metrics, including acceleration, orientation, and
gravity, along three distinct axes. Capturing these features enables us to
distinguish between different hand movements based on orientation patterns.
flexible sensors, on the other hand, are sensitive to bending; thus, their
values change with the closing and opening of fingers. Considering the setup
above, our glove is capable of covering a wide range of hand gestures while
being cost-effective and easily repairable.

###  4.2. Seq2Seq Model

This section describes the specifics of our model, such as preprocessing
steps, the model architecture and the loss function we use in our model.

####  4.2.1. Preprocessing

Since errors (caused by human or hardware factors) may affect the data
collection process, it is necessary to identify outliers and remove them from
the dataset before using them. To achieve this, we used the Interquartile
Range (IQR) method. In this method, the first quartile $Q_{1,f}$ and the third
quartile $Q_{3,f}$ of data are calculated for each feature and the difference
between them $IQR_{f}$ is used to determine a range for non-outlier data. To
be precise, when a data point falls outside the range of $[Q_{1,f}-1.5\cdot
IQR_{f},Q_{3,f}+1.5\cdot IQR_{f}]$ for at least one feature, the whole sample
is considered as an outlier and will be removed from the dataset.

Data normalization was another step in our preprocessing pipeline. To
normalize the data, we calculated the minimum value $x_{min,f}$ and the
maximum value $x_{max,f}$ of each feature based on the training data. Then, we
normalized the value of each feature $x_{f}$ in training and validation data
as follows:

(1) $x_{f}=\frac{x_{f}-x_{min,f}}{x_{max,f}-x_{min,f}}$

During the testing phase, the test data were normalized with respect to the
values we calculated beforehand using the training dataset.

Figure 3. Architecture of our Seq2Seq model

####  4.2.2. Model Architecture

As illustrated in Figure 3, we employed three main components in our neural
network model: a feature fusion (FF) module to combine input features, a
temporal feature extractor (TFE) to detect temporal patterns of each sign
language gesture, and an MLP classifier to output the predicted gesture using
the features obtained from the previous components.

As we had samples of varying lengths in our dataset, we added zero padding to
the samples in the same batch. More formally, we formed a $B\times N_{f}\times
T_{inp}$ matrix for each batch, where $B$ is the batch size, $N_{f}$ is the
number of features, and $T_{inp}$ is the length of the longest sequence in the
batch. By doing this, we were able to leverage the efficiency of batch
gradient descent, thereby accelerating the process.

The FF component aims to combine the features with each other. It is essential
to extract the dependencies between the values of different features before
feeding them to the TFE module, which extracts temporal dependencies between
them. Moreover, FF extracts local temporal dependencies, which guides TFE in
finding more complex patterns. In other words, the raw values of different
features in FF are combined with the help of a convolution layer to create
higher-level features. For this purpose, we used a 2D convolutional layer with
a kernel size of $3\times 3$ to extract local time-related dependencies while
combining different features.

The features generated by the FF component enter the TFE module, which
includes two consecutive Bidirectional LSTM (BiLSTM) layers with hidden state
sizes $n_{1}=64$ and $n_{2}=128$, respectively. Each hidden state unit $h_{t}$
in BiLSTM contains information from the past and future of the input sequence
at time $t$; thus, it helps us distinguish sign language gestures with the
same beginning but different endings. Ultimately, each hidden state unit in
the last BiLSTM layer yields an output in the form of a matrix with size
$B\times n_{2}$, representing the spatio-temporal features distilled from the
sequences of each batch.

The final step in our model is when the generated spatio-temporal feature
matrices are fed into an MLP classifier to produce the predicted sequence. We
used a shared 3-layered perceptron network containing 64, 32, and
$N_{classes}+1$ perceptrons along with the CTC loss function to predict the
output label at any time step. Since the CTC loss function requires an extra
blank label, we added an extra unit to the last layer, resulting in a total of
17 distinct classes. We further discuss the details of the CTC loss function
in Section 4.2.3.

To improve the performance of our neural network, we used batch normalization
and dropout techniques. Batch normalization contributes significantly to the
faster convergence of the network by preventing the covariant shift.
Therefore, we used it after the convolution layer of the FF module and between
each two layers of the MLP classifier. Later, in Section 5.2, we will examine
the effect of adding batch normalization between different layers of our
network. Finally, we placed a dropout layer with a probability of 0.3 after
each of the BiLSTM layers. This prevents the model from overfitting and allows
it to learn generalizable patterns while training.

####  4.2.3. CTC Loss

The Connectionist Temporal Classification (CTC) loss function is one of the
most popular loss functions in sequence-to-sequence tasks, such as optical
character recognition and speech recognition. The main advantage of this
function is its capability to align the input sequences to the target
sequence, especially in scenarios where this alignment is not one-to-one. In
other words, this function is used when the length of input and target
sequences do not match, and the exact location of each output event in the
input sequence is not specified. Due to the inherent variability in the
execution of sign language gestures, using the CTC loss function in SLR can be
an effective solution to improve the performance of deep learning models in
this field. That is because people execute words at different speeds, and
there may be long pauses between movements, which can pose a challenge in
accurately recognizing sign language gestures.

CTC introduces a blank symbol and allows the neural network to output repeated
occurrences between the actual symbols, thus enabling variable-length output
sequences. For example, both (AA-B-C-CC) and (A-B–CC-C) represent the sequence
ABCC. During training, the CTC algorithm aligns these output sequences with
the target sequences, considering all possible alignments using a dynamic
programming approach. Finally, it computes a loss function based on the
negative log probability of these alignments, guiding the network to produce
output sequences that are likely to match the target sequences.

###  4.3. Sequence Alignment

In this section, we propose a novel method based on ensembling to improve the
performance of SLR Seq2Seq models. One way to improve the models’ performance
in deep learning is to use ensembling. In ensemble methods, several models are
trained on the data, and their outputs are used to produce the final
prediction. One way to build each of these models is to split the data into
$k$ folds and train $k$ different models where each model is trained on $k-1$
folds and validated using the remaining fold (K-fold cross-validation). In the
SLR task, however, given a single sample, the outputs of the models trained
using K-fold cross-validation could have variable lengths, making it
challenging to combine them to generate the final result. To overcome this
problem, we suggest using multiple sequence alignment (MSA) algorithms to make
the output of the models equal in length and then perform a voting process to
produce the final prediction.

Algorithm 1 Pairwise Global Alignment

1:Input:  Sequences $s_{1}[0,\ldots,m-1]$ and $s_{2}[0,\ldots,n-1]$

2:Parameters:  $S_{mis}$, $S_{match}$, $S_{gap}$

3:Output:  2 Aligned Sequences, The Alignment Score

4:Initialize $dp[m+1,n+1]$ with zeros

5:Initialize $dp[i,0]=i\times S_{gap}$

6:for $i$ from $1$ to $m$ do

7: for $j$ from $1$ to $n$ do

8: if $s_{1}[i]=s_{2}[j]$ then

9:
$dp[i,j]=\max\\{dp[i-1,j-1]+S_{match},dp[i-1,j]+S_{gap},dp[i,j-1]+S_{gap}\\}$

10: else

11:
$dp[i,j]=\max\\{dp[i-1,j-1]+S_{mis},dp[i-1,j]+S_{gap},dp[i,j-1]+S_{gap}\\}$

12: end if

13: end for

14:end for

15:$Aligned_{s1}$, $Aligned_{s2}$ = Backtrack($dp$)

16:return $Aligned_{s1}$, $Aligned_{s2}$, $dp[m,n]$

Needleman-Wunsch (NW) is an algorithm that uses dynamic programming to compute
optimal global alignment (GA) between two strings. As shown in Algorithm 1, NW
creates a table of size $m+1$ by $n+1$ to align two strings of length $m$ and
$n$. Then, the first row and the first column are initialized as follows:

(2) $D_{0,i}=D_{i,0}=i*S_{gap}$

where $S_{gap}$ is the penalty for creating a gap in one of the sequences.
Finally, it fills the table using Eq. 3 and Eq. 4:

(3) $D_{ij}=\max\begin{cases}D_{i-1,j-1}+S(X_{i},Y_{j})\\\
D_{i-1,j}+S_{gap}\\\ D_{i,j-1}+S_{gap}\end{cases}$

(4) $S(c_{1},c_{2})=\begin{cases}S_{match}&\text{if }c_{1}=c_{2}\\\
S_{mis}&\text{o.w}\end{cases}$

where $X$ and $Y$ are the input sequences, $S_{match}$ is the score of
matching two characters in the sequences, and $S_{mis}$ is the penalty of
mismatch between two characters. Then, the final alignment score and the
aligned sequences can be obtained by backtracking from the final position
$[m,n]$ to the starting position $[0,0]$. Although this method can be extended
to align $k$ sequences, its exponential computational complexity prevents us
from using it in a real-time system. Therefore, a method with less
computational complexity is preferable to align the sequences.

Star Alignment (SA) is a heuristic method for approximating GA, primarily
applied in bioinformatics for aligning DNA sequences. Algorithm 2 shows how SA
aligns multiple sequences. First, it calls NW algorithm on each pair of input
sequences to compute their alignments and the corresponding similarity scores.
Then, the sequence exhibiting the maximum cumulative similarity is designated
as the central sequence. Subsequently, other sequences are arranged in
descending order based on their similarity to the central sequence. Finally,
traversing the sequences in order, some gaps are added to the central string
and all previously traversed strings (if necessary) to equalize their lengths,
using the alignments calculated in the first step. For instance, Figure 4
illustrates an step-by-step progress of running SA on five arbitrary
sequences.

Algorithm 2 Star Alignment

1:Input: $k$ sequences $s_{0}$ to $s_{k-1}$

2:Output: $k$ Aligned Sequences, The Alignment Score

3:Calculate pairwise alignments and scores between all pairs $(s_{i},s_{j})$
where $i\neq j$

4:Choose the sequence $s_{c}$ as the center, where
$c=\arg\max_{j}\left(\sum_{i=0}^{k-1}\text{Score}(s_{j},s_{i})\right)$

5:Sort indices $i\neq c$ in decreasing order with respect to
$\text{Score}(s_{c},s_{i})$

6:for each index $i$ in the sorted indices do

7: Utilize the alignments between $s_{c}$ and $s_{i}$ to update the gap
locations of the aligned sequences $s^{\prime}_{c}$ and $s^{\prime}_{j}$ where
$j\leq i$

8:end for

9:return The Aligned Sequences $s^{\prime}_{i}$, Total Alignment Score

Figure 4. An illustration of how Star Alignment algorithm computes the
similarity matrix between 5 sequences ($S_{1}="ABCFB"$, $S_{2}="ABCBBC"$,
$S_{3}="ACFBC"$, $S_{4}="BCFCC"$, $S_{5}="ABEFBBC"$) and use it to
progressively align them. Sequence $S_{2}$ is designated as the center since
the sum of similarities in the second column is the highest. ($S_{match}=3$,
$S_{gap}=-2$, and $S_{mis}=-1$)

We utilize SA along with K-fold cross-validation to enhance the accuracy of
our system. Algorithm 3 provides a detailed overview of our method. In the
training phase, we use 5-fold cross-validation to train five distinct models
on Dataset1-3, and when testing, we use those models to predict five sequences
$s_{0}$ to $s_{4}$ for each input sample. Next, we use the predicted sequences
as the inputs of the SA algorithm to obtain aligned sequences $s^{\prime}_{0}$
to $s^{\prime}_{4}$. The final step includes voting at each time step $t$ of
the final alignments. In other words, the most prevalent character at $t$ is
added to the final result unless the character is a gap. It should be noted
that before using SA, it is necessary to map the sign language words to
arbitrary characters. Therefore, we restore the words using inverse mapping
after the final voting. Figure 5 depicts the execution of the explained
process.

Algorithm 3 Our Ensembling Algorithm

1:Input: Sample $x$, Mapping T from the chosen sign language words to
arbitrary characters

2:Output: The Predicted Sequence

3:Train 5 models via 5-fold cross-validation

4:Predict the outputs ($s_{0}$ to $s_{4}$) for the input $x$

5:Replace each character $ch$ in $s_{i}$ with $T[ch]$

6:Pass sequences $s_{i}$ to Star Aignment to get sequences $s^{\prime}_{i}$

7:for $t$ from 0 to $s^{\prime}_{0}.length-1$ do

8: $out=out+\text{Vote}(s_{0}[t],...,s_{4}[t])$

9:end for

10:Remove gaps from $out$

11:Use the inverse mapping of $T$ to update $output$

12:return $out$

Figure 5. An example of the execution of our ensembling algorithm. The outputs
produced by all models are aligned using Star Alignment, after which a voting
process is performed to obtain the most probable gesture (denoted by
characters A to F) at each position. Finally, the gaps are ignored, and the
final result is generated. This example is a situation when the best model
(Model 1) can not completely predict the ground truth sequence, but the
ensembling method helps the system to generate it successfully.

The intuition behind our model relies on the fact that the models trained on
distinct folds tend to learn different patterns, especially when the dataset
is not very large. That is, their accuracy in detecting different classes may
vary, and each could manage to predict segments of the ground truth sequence.
As a result, by aligning their predictions and voting between them, the final
result is expected to have a lower error rate than the prediction of the best
one. As will be explained in Section 5.3, this method does not always improve
the accuracy. This situation occurs when the best model has much higher
accuracy than others or when all models do not have enough accuracy, so the
performance drops by voting between the predictions. However, the results
depict that, on average, the method could significantly improve the
performance of the system. Furthermore, since our ensembling technique does
not depend on any linguistic feature, it can be adapted for studies on other
variations of sign language or language-independent sequence-to-sequence tasks
such as gesture recognition.

##  5\. Experiment Setup and Results

In this section, we examine the performance of the proposed model and our
ensembling algorithm. We use a subject-independent approach in all experiments
unless where it is mentioned explicitly. By using leave-one-subject-out cross-
validation, we train each model on the data of 4 subjects and test it with the
data of the remaining subject. In the training phase, we use Dataset1-3,
mentioned in Section 3, which includes sentences with up to three words. In
addition, we use the 5-fold cross-validation approach to apply our ensembling
algorithm, where each fold contains 20% of the data. Also, we use the AdamW
optimizer with a batch size of 9 and learning rate scheduling in 3 steps to
optimize the cost function as much as possible. On the other hand, when
testing, our experiment settings involve both Dataset1-3 and Dataset4-8,
enabling us to evaluate user-independent generalization as well as
generalization in longer sentences, respectively. Furthermore, in the testing
phase, we assigns scores to gap penalties, mismatches, and matches as follows:
$S_{gap}=-1$, $S_{mis}=-1$, and $S_{match}=0$. This configuration was
determined to be optimal for our recognition task through rigorous
experimentation, resulting in the best performance.

###  5.1. Evaluation Metrics

We use the following four metrics to analyze the performance of our models:
Sequence Length Accuracy (SLAcc): It depicts the percentage of sequences whose
lengths are accurately predicted by the model relative to the total number of
sequences.
Sequence Accuracy (SAcc): It indicates the percentage of sequences that
exactly match their ground truth out of the total number of sequences. Since
achieving high results in this metric is quite challenging, mainly when the
dataset contains lengthy sequences, many previously conducted studies have not
reported their results based on this metric. Despite this, we report our
results to show the effectiveness of our ensembling approach in correcting the
mispredicted sequences.
Word Accuracy (WAcc): This metric is defined based on another metric called
Word Error Rate (WER), which is used in tasks where the length of the
predicted sentences could be different from their actual length. To calculate
WER for a given pair of ground truth and predicted sequences, we use the
following equation:

(5) $WER=\frac{S+D+I}{L}\times 100$

where $L$ is the length of the ground truth sequence, $S$ is the number of
substitutions, $I$ is the number of insertions, and $D$ is the number of
deletions needed to convert the ground truth sequence to the predicted
sequence. In fact, the numerator is the widely used Levenshtein distance,
which calculates the minimum number of single-character edits needed to
transform one sequence into another. In our task, each individual sign
language gesture in predicted sequences corresponds to a character in the
Levenshtein distance calculation. Then, we compute word accuracy to show the
similarity of the predicted sequence to the corresponding ground truth:

(6) $\text{WAcc}=100-\text{WER}$

Finally, we calculate total word accuracy as follows:

(7) $\text{Total}\ \text{WAcc}=\frac{\sum\text{WAcc}_{i}}{N}$

where $WAcc_{i}$ is the word accuracy for the $i$-th sample in the dataset,
and $N$ is the total number of samples.
Weighted Word Accuracy (WWAcc): Instead of computing a simple average over the
obtained word accuracies, we can take a weighted average to amplify the impact
of lengthy sentences in the final result. Thus, we can calculate weighted word
accuracy using the following equation:

(8) $\text{WWAcc}=\frac{\sum\text{WAcc}_{i}.L_{i}}{\sum L_{i}}$

Contrasting these results with the previous metric can showcase the
extendability of our approach to longer sentences.

###  5.2. Ablation Study

As mentioned in Section 4.2.2, our model includes a $3\times 3$ convolutional
layer, two BiLSTM layers, and an MLP classifier with batch normalization
between any two layers. To show the impact of different components in the
proposed architecture, we remove or change different parts of the model. Table
3 shows the results of this work without applying our ensembling method. In
the last column of this table, the accuracies obtained from the performance of
the proposed model are shown. Other columns depict the amount of change in the
model’s performance after removing or changing one of the model’s components.
It should be noted that the numbers shown are the average test results of the
model across different subjects while having trained on a specific fold of
Dataset1-3 without applying our ensembling method.

| CNN | LSTM | Bidirectional | Batch Norm | Our Model
---|---|---|---|---|---
| No | $5\times 5$ | 1 | No | 0 | 1 |
SLAcc | -4.92 | -3.05 | -4.22 | -12.33 | -1.50 | -1.86 | 91.47
SAcc | -15.12 | -2.51 | -1.55 | -12.09 | -4.41 | -0.23 | 81.69
WAcc | -8.91 | -2.07 | -1.24 | -6.97 | -2.71 | -0.39 | 90.39
WWAcc | -8.71 | -1.38 | -0.70 | -6.01 | -2.56 | -0.11 | 90.85
Average | -9.41 | -2.25 | -1.93 | -9.35 | -2.79 | -0.65 | 88.60

Table 3. The percentage of performance degradation of models resulting from
the deletion or modification of each component within our proposed
architecture.

The model achieves a WAcc of 90.39% and a WWAcc of 90.85%, resulting in an
exact match ratio of 81.69%. Moreover, the model is able to produce sequences
with the same length as the ground truth in 91.47% of samples. Due to the
negativeness of other numbers in the table, applying any of the mentioned
changes in the architecture causes the performance of the model to drop.
Avoiding the use of the convolutional or BiLSTM layers has a significant
impact on the model’s performance, which respectively indicates the high
importance of the Feature Fusion component and the role of the bidirectional
LSTM in detecting gestures with the same opening movements. Also, using just
one LSTM layer instead of two layers or using a $5\times 5$ kernel for the
convolution layer reduces the performance. The former is due to the fact that
using two layers of LSTM can help us detect more complex patterns in the data,
and the latter is because the $3\times 3$ kernel allows features to be
combined in more ways. Finally, the use of batch normalization between each
pair of perceptron layers makes the length of the predicted sentences much
closer to their actual value, which ultimately increases the accuracy of the
model.

|  |  Subject 1 |  Subject 2 |  Subject 3 |  Subject 4 |  Subject 5 |  Average
---|---|---|---|---|---|---|---
Metric |  Model |  |  |  |  |  |
SLAcc |  1 |  93.22 / 90.58 |  91.54 / 95.34 |  96.01 / 95.51 |  95.45 / 97.56 |  97.55 / 96.73 |  94.79 / 95.15
|  2 |  86.94 / 90.91 |  96.03 / 97.58 |  95.85 / 93.69 |  96.10 / 97.89 |  96.73 / 98.85 |  94.33 / 95.78
SAcc |  1 |  82.31 / 82.98 |  86.87 / 89.64 |  89.04 / 90.70 |  83.77 / 85.39 |  91.82 / 91.16 |  86.76 / 87.95
|  2 |  78.02 / 82.81 |  89.12 / 91.88 |  86.38 / 85.71 |  87.01 / 87.50 |  91.82 / 91.82 |  86.46 / 87.92
WAcc |  1 |  91.32 / 91.76 |  93.09 / 94.39 |  94.38 / 95.60 |  90.34 / 91.31 |  95.34 / 95.74 |  92.89 / 93.75
|  2 |  89.39 / 91.76 |  94.27 / 95.54 |  93.27 / 92.91 |  91.75 / 92.32 |  95.74 / 95.72 |  92.87 / 93.63
WWAcc |  1 |  91.10 / 91.48 |  93.68 / 95.00 |  93.82 / 94.94 |  92.22 / 93.13 |  95.94 / 95.72 |  93.35 / 94.04
|  2 |  89.43 / 91.33 |  94.77 / 96.02 |  92.45 / 92.37 |  93.66 / 94.18 |  96.17 / 96.02 |  93.29 / 93.97

Table 4. Evaluation of models based on the subject-independent approach on
Dataset1-3. Each column (except the average column) illustrates the results
obtained by testing the model for the corresponding subject while having
trained on the data from other subjects. The numbers in the first and second
rows of each cell represent models $M_{1}$ and $M_{2}$, respectively. The
numbers on the left indicate the performance before ensembling, while the
numbers on the right indicate the performance after ensembling.

###  5.3. Ensembling Performance

In this section, we examine the performance of the proposed ensembling method
and its effectiveness in improving accuracy. According to Table 3, since the
results of the model that uses one batch normalization layer ($M_{1}$) are, on
average, close to our proposed model ($M_{2}$), we will compare these models
in our experiments. Also, in addition to Dataset1-3, we use Dataset4-8 to
evaluate the performance of the model in longer sentences.

The tables in this section show the performance of $M_{1}$ and $M_{2}$ before
and after applying ensembling on different datasets and with different
approaches. The models related to Table 4 and Table 5 were trained on the data
of 4 subjects from Dataset1-3 and tested on the data of the remaining subject
from Dataset1-3 and Dataset4-8, respectively. On the other hand, Table 6 shows
the performance of the models by being trained on the data of 4 subjects from
Dataset1-3 and tested on all the data of Dataset4-8; therefore, the results in
this table are not based on the subject-independent approach. It should be
noted that in each cell, the numbers on the left indicate the performance of
the best model among the five models trained using 5-fold cross-validation
before applying ensembling. In contrast, the numbers on the right show the
performance of the ensembled models.

Although some cells in the tables show minor degradation, in the average
column, either one of the ensembled models outperforms others across all 12
cases. Notably, in eight cases, the top two ranks are also secured by
ensembled models. In the remaining four cases, $M_{2}$ alone outperforms the
ensembled model of $M_{1}$ due to its inherent excellence. By assigning
numerical values from 1 to 4 to the spectrum of colors, ranging from red to
green, and substituting them with the corresponding values in each cell, we
can compute an average rank for each model. This provides an overall view of
the effect of the ensembling method. The result of this analysis is
represented in Table 7. It illustrates a similar pattern in the results of
Table 4 and Table 6, where the ensembled models of $M_{2}$ and $M_{1}$ are in
the first and second place, respectively. On the contrary, the results of
Table 5 show $M_{2}$ and its ensemble yielding the best results. This
observation indicates that the use of batch normalization between all layers
in the classifier improves the performance of $M_{2}$, such that it can
perform even better than the ensembled version of $M_{1}$. In other words,
batch normalization enables the model to generalize better in longer unseen
sequences when the subject-independent approach is used.

|  |  Subject 1 |  Subject 2 |  Subject 3 |  Subject 4 |  Subject 5 |  Average
---|---|---|---|---|---|---|---
Metric |  Model |  |  |  |  |  |
SLAcc |  1 |  55.00 / 75.00 |  75.00 / 90.00 |  100.0 / 100.0 |  95.00 / 100.0 |  85.00 / 85.00 |  82.00 / 90.00
|  2 |  90.00 / 90.00 |  85.00 / 90.00 |  100.0 / 100.0 |  100.0 / 100.0 |  95.00 / 95.00 |  94.00 / 95.00
SAcc |  1 |  45.00 / 60.00 |  55.00 / 60.00 |  90.00 / 95.00 |  85.00 / 90.00 |  55.00 / 70.00 |  66.00 / 75.00
|  2 |  75.00 / 75.00 |  45.00 / 55.00 |  95.00 / 95.00 |  95.00 / 95.00 |  65.00 / 75.00 |  75.00 / 79.00
WAcc |  1 |  82.04 / 88.90 |  90.20 / 90.73 |  94.38 / 95.00 |  96.92 / 98.33 |  89.49 / 93.87 |  90.61 / 93.37
|  2 |  94.88 / 93.88 |  86.68 / 88.90 |  95.00 / 95.00 |  99.17 / 99.17 |  94.61 / 95.95 |  94.07 / 94.58
WWAcc |  1 |  83.33 / 90.83 |  90.83 / 90.83 |  93.33 / 94.17 |  97.50 / 98.33 |  90.83 / 94.17 |  91.16 / 93.67
|  2 |  95.83 / 95.00 |  87.50 / 89.17 |  94.17 / 94.17 |  99.17 / 99.17 |  94.17 / 95.83 |  94.17 / 94.67

Table 5. Evaluation of models based on the subject-independent approach on
Dataset4-8

|  |  Subject 1 |  Subject 2 |  Subject 3 |  Subject 4 |  Subject 5 |  Average
---|---|---|---|---|---|---|---
Metric |  Model |  |  |  |  |  |
SLAcc |  1 |  86.00 / 91.00 |  84.00 / 95.00 |  89.00 / 92.00 |  84.00 / 94.00 |  87.00 / 94.00 |  86.00 / 93.20
|  2 |  90.00 / 94.00 |  88.00 / 94.00 |  87.00 / 95.00 |  93.00 / 96.00 |  91.00 / 97.00 |  89.80 / 95.20
SAcc |  1 |  74.00 / 81.00 |  73.00 / 86.00 |  79.00 / 90.00 |  75.00 / 88.00 |  73.00 / 84.00 |  74.80 / 85.80
|  2 |  79.00 / 86.00 |  77.00 / 84.00 |  79.00 / 88.00 |  87.00 / 92.00 |  82.00 / 90.00 |  80.80 / 88.00
WAcc |  1 |  92.46 / 95.21 |  92.40 / 96.15 |  95.30 / 97.05 |  94.27 / 97.03 |  93.21 / 96.08 |  93.53 / 96.30
|  2 |  95.55 / 96.50 |  94.14 / 95.28 |  94.78 / 96.66 |  96.46 / 97.70 |  95.99 / 97.37 |  95.38 / 96.70
WWAcc |  1 |  92.17 / 95.33 |  92.83 / 96.17 |  95.33 / 97.17 |  94.33 / 97.00 |  93.67 / 96.17 |  93.67 / 96.37
|  2 |  95.50 / 96.50 |  94.17 / 95.50 |  95.00 / 96.83 |  96.67 / 97.67 |  96.00 / 97.33 |  95.47 / 96.77

Table 6. Evaluation of models based on the subject-dependent approach on
Dataset4-8. Each column (except the average column) illustrates the results
obtained by testing the model for the corresponding subject while having
trained on the data from all subjects.

So far, we have demonstrated that despite causing minor degradations in
certain metrics for specific subjects, the application of ensembling, on
average, improves the overall performance. Now, we will examine the extent of
this impact. Table 4 indicates that ensembling boosts both WAcc and SLAcc,
resulting in 1.19% and 1.46% improvement in SAcc of $M1$ and $M2$,
respectively. Table 5 reveals that ensembling effectively addresses $M_{1}$’s
weakness in recognizing sequences with 4 to 8 words, bringing it closer to
$M_{2}$’s performance with a 2.76% and 0.51% increase in WAcc for $M_{1}$ and
$M_{2}$, respectively. This translates to a remarkable 9.00% and 4.00%
enhancement in recognizing complete sequences for each model. Moreover, the
models achieved slightly higher results in terms of WWAcc, indicating that
they are capable of maintaining their accuracy while dealing with longer
sentences. The results of Table 6 show that the impact of ensembling is even
higher while using the subject-dependent approach. In that experiment, $M_{1}$
and $M_{2}$ witnessed a 2.77% and 1.32% increase in WAcc, resulting in an
11.00% and 7.20% improvement in recognizing complete sequences, respectively.
Moreover, our method significantly improved the ability of models to predict
the length of output sequence, resulting in 7.20% and 5.40% improvement,
respectively.

To wrap up, our experiments in this section show two main points. First, a
better generalization in longer sequences is the underlying cause for using
batch normalization between all layers in the classifier. Second, using our
ensembling approach yields considerable improvements in both models, making it
a suitable choice for our SLR task.

|  Model |  Table 2 |  Table 3 |  Table 4
---|---|---|---|---
Average ranks |  1 |  1.96 / 2.83 |  1.21 / 2.33 |  1.13 / 3.21
|  2 |  2.08 / 3.13 |  2.75 / 3.71 |  1.95 / 3.71

Table 7. The average ranking of each model based on the results of
experiments. The ranks range from 1 to 4 corresponding to models’ performance
in ascending order.

##  6\. Limitations and Future Work

###  6.1. Neglecting Non-manual Markers

Although PenSLR is capable of capturing diverse hand movements and finger
positions, it lacks the ability to detect non-manual markers, such as facial
expressions and head movements. As previously mentioned, non-manual markers
play a crucial role in all sign language variations. In fact, some gestures
may share similar hand movements and finger configurations, yet differ in
subtle facial expressions. Therefore, incorporating non-manual markers not
only expands the range of glosses but also enhances the system’s ability to
differentiate between gestures with similar manual markers. In the future, we
could manipulate and improve our glove-based design so that we can take into
account these non-manual markers. To achieve this goal, one promising approach
involves employing stretchable sensors attached to the face, as suggested in
(Zhou et al., 2020), or harnessing earbud signals, as explored in SmartASL
(Jin et al., 2023).

###  6.2. Limited Dataset

As explained in Section 3, we collected a dataset of over 3000 samples from 16
PSL glosses. Although our dataset stands as the largest within the PSL domain,
the PSL lexicon extends beyond this limited scope, and for effective
communication among sign language users, a broader vocabulary is essential.
Therefore, it is crucial to prioritize the expansion of our dataset to include
a more extensive array of PSL signs. This will facilitate more comprehensive
communication within the PSL community. Additionally, collaborating with the
Persian Deaf Community Association, particularly in the data gathering
process, could help us to improve the quality of our dataset and tailor it to
real-world applications.

###  6.3. Ensembling Limitations

Our ensembling algorithm does not take into account any linguistic
relationship between the predicted signs. Although this counts as a benefit in
language-independent tasks, such as gesture recognition, the situation is
different when linguistic features are decisive. For instance, in sign
language translation, where the goal is to convert sign language glosses into
natural language text, our algorithm may fall short in preserving lexical and
grammatical correctness. This limitation arises because the use of sequence
alignment can disrupt the coherence of sentence structures. However, to
overcome this issue, a language model can be used to revise the output of
ensembling to preserve the sentence structure.

##  7\. Conclusion

In this paper, we propose PenSLR, the first Persian sign language recognition
system capable of detecting sign language sentences in an end-to-end manner.
To achieve this, we design a cost-effective glove-based system that can
capture a wide range of manual markers. Moreover, we construct and publish the
largest PSL dataset to date, comprising more than 3000 samples from 16
commonly used signs. To tackle the recognition task, we deploy a CRNN
architecture coupled with the CTC loss function, facilitating the generation
of variable-length outputs based on input signals. Furthermore, we introduce a
novel ensembling technique based on sequence alignment and utilize multiple
instances of our model to generate an improved output sequence. PenSLR
achieves an impressive 94.07% word-level accuracy while testing on longer
unseen sentences using the subject-independent approach. Notably, when
employing our ensembling scheme, this represents a 0.51% enhancement over the
performance achieved solely with the CRNN model, leading to a 94.58% word-
level accuracy. This effect of the ensembling is even higher when a subject-
dependent approach is used, where the test data belongs to the same subjects
as the training data.

###### Acknowledgements.

We would like to extend a special thanks to the five volunteers who generously
donated their time and effort to help us record and collect the necessary
data. Their unwavering commitment was crucial to the success of our project.

## References

  * (1)
  * Basiri et al. (2023) Salar Basiri, Alireza Taheri, Ali F. Meghdari, Mehrdad Boroushaki, and Minoo Alemi. 2023.  Dynamic Iranian Sign Language Recognition Using an Optimized Deep Neural Network: An Implementation via a Robotic-Based Architecture.  _International Journal of Social Robotics_ 15, 4 (April 2023), 599–619.  https://doi.org/10.1007/s12369-021-00819-0
  * Chao et al. (2022) Jiannan Chao, Furong Tang, and Lei Xu. 2022.  Developments in Algorithms for Sequence Alignment: A Review.  _Biomolecules_ 12, 4 (April 2022), 546.  https://doi.org/10.3390/biom12040546
  * Chouhan et al. (2014) Tushar Chouhan, Ankit Panse, Anvesh Kumar Voona, and S. M. Sameer. 2014.  Smart glove with gesture recognition ability for the hearing and speech impaired. In _Proceedings of the 2014 IEEE Global Humanitarian Technology Conference - South Asia Satellite (GHTC-SAS)_. IEEE, Trivandrum, India, 105–110.  https://doi.org/10.1109/GHTC-SAS.2014.6967567
  * DelPreto et al. (2022) Joseph DelPreto, Josie Hughes, Matteo D’Aria, Marco De Fazio, and Daniela Rus. 2022.  A Wearable Smart Glove and Its Application of Pose and Gesture Detection to Sign Language Classification.  _IEEE Robotics and Automation Letters_ 7, 4 (Oct. 2022), 10589–10596.  https://doi.org/10.1109/LRA.2022.3191232
  * Graves et al. (2006) Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In _Proceedings of the 23rd international conference on Machine learning (ICML)_. ACM Press, Pittsburgh, Pennsylvania, 369–376.  https://doi.org/10.1145/1143844.1143891
  * Hu et al. (2021) Hezhen Hu, Wengang Zhou, Junfu Pu, and Houqiang Li. 2021.  Global-Local Enhancement Network for NMF-Aware Sign Language Recognition.  _ACM Transactions on Multimedia Computing, Communications, and Applications_ 17, 3 (Aug. 2021), 1–19.  https://doi.org/10.1145/3436754
  * Jin et al. (2021) Yincheng Jin, Yang Gao, Yanjun Zhu, Wei Wang, Jiyang Li, Seokmin Choi, Zhangyu Li, Jagmohan Chauhan, Anind K. Dey, and Zhanpeng Jin. 2021.  SonicASL: An Acoustic-based Sign Language Gesture Recognizer Using Earphones. In _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)_ , Vol. 5. ACM, New York, NY, USA, 1–30.  https://doi.org/10.1145/3463519
  * Jin et al. (2023) Yincheng Jin, Shibo Zhang, Yang Gao, Xuhai Xu, Seokmin Choi, Zhengxiong Li, Henry J. Adler, and Zhanpeng Jin. 2023.  SmartASL: ”Point-of-Care” Comprehensive ASL Interpreter Using Wearables. In _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)_ , Vol. 7. ACM, New York, NY, USA, 1–21.  https://doi.org/10.1145/3596255
  * Karami et al. (2011) Ali Karami, Bahman Zanj, and Azadeh Kiani Sarkaleh. 2011.  Persian sign language (PSL) recognition using wavelet transform and neural networks.  _Expert Systems with Applications_ 38, 3 (March 2011), 2661–2667.  https://doi.org/10.1016/j.eswa.2010.08.056
  * Khomami and Shamekhi (2021) Sara Askari Khomami and Sina Shamekhi. 2021.  Persian sign language recognition using IMU and surface EMG sensors.  _Measurement_ 168 (Jan. 2021), 108471.  https://doi.org/10.1016/j.measurement.2020.108471
  * Li et al. (2020) Wenguo Li, Zhizeng Luo, and Xugang Xi. 2020.  Movement Trajectory Recognition of Sign Language Based on Optimized Dynamic Time Warping.  _Electronics_ 9, 9 (Aug. 2020), 1400.  https://doi.org/10.3390/electronics9091400
  * Liu et al. (2023) Yuxuan Liu, Xijun Jiang, Xingge Yu, Huaidong Ye, Chao Ma, Wanyi Wang, and Youfan Hu. 2023.  A wearable system for sign language recognition enabled by a convolutional neural network.  _Nano Energy_ 116 (Nov. 2023), 108767.  https://doi.org/10.1016/j.nanoen.2023.108767
  * Lu et al. (2023) Chenghong Lu, Shingo Amino, and Lei Jing. 2023.  Data Glove with Bending Sensor and Inertial Sensor Based on Weighted DTW Fusion for Sign Language Recognition.  _Electronics_ 12, 3 (Jan. 2023), 613.  https://doi.org/10.3390/electronics12030613
  * Mittal et al. (2019) Anshul Mittal, Pradeep Kumar, Partha Pratim Roy, Raman Balasubramanian, and Bidyut B. Chaudhuri. 2019.  A Modified LSTM Model for Continuous Sign Language Recognition Using Leap Motion.  _IEEE Sensors Journal_ 19, 16 (Aug. 2019), 7056–7063.  https://doi.org/10.1109/JSEN.2019.2909837
  * Muralidharan et al. (2022) Nitin Thoppey Muralidharan, Rahul Ram S., Rohidh M. R., Senthil Nathan M., and Harikumar M. E. 2022.  Modelling of Sign Language Smart Glove Based on Bit Equivalent Implementation Using Flex Sensor. In _2022 International Conference on Wireless Communications Signal Processing and Networking (WiSPNET)_. IEEE, Chennai, India, 99–104.  https://doi.org/10.1109/WiSPNET54241.2022.9767137
  * Needleman and Wunsch (1970) Saul B. Needleman and Christian D. Wunsch. 1970.  A general method applicable to the search for similarities in the amino acid sequence of two proteins.  _Journal of Molecular Biology_ 48, 3 (March 1970), 443–453.  https://doi.org/10.1016/0022-2836(70)90057-4
  * Sanjabi et al. (2016) Ali Sanjabi, Abbas Ali Behmanesh, Ardavan Guity, Sara Siyavoshi, Martin Watkins, and Julie A. Hochgesang. 2016.  Zaban Eshareh Irani (ZEI) and Its Fingerspelling System.  _Sign Language Studies_ 16, 4 (2016), 500–534.  https://doi.org/10.1353/sls.2016.0010
  * Santhalingam et al. (2020) Panneer Selvam Santhalingam, Al Amin Hosain, Ding Zhang, Parth Pathak, Huzefa Rangwala, and Raja Kushalnagar. 2020.  mmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals. In _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)_ , Vol. 4. ACM, New York, NY, USA, 1–30.  https://doi.org/10.1145/3381010
  * Shang and Wu (2017) Jiacheng Shang and Jie Wu. 2017.  A Robust Sign Language Recognition System with Multiple Wi-Fi Devices. In _Proceedings of the Workshop on Mobility in the Evolving Internet Architecture (MobiArch)_. ACM, Los Angeles CA USA, 19–24.  https://doi.org/10.1145/3097620.3097624
  * Sharma et al. (2023) S. Sharma, R. Gupta, and A. Kumar. 2023.  Continuous sign language recognition using isolated signs data and deep transfer learning.  _Journal of Ambient Intelligence and Humanized Computing_ 14, 3 (March 2023), 1531–1542.  https://doi.org/10.1007/s12652-021-03418-z
  * Thompson et al. (1994) Julie D. Thompson, Desmond G. Higgins, and Toby J. Gibson. 1994.  CLUSTAL W: improving the sensitivity of progressive multiple sequence alignment through sequence weighting, position-specific gap penalties and weight matrix choice.  _Nucleic Acids Research_ 22, 22 (1994), 4673–4680.  https://doi.org/10.1093/nar/22.22.4673
  * Wang et al. (2020) Zhibo Wang, Tengda Zhao, Jinxin Ma, Hongkai Chen, Kaixin Liu, Huajie Shao, Qian Wang, and Ju Ren. 2020.  Hear Sign Language: A Real-time End-to-End Sign Language Recognition System.  _IEEE Transactions on Mobile Computing_ 21, 7 (2020), 2398–2410.  https://doi.org/10.1109/TMC.2020.3038303
  * Wen et al. (2021) Feng Wen, Zixuan Zhang, Tianyiyi He, and Chengkuo Lee. 2021.  AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove.  _Nature Communications_ 12, 1 (Sept. 2021), 5378.  https://doi.org/10.1038/s41467-021-25637-w
  * Wu et al. (2016) Jian Wu, Lu Sun, and Roozbeh Jafari. 2016.  A Wearable System for Recognizing American Sign Language in Real-Time Using IMU and Surface EMG Sensors.  _IEEE Journal of Biomedical and Health Informatics_ 20, 5 (Sept. 2016), 1281–1290.  https://doi.org/10.1109/JBHI.2016.2598302
  * Zhang et al. (2023) Huaiwen Zhang, Zihang Guo, Yang Yang, Xin Liu, and De Hu. 2023.  C${}^{\textrm{2}}$ST: Cross-modal Contextualized Sequence Transduction for Continuous Sign Language Recognition. In _Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE, Paris, France, 20996–21005.  https://doi.org/10.1109/ICCV51070.2023.01925
  * Zhang et al. (2022) Qian Zhang, JiaZhen Jing, Dong Wang, and Run Zhao. 2022.  WearSign: Pushing the Limit of Sign Language Translation Using Inertial and EMG Wearables. In _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT)_ , Vol. 6. ACM, New York, NY, USA, 1–27.  https://doi.org/10.1145/3517257
  * Zhang et al. (2019) Qian Zhang, Dong Wang, Run Zhao, and Yinggang Yu. 2019.  MyoSign: enabling end-to-end sign language recognition with wearables. In _Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI)_. ACM, Marina del Ray California, 650–660.  https://doi.org/10.1145/3301275.3302296
  * Zhou et al. (2020) Zhihao Zhou, Kyle Chen, Xiaoshi Li, Songlin Zhang, Yufen Wu, Yihao Zhou, Keyu Meng, Chenchen Sun, Qiang He, Wenjing Fan, Endong Fan, Zhiwei Lin, Xulong Tan, Weili Deng, Jin Yang, and Jun Chen. 2020.  Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays.  _Nature Electronics_ 3, 9 (June 2020), 571–578.  https://doi.org/10.1038/s41928-020-0428-6
  * Zuo and Mak (2024) Ronglai Zuo and Brian Mak. 2024.  Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal.  _ACM Transactions on Multimedia Computing, Communications, and Applications_ 20, 6 (June 2024), 1–25.  https://doi.org/10.1145/3640815