# Enhancing Sign Language Detection through Mediapipe and Convolutional Neural
Networks (CNN)

Gagandeep Singh1 Aditya Raj Verma2
<EMAIL_ADDRESS><EMAIL_ADDRESS>Banavath RamJi3 Karnim
Meghwal4
<EMAIL_ADDRESS><EMAIL_ADDRESS>(Department of Computer
Science and Engineering)
Indian Institute of Information and Technology Kota  Praveen Kumar Dadheech
(Indian Institute of Information Technology)
<EMAIL_ADDRESS>
###### Abstract

This research combines MediaPipe and CNNs for the efficient and accurate
interpretation of ASL dataset for the real-time detection of sign language.
The system presented here captures and processes hands’ gestures in real time.
the intended purpose was to create a very easy, accurate, and fast way of
entering commands without the necessity of touching something.
MediaPipe supports one of the powerful frameworks in real-time hand tracking
capabilities for the ability to capture and preprocess hand movements, which
increases the accuracy of the gesture recognition system. Actually, the
integration of CNN with the MediaPipe results in higher efficiency in using
the model of real-time processing.The accuracy achieved by the model on ASL
datasets is 99.12%.The model was tested using American Sign Language (ASL)
datasets. The results were then compared to those of existing methods to
evaluate how well it performed, using established evaluation techniques. The
system will have applications in the communication, education, and
accessibility domains. Making systems such as described in this paper even
better will assist people with hearing impairment and make things accessible
to them. We tested the recognition and translation perfor- mance on an ASL
dataset and achieved better accuracy over previous models. It is meant to the
research is to identify the characters that American signs recognize using
hand images taken from a web camera by based on mediapipe and CNNs
Keywords: MediaPipe, Convolutional Neural Networks(CNN), Real-Time Processing,
Hand Tracking, Sign language Detection

##  1 Introduction

Sign language, an important means of communication unicatipon to people who
are deaf or a hard of hearing. deficits, allowing them to ”talk” ideas, and
emotions. However, the absence tools automated to real-time read It´s
Difficult for Foster Care-Language Sign Communication Resources and Access for
the Deaf and the Dumb community. [1]
Deafness may be defined as hearing loss that is so profound that it renders
the person unable to hear any sound. This magnitude of hearing loss may result
from aging, genetics, noise, and other etiologies such as repeated otitis
media or the effect of certain chemicals or medicines that eventually may
affect our hearing. Our hearing loss amplitudes for this extent of loss also
vary from a mild loss that is between 25 to 40 dB; moderate loss, between
41-55 dB; moderate to severe that is 56 to 70 dB, severe that is 71 to 90 dB;
and profound that is beyond 90 dB. It is estimated that at sound levels above
90 dB, there are 1.33 billion people with a hearing impairment, making up
18.5% of the world’s population in 2015.

Mutism is a condition where a person’s ability to speak is either partially or
lacking. Similarly to deafness, mutism may result organically or from
psychological, developmental, or nervous system disturbances. There are
several forms of therapy, although not all can benefit from auditory
habilitation, and not everyone can get involved in the formal labor market.

There were 124 million people severely disabled in the low and middle-income
countries, and 108 million people suffer from such disabilities in 2013. Many
in the deaf community see their deafness as just another culture, and the
implant is threatening that way of life.[2]

The most notable ones include the sign language. These can be further broken
down into vision-based and sensor-based methods. Whereas vision-based methods
utilize cameras to capture hand moments, sensor-based methods use gadgets such
as data gloves and accelerometers to measure hands. However, some of these
technologies are effective but also costly and specialized.

We tested our system using ASL datasets. Image-based recognition algorithms in
ASL for the static method need preprocessing of the images to establish
meaningful features. Below, the ASL databases with 87,000 images of American
signs, which will be used to enrich and generalize our models, ensue in this
paper.[3]

Figure 1: American Sign Language hand gestures (ASL)[4]

This research displays the application of 21 hand coordinate points according
to the newly developed tool, MediaPipe, applied to get the 2D images of the
hands[5]. The language tool was tested before concluding. The best among all
was MediaPipe since the work is reliable, and with this accuracy, it is
helpful to us. Then, based on these hand coordinates or joint points, features
can be extracted based on distances and angles. Sign recognition is then
performed by using a set of convolutional neural networks. No special devices
are needed since the feature extraction process can get 3D skeletal joint
points from 2D images. For example, applying CNN on the ASL dataset resulted
in very high accuracy, as shown in one study: for recognition of five letters,
it was 98%, and for recognition of ten letters, it was 74%. From that
perspective, this result is promising and in line with our objective of
enabling people to recognize alphabets communicated using ASL[6]. Our
application should be able to understand sign language and convert it to text.

##  2 Related Work

Nagi et al[7] developed a CNN-based recognition system for gesture
applications in human-robot interactions. Work on the realization of
convolution neural networks for real-time translation of various hand gestures
can be considered vital towards finding effective communications between
humans and robots. In truth, the system predominantly deals with well-posed
gestures and, at times, misinterprets some of the hand signs, which are parts
of sign language. Van den Bergh[8] and his colleagues developed a hand posture
recognition system by Haar wavelet-based feature extraction and database
searching. They had to find a match for features extracted using Haar wavelets
inside a database of gestures. They had excellent results, though in the
study, they only tested for six classes of gestures.Kang et al.[9]have
developed in support of CNN, a natural time recognition system for ASL, using
31 hand symbols, which are acquired by taking 31,000 images using the Creative
Senz3D Camera towards the acquisition of depth maps. The system achieved an
accuracy of around 85.49% by excluding numerous gestures, which mainly require
hands to be gripped closer. Gesture recognition system, hand gesture
interpretation, development of the system: The system was developed by Song et
al. using the Creative Senz3D depth map camera to interpret hand gestures.
Their recognition methodology was worthy of the whole solution, yet the
equipment on which the research was done is relatively expensive and does not
make it practical for everyday use. Salian et al.[4] developed a hand gesture
recognition system using Haar-like features with a cascade of classifiers
coupled with a convolutional neural network engine. They trained a cascade
function with a huge number of positive and negative images, which made the
observation of hand gestures quite effective. The case of Rahman et al.[10]
affirms a model designed through SLRNet with eight layers over the CNN
architecture for the problem of identification of hand gestures, which could
attain a performance of 67% for letters and 82.5% for MU HandImages ASL
numbers using the developed ASL dataset. The developed system consists of 6
convolution layers, three pooling layers, and one fully connected layer and
can recognize ASL and finger-spelling gestures.
In their research, two datasets were employed: one that was generated by
themselves and another dataset called MU HandImages ASL [12] (Vivek Bheda et
al., 2010). In case of letters their method achieved 67% accuracy while
raising this number up to 82.5% when dealing with number recognition as well
(Bheda et al., 2010). . In this regard, Starner et al.[17] proposed two real-
time ASL recognition systems. One was done using a desk-mounted camera with a
recognition rate of 92%, and another was done using a cap-mounted camera,
which reached 98% recognition. Both the two implementations that they used had
a 40-word lexicon, results indicating the efficiency of the recognition of
continuous ASL gestures was high. Suk et al.[15] developed a dynamic Bayesian
network model for ASL recognition from video streams. His evaluated system,
with ten isolated gestures, performed better—a recognition rate of more than
99%—using the observed skin extraction technique of modeling and motion
tracking. Later, W. Huang et al.[13] designed a 3D Hopfield neural network to
recognize hand gestures. The system processes a gestural hand-tracking
process, feature extraction, and recognition; the average recognition rate
reached 91% for 15 hand gestures. T. Starner et al.[14] developed a real-time
system for recognizing sentence-level continuous ASL. The test systems were
mounted on the desk with 92% accuracy and the other mounted in the form of a
cap with 98% recognition. Both of these systems used the 40-word lexicon, and
this approach showed effectiveness.
Another study employed deep learning, by designing a feed-forward neural
network to classify a sign. Well-known image pre-processing methods require
background subtraction, image normalization, image segmentation, and contrast
optimization. Gabor filters used Principal component analysis for image
feature extraction. Accuracy was achieved in this case is 98.5%[18].
They developed a CNN-based hand gesture recognition over the fingerspelling
data of Rasha et al.[19] research, showing the possibility of surpassing 99%
recognition in 28 isolated gestures. Their system has shown itself to work
with high accuracy during real-time ASL recognition, thereby promising CNN for
gesture interpretation. But the generally recommended system is the hand
gesture recognition application in which SSD, the Inception v3, and the SVM
are combined. Their system had pointed toward the automatic recognition of
hands from camera images and the classification of those into 24 different ASL
signs. Thus, interestingly, the system shows the incorporation of different
machine-learning models into that just for effective gesture recognition. A
simplified sign language recognition model using the MediaPipe framework and
state-of-the-art machine learning algorithms was proposed in this work by
Arpita Halder et al.[20] The deployment of such a model seems to yield an
average accuracy of 99% over different datasets for sign languages and is easy
to do in real-time with acceptable detection. All these manifold advantages of
the proposed system are cost-effective and practical for deployment in
commercial settings.

##  3 Methodology

The real-time sign language translation system was developed through a
comprehensive approach. We cover the whole pipeline starting from data
collection and prepro cessing to model development and system implementation.
Special emphasis is placed on the use of MediaPipe for hand tracking and CNN
for recognizing hand gestures, discussing the architectural choices, training
process, and optimization strategies employed.. Further more, we illustrate
our means of capturing and processing sign language gestures in real-time, to
ensure a high level of accuracy and efficiency in translation.

###  3.1 System Architecture

Proposed architecture aims to accurately interpret and classify ASL hand
gestures. The architecture for a sign language recognition system leveraging
MediaPipe and Convolutional Neural Networks (CNN) involves several key
components and stages. This diagram shows a visual representation of how our
model works.

Figure 2: : Work Flow Model of our Architecture[20]

####  3.1.1 Data Collection

In this paper, we use ASL datasets to analyze the performance of system. The
ASL data set is the ASL Alphabet data set[21], which contains over 87,000
samples of 29 classes: 26 for the letters A-Z, and three special characters:
delete, nothing, and space. Some samples of the data set are shown in Fig. The
large size of such a dataset would enable the proposed architecture in this
work to learn features supported by a great deal of data during training.
Thus, it would genuinely learn even small details of hand movements and subtle
differences between ASL signs. Using a data set of this size allows the system
to grasp the subtle nuances and features between one another.

####  3.1.2 Detection of hand landmarks using MediaPipe

##### Mediapipe:

MediaPipe hand keypoint detection runs in a machine-learning pipeline. The
models in the MediaPipe ML pipeline can execute palm detection on an entire
image in a single forward pass to output a bounding box around the hand. This
is followed by accurate 3D hand keypoint[23] detection on the region of
interest. In the same line, the ML pipeline goes on to crop the hands using
the hand landmarks from the previous frames, so detection of the palm will be
applied to localize only if the last step failed.

Figure 3: Mediapipe Hand Landmarks[22]

Figure 3: 21 hand landmarks can be tracked using MediaPipe’s hand landmark
detector. This can be used for hand gesture classification, as shown by
Arsheldy Alvin et al.[24], using both MediaPipe and the K-Nearest Neighbors
algorithm. Here, in the use of MediaPipe is for extracting hand key points,
while the training/deployment is based on CNN to detect the applied ML
algorithm. It runs on both CPU and GPU; MediaPipe uses no added-on processing
power. In the MediaPipe hand tracking library that is used to generate the
data, there are almost 21 critical features identified in the hand. The data
contains all 26 alphabets and three unique characters, namely delete, nothing,
and space, corresponding to hand gestures with 30 frames of each hand gesture.
Also, the data constitutes 30 variations of each alphabet.[22]

####  3.1.3 Feature Extraction and Preprocessing

Feature extraction is known as the process of changing raw data into numerical
properties that can be processed, while at the same time retaining the
information evident in an initial data set. Applying machine learning directly
to the raw data will not give a good result compared to feature extraction. It
is during this stage that the feature extraction is turned into image tensors
compatible with CNN processing. . We confirm the dataset’s balance by
visualizing label distribution, ensuring no class bias with nearly equal
examples per class.

Figure 4: some feature extracted images

We apply Min-max Normalization on extracted features. Find min and max of
extracted feature then apply this equation.Mathematically:

$\text{Normalized Pixel Value}={\frac{x-\min(x)}{\max(x)-\min(x)}}$ (1)

,where x is Orignal Pixel value.
This normalization serves two primary purposes.First, it minimizes the effect
of the differences in illumination between the images. Differences in
illumination can be very damaging to the performance of an image-processing
algorithm[25]. The min-max normalization scaling of pixel values into a common
range makes relative brightness in different images more or less similar and
reduces this effect.

Secondly, Convolutional Neural Networks (CNNs) are known to converge faster on
data scaled to [0,1] versus [0,255]. Also, this is because scaling affects
gradient descent optimization, where the process of convergence is smooth and
stable[26]. In the step ‘normalization of data’, we re-scale from its original
level the pixel intensity value, which falls into the range [0,255], into a
normalized [0, 1]. This normalization could be done with min–max
normalization. Normalization is crucial for optimizing the performance and
convergence speed of our CNN model. Following normalization, we flatten the
landmarks each sample of 1D array of length 21×3=63 so shape of dataset will
be (N,63), where N is no of samples. For CNN input we reshape the dataset to
4D array. so shape of dataset will be (N,21,3,1), where N is no of samples, 21
is number of landmarks, 3 is the number of coordinates per landmark and 1
represents the number of colors channels. This step is essential for the CNN
to process the image data correctly:

Reshape Data Dimension=(87000,21,3,1) (2)

####  3.1.4 Data Augmentation

To mitigate the risk of overfitting and enhance the ro bustness of our model,
we employed data augmentation techniques on our training dataset. Data
augmentation in volves making minor modifications to the training images,
thereby creating additional training data from the existing dataset. This
approach maintains the label but changes the array representation of the image
[31]. Popular data aug mentation techniques include rotations, translations,
zoom ing, and more. In our approach, we implemented the following transfor
mations:

  * •

Random rotation of images by 10 degrees

  * •

Random zoom on images by 10

  * •

Random horizontal shift of images by 10width.

  * •

Random vertical shift of images by 10

###  3.2 CNN Model Description

The subsequent action involves employing the dataset to train the CNN Model.

##### CNN:

Convolutional Neural Networks belong to the class of deep learning neural
network architecture. This is where computer vision using artificial
intelligence is made for the computer to be able to read photo or image data.
The architecture design is set for CNN to ensure good performance and
consideration of the matching chosen convolution layers and the number of
neurons. No international standard has been accepted for selecting the count
of neurons and convolution layers. Some of the CNN’s various results are shown
based on the variation of the number of layers and the number of neurons
within each layer. Here, we have designed a maximization of recognition
accuracy-based CNN. The newly proposed CNN consists of three convolution
layers, two pooling layers, and a fully connected layer, apart from the input-
output layers. The primary significant steps of the proposed ASL recognition
method are illustrated in

Figure 5: Architecture of CNN[33]

Fig. 6.

Figure 6: Architecture of the proposed CNN.

##### Input Layer:

The pre-processed images are fed directly into the network via its input
layer.An input layer is a very first layer of a neural network, which receives
the raw input data—a kind of data, be it images or just text. The neurons in
this layer represent the features or dimensions of the input data, and the
number of neurons in this layer is derived from the shape of the input data.
In most cases, there is no activation function implemented to the input layer,
making the first stop in the processing pipeline of the neural network.

##### Convolution Layer:

The first layer is the convolution in the CNN: one of the basic building
blocks aimed at feature extraction from input data. The kernels in the salient
feature extraction layer apply themselves to the input data from the input
signal via forward and backward propagations. In our study, convolutional
layer convolves over input data to provide filters. It computes a dot product
along its sliding window with local regions, producing feature maps belonging
to different patterns or characteristics of the input. Key features of a
convolutional layer are weight sharing and the fact the filter is slided
across the input. Spatial hierarchies of features are captured in the now
evident feature maps. Optionally available parameters include padding, which
can control the spatial dimensions of the output feature maps. In general,
convolutional layers extract effective local patterns from input data, which
allow CNNs to learn a hierarchy of representations for various tasks like
image recognition or gesture recognition.

$Y_{i,j,k}=\sum_{u,v,c}X_{(i\times S+u-P),(j\times S+v-P),c}\times
W_{u,v,c,k}+b_{k}$ (3)

,where $Y_{i,j,k}$ is the value of the output feature map at position (i ,j )
in the kth channel.
$X_{(i\times S+u-P),(j\times S+v-P),c}$ is the value of the input feature map
at position $(i\times S+u-P,j\times S+v-P)$ in the cth channel.
$X_{(i\times S+u-P),(j\times S+v-P),c}$ is the weight parameter of the filter
at position (u,v) in the cth input channel and the kth output channel.
$b_{k}$ is the bias term associated with the kth output channel.
F represents the spatial dimension of the filters.
S is the stride.
P is the amount of zero padding applied to the input feature maps.
$C_{in}$ is the number of input channels.

##### Activation Function:

Activation functions decide in which way the next node should get activated.
We use the ReLU activation function[27], which makes all the negative numbers
zeros and retains the positive numbers as it is. The choice of ReLU I make
directly is because this model is getting trained. As defined, during
training, ReLU is many orders of magnitude faster than bounded activation
functions like the sigmoid or hyperbolic tangent function[28] and allows
training deep models without suffering from the vanishing gradients problem.
Definitions of a function like this:

$\text{ReLU(x)}=\text{max(0,x)}$ (4)

where x refers to the input to a neuron.

##### Pooling Layer:

In this case, the process of pooling is integral to the working of a deep-
learning system. It allows for super-fast training in a CNN, reduces the
network memory size, and reduces the number of connections between the
convolutional layers. We have implemented max-pooling in this regard. Through
max pooling, a sliding window is swept across an input space to output the
most significant value within that window. We have used a window size of 2x2
for max pooling, assuming that the stride size would have been taken as 2, so
there would have been no problem of overlap. The formula of max pooling is:

$N_{\text{out}}=\text{floor}\left(\frac{N_{\text{in}}-F}{S}\right)+1$ (5)

where, $N_{o}ut$: The output dimension (height or width) of the feature map
after the convolution operation.
$N_{i}n$: The input dimension (height or width) of the feature map before the
convolution operation.
F: The size of the filter (kernel) used in the convolution operation. If the
filter is square, F is the length of one side.
S: The stride of the convolution operation, which is the number of pixels by
which the filter moves at each step.

##### Global Average Pooling Layer:

The global average pooling (GAP) layer is a lot like the max-pooling layer.
The dissimilarity is that the mean value, instead of the maximum value,
replaces the whole region.Global average pooling reduces the number of total
model parameters, which might prevent overfitting and improve generalization.
These coefficients are very effective in providing spatial information for the
network and are used in tasks like object localization and detection. Global
average pooling keeps the important spatial information of feature maps in the
CNNs but reduces their dimensionality.

$\text{GAP}(X)=\frac{1}{H\times W}\sum_{i=1}^{H}\sum_{j=1}^{W}X_{ij}$ (6)

$X_{ij}$ is the value of the feature map X at spatial location (i,j) and
channel c,H represents the height dimension of the feature map, while W
denotes its width.

#####  Fully Connected Layer

The Fully Connected (FC) layer is fed with the MAP features generated by the
GAP layer, to which it applies the Hard Max Layer. In the FC layer, neurons
connect with neurons from other layers in one-to-one style. Because that’s
what its filters of size 1x1 represent, one may think of fc layer as a
convolution layer.

$Y=\sum_{i=1}^{H}\sum_{j=1}^{W}X_{ij}\times W_{ij}+b$ (7)

where Y is the output of the fully connected layer,
$X{ij}$ are the input features from the previous layer,
$W{ij}$ are the weights connecting each input feature to the output neuron, b
is the bias term,
H and W represent the dimensions of the input feature map, denoting height and
width, respectively.

##### Dropout Layer:

Dropout is a regularization technique most widely applied to reduce
overfitting in a neural network model, mainly in LSTMs. Each element is
dropped to zero with a probability of per iteration. Training of a model tends
to overfit, i.e., it overfits because the accuracy is way too high concerning
the testing accuracy. Till today, an example of a dropout layer in the final
fully connected layer of all the models, as in the picture above, helps save
the model from overfitting[20] and, therefore, enhances the accuracy by
randomly nulling the activation. The process conducted in this study used a
Dropout Probability of 0.5[21].

$\text{Dropout}(X)=\frac{1}{1-p}\cdot(X\odot M)$ (8)

where X, p, M are input to the dropout layer, probability and binary mask
matrix

##### Output Layer:

Now, the output from such models takes the form of the predictions that come
out of this classification model. This layer spits out the probability of
being any one of those classes. The target class is just the one that has the
most probability. In this final layer, the number of neurons is equivalent to
the number of output classes in a multiclass classification problem. This
softmax of the last layer allows us to develop probabilities of the classes so
that the target class is the one with a higher probability. CNNs minimize some
loss functions (objective function). We have used a softmax-based loss
function :

$\text{Loss}=\frac{1}{N}\sum_{i=1}^{N}-\log\left(\frac{e^{f_{i,y_{i}}}}{\sum_{j=1}^{C}e^{f_{i,j}}}\right)$
(9) $\text{softmax}(z_{i})=\frac{e^{z_{i}}}{\sum_{j=1}^{N}e^{z_{j}}}$ (10)

where N, C is total number of elements and classes and $z_{i}$ is the $i_{th}$
element of input vector z

####  3.2.1 Training

In our study, Adam optimizer has been used, as it is an optimization algorithm
adjusting learning rates in adaptive manners. We placed the learning rate at
0.001 because a smaller learning rate will make the dynamics of training
stable. Besides, we adopted categorical cross-entropy loss function suitable
for multi-classification tasks. This is a function that computes the
difference between the predicted probability distribution and the true
probability distribution of classes. Therefore, we picked a batch size of 32
for training the model since smaller batch sizes, like 32, are well balanced
between memory consumption and computational efficiency. The number of epochs
was also fixed at 200. Epochs determine the count of how many times an entire
training dataset is passed forward and backward through the neural network. We
use early stopping function to prevent overfitting and it monitores validation
loss, and stop training if the metrics stop improving for a certain number of
epochs. We train our model for as long as it can take, that is, 200 epochs, so
that we can get improved accuracy on the test dataset. We usually use a
validation split of 10%.

It does this by splitting up the validation data from the training set in such
a way that it creates a form of monitoring performance during training, which
represents 10% of the whole data. That validation set helps to do early
stopping and hyperparameter fine-tuning to curb overfitting. Secondly, we
instantiated two callback functions: EarlyStopping and ModelCheckpoint. The
function ‘EarlyStopping‘ stops the training of a model when there are no
further improvements beyond a certain limit in order to avoid overfitting
performance on the validation dataset. On the other hand, the
‘ModelCheckpoint‘ function saves the weights of the model for later use by,
let’s say, the best model depending on validation performance. The accuracy of
the trained model we get is 99.12%.The performance of the proposed model was
evaluated for recognition of ASL sign language.

##  4 Experiments and Results

###  4.1 Dataset

The Dataset has been broadly classified into Training and Testing Data.
Training Data has been classified and segregated into 29 classes, of which 26
alphabets A-Z and 3 other classes of SPACE, DELETE, NOTHING. The test data set
contains a mere 29 images, to encourage the use of real-world test images.

Figure 7: No. of Samples of each letter

The dataset has different parameters that are x ,y and z axes of the hand
joints. The dataset is divided into 80% for training,10% for validation and
10% for testing. In Figure 1, the ASL hand gestures of all the alphabets from
A to Z used in the proposed system are shown.

Figure 8: Class Distribution of Pie chart Figure 9: Distribution of Dataset
into Training, Validation, and Test Sets

###  4.2 CNN

Our CNN is proposed to be composed of three convolution layers, three pooling
layers, and two fully connected layer beside the input–output layers in Fig.6.
The training and assessment data are reshaped into a 4D array that will serve
as input to the Convolutional Neural Network model. The CNN model consists of
multiple layers from flattening layers, dense layers, and dropout layers.
Also, using the compile function, we have specified the optimizer and
specified which loss and metrics to be used when measuring the training.

Figure 10: Summary of Our Model Architecture

##### Activation Function:

This graph illustrates a few of the most heavily used activation functions
within neural networks. Sigmoid shrinks its values between 0 and 1, ReLU keeps
positive values as they are, tanh compresses its values between -1 and 1,
while softmax squashes logit values into a probability distribution. These
functions are important for modeling the complex relationships in the data.

Figure 11: Activation Functions

###  4.3 Model Accuracy and loss

We train the model with 200 epoch and we use early function to improve
overfitting and generalization. It monitors a specified validation loss and
stops training, if the metric stops improving for a cer- tain number of epochs
(defined by patience).

Figure 12: Epoch vs Accuracy on train and validation set

Fig. 12 shows the accuracy plot of the model over 11 epochs with respect to
both the training and validation datasets. Accuracy increases steadily during
training, but fluctuations in the validation may, in fact, indicate
overfitting or instability in learning features and may require further tuning
to achieve better generalization. Fig.13 shows the loss of the model for both
the training and validation datasets when it comes to 11 epochs. We can say
that the training data improves as the training loss goes down, but looking at
the validation loss, we see it’s not steady; it seems to be more like the
model is either overfitting or getting unstable.

Figure 13: Epoch vs Loss

###  4.4 Performance metrics

There are four commonly used evaluation metrics in the proposed system:
Utilized. They are accuracy, recall, precision and F-measure. Confusion matrix
corresponding to False Negative (FN), True negative (TN), true positive (TP)
and false positive (FP) are used for the same measuring the parameters for the
above classifier. The The metric definitions are given by the following
formulas: TP: TP is a good instance that accurately classifies the positive
class. TN: TN is short for the negative that is properly characterized of
negative. FP: FP tells bad example that is mischaracterized as Positive. FN:
FN stands for false negative, which is the ideal example that is not quite
fulfilled Is this bad?. Accuracy: It is a measure of the correctness of
detection.

$\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$ (11)

Precision: it defined the ratio of predicted positives which are actual
positive.

$\text{Precision}=\frac{TP}{TP+FP}$ (12)

Recall: it defines that the number of positives recognised properly.

$\text{Recall}=\frac{TP}{TP+FN}$ (13)

Figure 14: Performance of Model

F-measure: it defines the vocal means of recall and precision.

$\text{F-measure}=\frac{2TP}{2TP+FP+FN}$ (14)

Figure 15: Class vs Score

###  4.5 Performance Analysis

ROC curve:
An ROC curve is best represented for a multi-class classification model,
showing True Positive Rate versus False Positive Rate for 29 classes. All of
these have an Area Under the Curve of 1.00, which shows that the model is
excellent in classification with great discrimination power between positive
and negative classes for all categories. Fig. 16 shows ROC curve.

Figure 16: ROE Curve

PR curve:
PR curve, which stands for Precision/Recall curve, illustrates the performance
of multiple class models over 29 classes in this case. For every class, the AP
value is approximate to or equal to 1, indicating they perform excellently.

Figure 17: Precision Recall curve

It yielded high values for the F-score represented by the number 00, which
simply shows that the model is good for the performance and coordinated with
poor accuracy for both precision and recall.Fig. 17 shows PR curve.

Learning curve:
Fig. 18 shows learning curve for the model. For completeness, both the
training and validation losses are shown, as well as accuracy for both. The
two loss curves are indeed very low and flat, while accuracy is raised to 1.0,
which actually gives an overfitting interpretation or an infinitely simple
task, so the model quickly arrives at the optimum performance.

Figure 18: Learning Curve

Confusion Matrix
The confusion matrix library is used for the analysis of the real-time
performance of the model. It provides a confusion matrix measuring the number
of accurately predicted labels. It also enables visualization of deviation of
prediction in case of false predictions by the model. Fig. 19 shows the
confusion matrix of the model.

Figure 19: Confusion Matrix

###  4.6 Real time recognition

The output of Figure 18 and 19 presents snapshots of sign language ’R’ and ’O’
from a real-time recognition system implemented with the OpenCV open-source
library using the model.

Figure 20: Sign language of R Figure 21: Sign language of O

the model is successfully extract landmarks and predict the sign. it is
accurately able to recognize live gestures from the American sign language.

##  5 Discussion

This paper presents an emergent approach to sign language detection using the
combined solutions by MediaPipe and Convolutional Neural Networks, which is
used for interpreting hand gestures. It is really going to be important in
assistive technology, aids for communication, and even human-computer
interaction. CNNs process the image of the hand sign to be classified. Use
MediaPipe to perform hand tracking. Data collection should initially be done,
like recording of sign language gestures. After that, preparatory steps must
also be done like normalization and augmentation. CNN architecture:
convolutional layers for the extraction of features, and pooling layers for
downsampling, combined with fully connected layers for classification. In
general, the CNN architecture uses convolutional layers to help carry out
features from the data. The pooling layers help in the downsampling process,
while the fully connected layers are used to give the output after
classification. Training optimizes model parameters with gradient-based
algorithms, and metrics are evaluated to assess performance. The final trained
model is now deployed into the real-time interpretation using MediaPipe for
efficient hand tracking. Challenges: hand shape, movement, changes in
lighting. Future research will be maintained to develop more robust models for
wider applicative usage on sign language detections.

##  6 Conclusion and Future Work

###  6.1 Conclusion

In short, the proposed ASL recognition model provides an accuracy of 99.12%
under a real-time system. The classification, through the features of
Mediapipe plus CNN as proposed in the model, is robust and effective in
identifying the hand motion with the American Sign Language. In this study, we
applied MediaPipe to identify landmarks of the hand; afterward, important
input features are extracted for the inputs of the CNN model to capture
spatial relationships and patterns of sign language gestures. The approach
yielded higher classification performance and thus holds immense potential for
use as a communication aid, educational tool, and human-computer interaction
system. The idea can be further extended in future work: improving performance
with the use of a more sophisticated CNN architecture, adding more contextual
information, and expanding the dataset.

###  6.2 Future Work

In the future, researchers should focus on a database with different sign
languages, dynamic symbols and constant sentences. Fundamentally, the model
needs to be enhanced so that it is better capable at translating texts in
real-time, employs both hands and remains secure amidst different
environments. To achieve this it is necessary to integrate gesture recognition
with NLP for text or speech translation and make the appropriate adjustments
based on individual signing styles resulting from transfer learning processes.
User studies are expedient for feedback and correction of ethical concerns on
data privacy that would bring about the acceptance and ethical deployment of
sign language detection systems.

##  7  References:

  1. 1.

Raghav Jaju , Parth Karwa, ”Real-time Vernacular Sign Language Recognition
using MediaPipe and Machine Learning.”Published, 2023-12-03. doi:crossref

  2. 2.

Jungpil Shin 1, Akitaka Matsuoka , Md. Al Mehedi Hasan and Azmain Yakin
Srizon, ”American Sign Language Alphabet Recognition by Extracting Feature
from Hand Pose Estimation”Published: 31 August 2021 crossref

  3. 3.

Arun C. , R. Gopikakumari,” Optimisation of both classifier and fusion based
feature set for static American sign language recognition”crossref

  4. 4.

Shashank Salian, Indu Dokare , Dhiren Serai, Aditya Suresh , Pranav Ganorkar
”Proposed System for Sign Language Recognition” Published in: 2017
International Conference on Computation of Power, Energy Information and
Commuincation (ICCPEIC)crossref

  5. 5.

Zhang, F.; Bazarevsky, V.; Vakunov, A.; Tkachenka, A.; Sung, G.; Chang, C.L.;
Grundmann, M. Mediapipe hands: On-device real-time hand tracking. arXiv 2020,
arXiv:2006.10214.

  6. 6.

Brandon Garcia, Sigberto Alarcon Viesca ,”Real-time American Sign Language
Recognition with Convolutional Neural Networks ”crossref

  7. 7.

J. Nagi, F. Ducatelle, G. Di Caro, D. Ciresan, U. Meier, A. Giusti, F. Nagi,
J. Schmidhuber, and L. Gambardella. Max-pooling convolutional neural networks
for vision-based hand gesture recognition. In Signal and Image Processing
Applications (ICSIPA), 2011 IEEE International Conference on, pages 342–347,
Nov 2011.crossref

  8. 8.

M. Van den Bergh and L. Van Gool. Combining rgb and tof cameras for real-time
3d hand gesture interaction. In Appli cations of Computer Vision (WACV), 2011
IEEE Workshop on, pages 66–72, Jan 2011.crossref

  9. 9.

Kang, B., Tripathi, S., & Nguyen, T. Q. (2015, November). Real-time sign
language fingerspelling recognition using convolutional neural networks from
depth map. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)
(pp. 136-140). IEEE.

  10. 10.

Md. Moklesur Rahman , Md. Shafiqul Islam, Md. Hafizur Rahman , Roberto Sassi ,
Massimo W. Rivolta and Md Aktaruzzaman,”A New Benchmark on American Sign
Language Recognition using Convolutional Neural Network” Published in: 2019
International Conference on Sustainable Technologies for Industry 4.0 (STI)
crossref

  11. 11.

V. Bheda and D. Radpour, “Using deep convolutional networks for gesture
recognition in american sign language,” arXiv:1710.06836, 2017.

  12. 12.

A. Barczak, N. Reyes, M. Abastillas, A. Piccio, and T. Susnjak, “A new 2d
static hand gesture colour image dataset for asl gestures,” 2011.

  13. 13.

C.-L. Huang and W.-Y. Huang, “Sign language recognition using model based
tracking and a 3d hopfield neural network,” Machine vision and applications,
vol. 10, no. 5-6, pp. 292–307, 1998.

  14. 14.

T. Starner, J. Weaver, and A. Pentland, “Real-time american sign language
recognition using desk and wearable computer based video,” IEEE Transactions
on pattern analysis and machine intelligence, vol. 20, no. 12, pp. 1371–1375,
1998.

  15. 15.

H.-I. Suk, B.-K. Sin, and S.-W. Lee, “Hand gesture recognition based on
dynamic bayesian network framework,” Pattern recognition, vol. 43, no. 9, pp.
3059–3072, 2010.

  16. 16.

Rasha Amer Kadhim , Muntadher Khamees,” A Real-Time American Sign Language
Recognition System using Convolutional Neural Network for Real Datasets”,TEM
Journal. Volume 9, Issue 3, Pages 937-943, ISSN 2217-8309, DOI:
10.18421/TEM93-14, August 2020.

  17. 17.

Starner, T., Weaver, J., & Pentland, A. (1998). Real time american sign
language recognition using desk and wearable computer based video. IEEE
Transactions on pattern analysis and machine intelligence, 20(12), 1371-1375.

  18. 18.

Admasu, Y. F., & Raimond, K. (2010, November). Ethiopian sign language
recognition using Artificial Neural Network. In 2010 10th International
Conference on Intelligent Systems Design and Applications (pp. 995-1000).
IEEE.

  19. 19.

Rahib H.Abiyev, Murat Arslan and John Bush Idoko1,”Sign Language Translation
Using Deep Convolutional Neural Networks ”,Ksii Transactions on Internet and
Information systems VOL. 14, NO. 2, Feb. 2020

  20. 20.

Akshit Tayade and A. Halder, “Real-time Vernacular Sign Language Recognition
using MediaPipe and Machine Learning.” Unpublished, 2021. doi:
10.13140/RG.2.2.32364.03203.

  21. 21.

ASL Alphabet crossref

  22. 22.

B Sundar,T Bagyammal, ”AmericanSignLanguageRecognitionforAlphabetsUsing
MediaPipeandLSTM”,4th International Conferenceon Innovative Data Communication
Technology and Applicationcrossref

  23. 23.

Fan Zhang ,Valentin Bazarevsky ,Andrey Vakunov ,Andrei Tkachenka ,George Sung,
Chuo-Ling Chang, Matthias Grundmann. Google Re search 1600 Amphitheatre Pkwy,
Mountain View, CA 94043, USA.

  24. 24.

Arsheldy Alvin1, Nabila Husna Shabrina2, Aurelius Ryo3, Edgar Christian4
Fakultas Teknik dan Informatika, Universitas Multimedia Nusan tara, Teknik
Komputer Tangerang, Indonesia. Hand Gesture Detection for American
SignLanguage using K- Nearest Neighbor with Mediapipe

  25. 25.

Gonzalez, R. C., & Woods, R. E. (2008). Digital Image Processing (2nd ed.).
Prentice Hall.

  26. 26.

LeCun, Y., Bottou, L., Orr, G. B., & M¨ uller, K. R. (2002). Efficient
backprop. In Neural networks: Tricks of the trade (pp. 9-50). Berlin,
Heidelberg: Springer Berlin Heidelberg.

  27. 27.

A. F. Agarap, “Deep learning using rectified linear units (relu),”
arXiv:1803.08375, 2018.

  28. 28.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with
deep convolutional neural networks,” in Advances in neural infor mation
processing systems, 2012, pp. 1097–1105.

  29. 29.

S. Park and N. Kwak, “Analysis on the dropout effect in convolutional neural
networks,” in Asian Conference on Computer Vision. Springer, 2016, pp.
189–204.

  30. 30.

B. Ko, H.-G. Kim, K.-J. Oh, and H.-J. Choi, “Controlled dropout: A different
approach to using dropout on deep neural network,” in Big Data and Smart
Computing (BigComp), 2017 IEEE International Conference on. IEEE, 2017, pp.
358–362.

  31. 31.

Perez, L., & Wang, J. (2017). The effectiveness of data augmentation in image
classification using deep learn ing. arXiv preprint arXiv:1712.04621.

  32. 32.

Teak-Wei ChongBoon Giin, LeeBoon Giin Lee, ”American Sign Language Recognition
Using Leap Motion Controller with Machine Learning Approach”,October
2018Sensors 18(10):3554 DOI: 10.3390/s18103554

  33. 33.

JEREMY JORDAN,”Common architectures in convolutional neural networks”crossref