# Hand Sign to Bangla Speech: A Deep Learning in Vision based system for
Recognizing Hand Sign Digits and Generating Bangla Speech

Shahjalal Ahmed1, Md. Rafiqul Islam2, Jahid Hassan3, Minhaz Uddin Ahmed4,
Bilkis Jamal Ferdosi5, Sanjay Saha6, Md. Shopon7  Department of Computer
Science and Engineering, University of Asia Pacific
Dhaka, Bangladesh
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent advancements in the field of computer vision with the help of deep
neural networks have led us to explore and develop many existing challenges
that were once unattended due to the lack of necessary technologies. Hand
Sign/Gesture Recognition is one of the significant areas where the deep neural
network is making a substantial impact. In the last few years, a large number
of researches has been conducted to recognize hand signs and hand gestures,
which we aim to extend to our mother-tongue, Bangla (also known as Bengali).
The primary goal of our work is to make an automated tool to aid the people
who are unable to speak. We developed a system that automatically detects hand
sign based digits and speaks out the result in Bangla language. According to
the report of the World Health Organization (WHO), 15% of people in the world
live with some kind of disabilities. Among them, individuals with
communication impairment such as speech disabilities experience substantial
barrier in social interaction. The proposed system can be invaluable to
mitigate such a barrier. The core of the system is built with a deep learning
model which is based on convolutional neural networks (CNN). The model
classifies hand sign based digits with 92% accuracy over validation data which
ensures it a highly trustworthy system. Upon classification of the digits, the
resulting output is fed to the text to speech engine and the translator unit
eventually which generates audio output in Bangla language. A web application
to demonstrate our tool is available at http://bit.ly/signdigits2banglaspeech.

###### Index Terms:

Hand sign recognition, Computer Vision, Deep Learning, Convolutional Neural
Network

##  I Introduction

Hand sign language and hand gestures are one of the most common methods for
communication. People with difficulties to speak rely on this technique most.
People without such disabilities also use sign language and hand gesture to
express their emotions or to communicate with each other. Hand gestures and
sign language involve the use of the hands, along with other parts of the
body. Sign language is a system of communication using hand gestures and
signs. Hence, hand gestures and hand sign language are significantly
integrated into our methods of communication. In this paper, we propose an
automated system for recognizing sign language with the hand gesture and
translating it into Bangla speech. This tool is basically for the people who
have to communicate using hand signs with those who do not know how to
interpret hand signs (for both English and Bangla listeners/readers). It can
also be used to interact with a computer using sign language. As a secondary
focus, this tool can also be applied to hand gesture recognition related
systems with little modifications. As an initial stride, we have focused on
hand based hand signs, especially digits.

For the last few decades, several approaches to human computer interaction
(HCI) has been proposed as an alternative to the traditional input devices
such as a keyboard or a mouse. However, the lack of intuitiveness of the new
techniques hinders the replacement of the old ones. For communication or to
interact with their environment human mostly used their hands. Thus, hand
gestures can play a significant role in human-computer interaction and hand
gesture based methods can stand out in providing a natural way of interaction
and communication.

With these possible rooms of improvisations in mind, we tried to build a
system for the vocally challenged people who want to utter their feeling
through hand-signs and in Bangla language. Hence, the main contribution of
this paper is a vision-based system based on deep neural networks for
recognizing hand sign-based digits and translating them into Bangla speech. In
our proposed system, at first, the hand sign is recognized by our deep
learning model and produce digits in a text. The deep-learning model is a
convolutional neural network, tested with different combinations of hyper-
parameters to get to the optimal results with minimal training time. After
recognizing the digits from the given input images of hand-signs, the tool
translates the predicted result to Bangla with the aid of Google Translator
API (Application Programming Interface). Then by Google Text to Speech API, we
convert this result to Bangla speech.

(a) Dataset from UNIPD Website

(b) Dataset created by the authors

(c) from UNIPD after cropping and re-sizing

(d) By authors after cropping and re-sizing

Figure 1: Snapshots from the datasets.

##  II Literature Survey

From the early era of computer vision related applications, automated hand-
sign and hand-gesture recognition has gained significant focus from
researchers. Prior to the advancement of deep neural networks in the field of
computer vision, traditional approaches were used to classify hand-signs.
However, as deep learning gained its popularity due to advanced computing
devices with highly powerful computation capabilities, it has been used to
solve many problems in computer vision as well. The use of neural networks in
computer vision is increasing even more as the results from deep-learning
based systems are outperforming traditional systems by notable margins. We
discuss some recent deep neural network based approaches by researchers to
classify hand-sign for different languages including Bangla.

Some major researches in this problem domain can be found in some recent
survey papers. One such survey[1] was conducted by Rautaray et al. which is a
very comprehensive analysis of studies and techniques applied for solving
problems related to our target area in recognizing hand-sign languages. Their
work also cover HCI based hand-gesture recognition as well.

In the last few years, many hand-gesture recognition techniques have been
proposed and many systems have been developed for different sign languages all
over the world. Rahman et al.[2] work on hand-sign recognition for Bengali
Sign Language to detect some of the Bengali Vowels and Consonants. They have
used Haar-like features and K-Nearest Neighbors Classifier to build their
model. This solution is better for only Bangla focused environment however, it
doesn’t scale to other languages. Another recent work[3] on recognizing Bangla
words and sentences were done by Santa et at. where they propose a real-time
system to classify words and sentences from live video images. Their work was
significant for the same targeted population as us, however, number of words
and sentences were fixed and small in number in their approach which would be
hard to scale as well.

A sign language recognition system[4] was developed by Kadam et al. for
American Sign Language (ASL) based on hand gesture using glove. For Indian
sign language, a very promising system[5] was proposed by Washef et al. where
their proposed system analyzes variations in the hand locations along with the
centre of the face. ‘Gesture Audio Video Conferencing Application’ for
communication between a deaf and dumb person and a Normal Person without any
special Sign Language Interpreter which is proposed by Tarte et al[6]. Starner
et al.[7] developed two real-time hidden Markov model-based systems for
recognizing sentence-level continuous American Sign Language (ASL). Their
first system used a desk mounted camera which is actually second-person
viewpoint for observing the signer’s hand and achieves 92 percent word
accuracy using a 40 word lexicon and the second system mounts the camera in a
cap worn by the user for observing his/her own hand gesture form first-person
viewpoint and achieves 98 percent word accuracy using a 40 word lexicon.

Other attempts on gesture or sign detection played different roles in this
field by limiting the conditions. A timely example can be the case of color
based hand detection which become possible because of the uniformity of skin
color. But hands need to differentiate from other skin colored object. This
approaches needs proper conditions like motion sensor or high resolution
cameras that catches every short details that makes it more reliable and
robust.

##  III Background Studies

There are mainly three types of approaches in the research of hand gesture
analysis: vision based analysis, glove based analysis, and analysis of drawing
gestures. Vision based analysis of hand gestures recognition techniques are
comprises of three main phases:

  * •

Tracking

  * •

Detection

  * •

Recognition

###  III-A Detection

Detection is the key step of hand gesture recognition system. Some
preprocessing and segmentation are needed primarily on the raw image data.
Thus it can figure out the necessary features from a random background. Then
the data are prepared for the further steps. Some major steps applied in hand
detection are -

  * •

Skin color based Threshold Calculation

  * •

Binarization

  * •

Background Removal

###  III-B Tracking

Tracking is the technique to read the position and movement of hand[8]. In
case of robust tracking, the significance of accurately tracking is highly
important. First one is to interconnect the features of frames with finger or
hand, thus get a trajectory that contains the information which can be use
afterward. And the second advantage is for model based system. In model based
system tracking can be helpful to limit the parameters and essential features.

###  III-C Recognition

The main work of vision based hand-sign recognition is find out the meaning of
specific hands-signs from the position of hand by classification. There are
two types of recognition techniques in the research of hand gesture: Static
hand gestures and Dynamic hand gestures. Static hand gestures mean stable
symbols in a time interval, in other words a symbol without any movement of
hand. On the other hand dynamic hand gestures includes motion of body parts
like waving of hand.

##  IV Prepossessing

Figure 2: Snapshots from few Augmented Images Figure 3: Architecture of Deep
Learning CNN Model in our proposed system

###  IV-A Collection of dataset

Our deep-learning model is trained and tested on a dataset which is a
combination of two datasets. One of those two datasets are collected from an
online source and the other dataset was collected by the authors. In our
experiments, we have used 3200 images in total, among which 1200 images are
from UNIP (http://lttm.dei.unipd.it/downloads/gesture/) and the other 2000
images are generated by us which is accessible at
http://bit.ly/sign_digits_dataset. The UNIPD dataset gives us the hand-sign
images with random backgrounds like room environment with different lighting
conditions. On the other hand, part of the dataset that we collected ourselves
are the images taken in a more focused environment for the system. The goal
for doing such is to train the model to learn hand-signs with both kind of
setups (arbitrary backgrounds and more application-friendly setup). A snapshot
of few images from the dataset is given in Figure 1.

###  IV-B Cropping and re-sizing dataset

From a raw image of a hand-sign with the natural backgrounds, we cropped out
the hand part and discarded all the other unnecessary elements in the image.
To do this, we applied the Python library named OpenCV. We applied this
operation on our whole dataset and also to the new images to be classified.
Although for HSV values and other corresponding features, the cropping was not
exactly 100% perfect but still it was able to crop most of the images
accurately. After getting the cropped out images, they were re-sized to
$32\times 32$ pixels. All the images were converted into gray scale before
feeding into the model.

###  IV-C Splitting dataset

We construct the dataset with 10 signs for each of the 10 digits. For each
sign there are 320 images. The dataset is divided into two sets, one for
training and the other for testing. We use a ratio of $80:20$ of train to test
dataset sizes. To enumerate in terms of each signs, for every sign there are
250 images in training set and other 70 images are in test set.

###  IV-D Augmentation

Real life data always tend to be random and comes in various types of
distortions like rotation, shifting, flipped images and so on. To emphasize
the training set with such scenarios, we used image augmentation techniques.
Here, the images are randomly rotated 0 degree to 30 degrees. Some images were
randomly sheared in a range of 0.2 degree and some images were horizontally
flipped. These augmentations eventually led to slightly less accurate
predictions, however, our system shows promising outcomes even for the highly
augmented images. A snapshot of the augmented images is given in Figure 2.

##  V Deep-learning Model

Figure 4: Flow diagram of System

###  V-A Convolutional Neural Network

Convolutional Neural Network (CNN) a special form of Artificial Neural Network
(ANN) where it contains repeated sets of neurons which are applied across the
space of an image. These sets of neuron are referred to as 2D Convolutional
Kernels, repeatedly applied over all the patches of an image. This helps to
learn various meaningful features from every subspace of the whole space
(image). The reason why CNN is chosen for image classification, is to exploit
spatial or temporal invariance in recognition. For the effectiveness of CNN,
it outperformed previously used techniques, such as Support Vector
Machine(SVM), K-Nearest Neighbors (K-NN) in classifying and learning from
images. Two major components of CNN that differ from regular neural networks
are:

  * •

Convolution Layer

  * •

Pooling Layer

(a) Before data augmentation

(b) After data augmentation

Figure 5: Loss and Accuracy Curves (Training and Validation)

A simple CNN is a sequence of layers, and every layer of a CNN transforms one
volume of activation values to another through a differentiable function. The
first layer is the Convolution layer. It could consist of one or more than one
kernels. It gives us a output passed through a non-linearities i.e. Rectified
Linear Unit(ReLu).

Pooling layer is the way to reduce the spatial dimension. Pooling layer adds
the spatial invariance to CNN model thus it avoids overfitting. There are
several kind of pooling such as Max pooling, Min pooling, Sum pooling etc.

Fully Connected layer is the last steps of CNN and it connects the CNN to the
output layer. The expected number of outputs constructed in this layer.

###  V-B Proposed Model

In our CNN model there are two convolution layers followed by $2\times 2$ max
pooling layer for each convolution layer. The first convolution layer has 32
kernels of size $3\times 3$ and the second convolution layer has 64 kernels of
the same size. We used 25% and 50% dropout regularization respectively for
each convolution and pooling layer pair, where it means one in every 4 and 2
inputs respectively will be excluded from each update cycle. We used ReLu and
Softmax as activation function in our fully connected layer. A diagram of the
model in our proposed system in given in Figure 3.

###  V-C Training phase

The model was trained for $100$ epochs with RMSProp optimizer and using
Categorical Cross Entropy as the cost function. The model converged well
before $100$ epochs and the weights were saved along with the model to make
use of for the next phases.

###  V-D Test phase

After the training phase the model provided us a promising test accuracy for
our test set with a low loss rate. However for the augmented images, the loss
rate was reduced significantly where the accuracy was almost same. For every
new image in the execution phase, it is preprocessed before feeding into our
model. The model gives us a vector of size ten of binary values where only one
of the ten values will be $1$ where as others will be $0$ denoting the
predicted class value of the given data.

##  VI Classified Text to Bangla speech

After the structure of our model, now we introduce the signature step of our
work of converting a hand-sign digit in to a speech in Bangla. There are two
steps in this phase.

The first step is to translate the recognized hand-sign in to Bangla text and
this step is done using the Translator API of Google Translator. Upon giving
an input text in English, the API provides with the translated Bangla text
from the English text.

Then we convert the resultant Bangla text to Bangla speech with the help of
gTTS (Google Text to Speech) API. Although these could be done in one step by
only using gTTS. However, we insert the intermediate step of translator to
give our work more compatibility.

##  VII Result and Analysis

TABLE I: training loss and validation loss along with training  | Validation loss | Validation Accuracy
---|---|---
Without Augmentation | 0.37 | 89%
With Augmentation | 0.31 | 92%
TABLE II: Confusion Matrices — Ac: Actual Class and Pr: Predicted Class

(a) Without image augmentation  Ac Pr | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
---|---|---|---|---|---|---|---|---|---|---
0 | 64 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 3
1 | 0 | 65 | 4 | 0 | 0 | 0 | 1 | 0 | 0 | 0
2 | 0 | 1 | 60 | 1 | 1 | 0 | 0 | 0 | 5 | 2
3 | 0 | 0 | 1 | 56 | 1 | 6 | 3 | 0 | 2 | 1
4 | 0 | 0 | 1 | 0 | 68 | 0 | 0 | 0 | 0 | 1
5 | 0 | 0 | 1 | 2 | 2 | 60 | 0 | 1 | 4 | 0
6 | 1 | 1 | 0 | 1 | 1 | 0 | 61 | 2 | 1 | 2
7 | 2 | 0 | 0 | 0 | 0 | 1 | 1 | 65 | 1 | 0
8 | 0 | 1 | 4 | 0 | 1 | 0 | 0 | 0 | 64 | 0
9 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 67

(b) With image augmentation Ac Pr | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
---|---|---|---|---|---|---|---|---|---|---
0 | 60 | 6 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 3
1 | 0 | 67 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2
2 | 0 | 1 | 61 | 0 | 2 | 2 | 0 | 0 | 4 | 0
3 | 0 | 0 | 0 | 60 | 4 | 2 | 3 | 0 | 0 | 1
4 | 0 | 0 | 0 | 1 | 65 | 2 | 0 | 1 | 0 | 1
5 | 0 | 1 | 0 | 0 | 0 | 68 | 0 | 0 | 1 | 0
6 | 1 | 1 | 0 | 2 | 3 | 0 | 61 | 1 | 0 | 1
7 | 0 | 1 | 0 | 0 | 0 | 0 | 4 | 65 | 0 | 0
8 | 0 | 1 | 2 | 0 | 3 | 3 | 0 | 0 | 59 | 2
9 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 68

Figure 6: ROC curve for the 10 classes

We built our model with different combinations of hyper-parameters to get the
best possible outcome. For 1 convolution layer with $32$ features and $0.25$
dropout value it we achieved $89\%$ accuracy on test set where the validation
loss was $0.37$. After adding another convolution layer with $64$ features and
$0.5$ dropout value the accuracy rate increased to $92\%$. Notable fact is
without image augmentation the validation loss result was incredibly reduced
from $0.37$ to $0.31$. Where the accuracy was increased from $89\%$ to $92\%$
as shown in table I. However, adding more layers results to overfitting which
is why no more convolution layers were added to the model afterwards.

The curve of training loss and validation loss along with training accuracy
and validation accuracy before and after image augmentation shown in Figure 5.
Before augmentation the validation accuracy graph remains below the training
accuracy because here the training accuracy is too high. And the loss graph
for training set is almost zero whereas the validation loss is high. But after
augmentation the training accuracy and validation loss decreases.

The ROC (Receiver operating Characteristic) curve is the graphical interface
that determines the diagnostic ability of a classifier. In our case the ROC
curve is for 10 classes and we followed the one vs all approach. We created 10
ROC curves and for each class we take that class as true positive and all the
rest class jointly as the false negative class. So it maps like -

  * •

Class 0 vs Classes (1-9)

  * •

Class 1 vs Classes (0 & 2-9)

  * •

Class 2 vs Classes (0,1 & 3-9) and so on.

The ROC graph for our model shown in Figure 6.

Confusion matrix shows the total picture of how the model performed in terms
to number of correct / wrong classifications made. The confusion matrices of
the test predictions before image augmentation and after image augmentation
are shown in Table II.

##  VIII Conclusion

Our main motive in this paper is to develop a complete tool for the people
with speaking disabilities and to facilitate them to communicate with Bangla
speech irrespective of the barrier of sign language. This tool can also be
implemented in hand-gesture based HCI scenarios as well. As an initial
footstep, we built the tool to successfully recognize hand digits with near
perfect accuracy and speak out in Bangla. However, there are room for
improvements in the tool which we opt for addressing soon. For example, taking
advantage of more accurate hand structure capturing device like Leap Motion or
Xbox Kinect can immensely improve the quality of the tool. Nonetheless, the
proposed model and the tool in the paper addresses two very essential yet
underestimated social problems and provides a solution to the problems with a
highly trustworthy and robust system.

## References

  * [1] S. S. Rautaray and A. Agrawal, “Vision based hand gesture recognition for human computer interaction: a survey,” _Artificial Intelligence Review_ , vol. 43, no. 1, pp. 1–54, 2015.
  * [2] M. A. Rahaman, M. Jasim, M. H. Ali, and M. Hasanuzzaman, “Real-time computer vision-based bengali sign language recognition,” in _2014 17th International Conference on Computer and Information Technology (ICCIT)_ , Dec 2014, pp. 192–197.
  * [3] U. Santa, F. Tazreen, and S. A. Chowdhury, “Bangladeshi hand sign language recognition from video,” in _2017 20th International Conference of Computer and Information Technology (ICCIT)_ , Dec 2017, pp. 1–4.
  * [4] K. Kadam, R. Ganu, A. Bhosekar, and S. Joshi, “American sign language interpreter,” in _Technology for Education (T4E), 2012 IEEE Fourth International Conference on_. IEEE, 2012, pp. 157–159.
  * [5] W. Ahmed, K. Chanda, and S. Mitra, “Vision based hand gesture recognition using dynamic time warping for indian sign language,” in _2016 International Conference on Information Science (ICIS)_ , Aug 2016, pp. 120–125.
  * [6] N. T. Tarte, S. N. Bhadane, and P. S. Kulkarni, “A gesture audio-video conferencing application for the ease of communication between normal person and deaf and dumb person,” _International Journal of Science, Engineering and Technology Research (IJSETR)_ , vol. 3, no. 5, pp. 1487–1490, 2014\.
  * [7] T. Starner, J. Weaver, and A. Pentland, “Real-time american sign language recognition using desk and wearable computer based video,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 20, no. 12, pp. 1371–1375, 1998.
  * [8] N. Ç. Kılıboz and U. Güdükbay, “A hand gesture recognition technique for human–computer interaction,” _Journal of Visual Communication and Image Representation_ , vol. 28, pp. 97–104, 2015.


T^{-2}\sum_{t=1}^{T}\bm{\zeta}_{2,t-1}u_{t}\overset{d}{\to}\int_{0}^{1}\bar{\bm{B}}_{2}(s)^{\prime}d\bm{B}_{u}(s);$
$\displaystyle(g)\quad
T^{-2}\sum_{t=1}^{T}\bm{\zeta}_{2,t}\bm{\zeta}_{0,t}^{\prime}\overset{d}{\to}\int_{0}^{1}\bar{\bm{B}}_{2}\bm{B}_{0}(s)^{\prime}$
$\displaystyle(h)\quad
T^{-3}\sum_{t=1}^{T}\bm{\zeta}_{2,t}\bm{\zeta}_{1,t}^{\prime}\overset{d}{\to}\int_{0}^{1}\bar{\bm{B}}_{2}\bm{B}_{1}(s)^{\prime}$
$\displaystyle(i)\quad
T^{-4}\sum_{t=1}^{T}\bm{\zeta}_{2,t}\bm{\zeta}_{3,t}^{\prime}\overset{d}{\to}\int_{0}^{1}\bar{\bm{B}}_{2}\bar{\bm{B}}_{2}(s)^{\prime}$

Using these results, we can now determine the appropriate rates in Assumption
3. Let $\eta_{T}$ denotes a sequence such that $\eta_{T}\to\infty$ at an
arbitrarily slow rate. First, it follows from (d), (f) and (h) that
$\phi_{T,01}=\phi_{T,02}=\eta_{T}T^{-1/2}$. Next, we may directly deduce from
(a) that $\kappa_{T,0}=\eta_{T}^{-1}$. Similarly, it follows from (e), (g),
(h) and (i) that $\kappa_{T,12}=\eta_{T}^{-1}$. This was shown for
$\kappa_{T,1}$ in Phillips and Hansen (1990), but the same reasoning can be
applied to the whole matrix. It follows from (b), (c) and (f) that
$\gamma_{T,i}=\eta_{T}$ for $i=0,1,2$, and finally result (d) of Assumption 3
follows from (a).

It then remains to check the rates. As we chose $\eta_{T}$ growing arbitrarily
slowly, it follows that $\eta_{T}^{3}T^{-1/2}\to 0$. We then have that

  1. 1.

$\kappa_{T,0}-(\phi_{T,01}^{2}+\phi_{T,02}^{2})/\kappa_{T,0}=\eta_{T}^{-1}-2\eta_{T}^{3}T^{-1}\geq\eta_{T}^{-1}/2=\kappa_{T,-0}$;

  2. 2.

$\phi_{T,01}+\phi_{T,02}+2(\phi_{T,01}^{2}+\phi_{T,02}^{2})/\kappa_{T,0}=2\eta_{T}T^{-1/2}+2\eta_{T}^{3}T^{-1}\leq
2\eta_{T}T^{-1/2}=\phi_{T,-0}$;

  3. 3.

$\gamma_{T,u,1}+\gamma_{T,u,2}+(\phi_{T,01}+\phi_{T,02})\gamma_{T,u,0}/\kappa_{T,0}=2\eta_{T}+2\eta^{3}T^{-1/2}\leq
2\eta_{T}=\gamma_{T,u}$;

  4. 4.

$\phi_{T,-0}(\phi_{T,-0}+\gamma_{T,u})/\kappa_{T,-0}\leq\eta_{T}^{3}T^{-1/2}\leq\delta_{T}$.

We complete this verification by noting that the final result simply follows
from result (b).

##  Appendix C Additional Empirical Analysis

Here we extend the analysis of Section 6 using the latest vintage of FRED-MD
at the moment of writing, namely that of September $2022$ (453 observations).
As mentioned, this vintage does include all latest crises too and VXOCLSx is
replaced by VIXCLSx.

Figure C.1: PDS-LA-LM, Causes of VIXCLSx, $\alpha=0.01$

Figure C.2: PDS-LA-LM, Caused by VIXCLSx, $\alpha=0.01$

Interestingly, a comparison of Figure 3 and Figure C.2 shows how all the
connections FRED$\rightarrow$VXOCLSx in the November 2019 FRED-MD vintage are
found also in the FRED$\rightarrow$VIXCLSx October 2022 vintage plus two more:
IPFUELS and OILPRICEx. This tells us two things: first, the change from
VXOCLSx to VIXCLSx has not changed the causal infrastructure among the FRED-MD
variables. This is sensible as VIXCLSx has been built from VXOCLSx and
precisely with the aim to replace it. Second, the various global crises
occurred between 2019 and 2022 have also not impacted the amount of insourcing
Granger causal connections onto VIXCLSx but have increased substantially more
the outsourcing amount of connections, as clear from Figure C.2. From 41
connections of VXOCLSx$\rightarrow$FRED in Figure 3 for the 2019 vintage, the
amount of connections increased in the latest vintage to a staggering 65
connections (at $1\%$ significance level). Therefore, this contributes to the
evidence that latest crises have increased substantially the perceived
economic uncertainty and such uncertainty is outpouring towards a large number
of macroeconomic variables.

Figure C.3: PDS-LA-LM, Causes of US-EPU, $\alpha=0.01$

Figure C.4: PDS-LA-LM, Caused by US-EPU, $\alpha=0.01$


# FaxPlainAC: A Fact-Checking Tool Based on EXPLAINable Models with HumAn
Correction in the Loop

Zijian Zhang<EMAIL_ADDRESS>0000-0001-9000-4678 L3S Research CenterAppel Str.
9aHannoverNiedersachsenGermany30167 ,  Koustav Rudra<EMAIL_ADDRESS>L3S
Research CenterAppel Str. 9aHannoverNiedersachsenGermany30167 and  Avishek
Anand<EMAIL_ADDRESS>L3S Research CenterAppel Str.
4HannoverNiedersachsenGermany30167

(2021)

###### Abstract.

Fact-checking on the Web has become the main mechanism through which we detect
the credibility of the news or information. Existing fact-checkers verify the
authenticity of the information (support or refute the claim) based on
secondary sources of information. However, existing approaches do not consider
the problem of model updates due to constantly increasing training data due to
user feedback. It is therefore important to conduct user studies to correct
models’ inference biases and improve the model in a life-long learning manner
in the future according to the user feedback. In this paper, we present
FaxPlainAC, a tool that gathers user feedback on the output of explainable
fact-checking models. FaxPlainAC outputs both the model decision, i.e.,
whether the input fact is true or not, along with the supporting/refuting
evidence considered by the model. Additionally, FaxPlainAC allows for
accepting user feedback both on the prediction and explanation. Developed in
Python, FaxPlainAC is designed as a modular and easily deployable tool. It can
be integrated with other downstream tasks and allowing for fact-checking human
annotation gathering and life-long learning.

interpretable machine learning; data gathering; fact-checking; human-in-the-
loop machine learning

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Information and Knowledge Management;
November 1–5, 2021; Virtual Event, QLD, Australia††booktitle: Proceedings of
the 30th ACM International Conference on Information and Knowledge Management
(CIKM ’21), November 1–5, 2021, Virtual Event, QLD, Australia††price:
15.00††doi: 10.1145/3459637.3481985††isbn: 978-1-4503-8446-9/21/11††ccs:
Human-centered computing User studies††ccs: Computing methodologies Learning
from critiques††ccs: Computing methodologies Neural networks

##  1\. Introduction

Massive amounts of information is produced daily on the Web and social media
(Holzmann et al., 2016). However, not all of it is credible, and it is usually
beyond the human scope to verify each one manually. Sometimes, non-credible
information inadvertently gets propagated by even popular personalities and
manages to reach a large audience. In recent times, researchers proposed
different fact-checking algorithms (Kotonya and Toni, 2020; Zhang et al.,
2021) and try to automatically flag inaccurate content. However, most of these
models are obscured in nature and rely on the explicit supervised data. Even
for the models that indeed predict the correctness of claims, a general lack
of interpretability inhibits general trust over its utility in general.

Figure 1. Architecture of FaxPlainAC.

Towards this, there have been some recent proposals that create models that
are interpretable-by-design approach that generate an explanation alongwith
the actual fact-checking prediction (Lei et al., 2016; Bastings et al., 2019;
Zhang et al., 2021). Specifically, these approaches not only perform fact-
checking on a piece of information but also output the evidence/reasons behind
the model’s decision in terms of an extractive piece of text with the end-
users. Moreover, approaches like Quin+(Samarinas et al., 2021) provides a web
front-end to present not only verification of the user’s claim but also the
evidence from the web supporting the verification. In such approaches, the
interpretable-by-design models serve as the engine of the fact-checking
system. Such interpretable-by-design models are, unfortunately, usually
trained on datasets with limited size like Fever(Thorne et al., 2018). Such
limited training data introduces a certain bias towards the used dataset. The
end-users, therefore, cannot fully trust the model’s prediction or
interpretation. They would also have demand to correct the model’s erroneous
prediction or explanation. A human intervention, fortunately, is easy to scale
up by conducting crowdsourcing tasks (Zhang et al., 2019).

In this paper, we present FaxPlainAC111https://faxplain.appspot.com/, the
first interpretable open-source fact-checking web front-end that gathers user
correction on model’s false prediction/interpretation. Our framework is
presented in Figure. 1. FaxPlainAC first retrieves documents potentially
related to the claim from online encyclopedias, followed by fact-checking
using the recent interpretable by design approach ExPred (Zhang et al., 2021).
The approach ExPred justifies (support/refute) the claim regarding each
document, together with provides evidence supporting the inference (details in
Section 2); and at last ask users feedback, or correction by false results,
about the model’s prediction and interpretation. FaxPlainAC is implemented in
Python with the objective to assist users with supporting/refuting evidence
along with the decision of the model, together with gathering user annotation
in fact-checking task. At present, we incorporate ExPred only into our
framework. However, our modular design allows us to incorporate different
standalone interpretable fact-checking algorithms into FaxPlainAC, so that the
benefits of such proposed algorithms become accessible to different
stakeholders such as end-users, professional developers, tech companies, etc.
For example, one can easily use an interpretable search system (Singh and
Anand, 2019, 2020; Fernando et al., 2019), or use evidence-gathering
mechanisms (Singh et al., 2016b, a; Holzmann and Anand, 2016). We believe such
an application will be immensely helpful in developing new socio-technical
systems that rely on automated fact checking.

##  2\. ExPred: Method Background

This section explains the idea behind ExPred (Zhang et al., 2021) approach and
gives examples from the fact-checking domain. ExPred is a two-phase pipeline
interpretable by a design approach that incorporates the task prediction and
the rationales.

In the first phase, it learns to extract explanations/rationales from the
input document supervised by both task-specific and rationale-based signals
under a multi-task learning (MTL) framework. Both predictions and explanations
are learned using a common BERT(Devlin et al., 2019)-based encoder. However,
separate decoders are utilized for task and explanation extraction. The MTL
criteria are the linear combination of both loss for the prediction and
explanation task and represented as follows:

(1) $\mathcal{L}_{loss}=\mathcal{L}_{task}+\lambda\mathcal{L}_{exp}\mbox{,}$

where $\mathcal{L}_{task}$ is a standard binary cross-entropy loss and is the
loss of the auxiliary prediction task. The $\mathcal{L}_{exp}$ is the
explanation loss. Because there are far more non-evidence tokens than evidence
tokens in the explanation supervision, the explanation loss is a balanced
version of explanation loss, i.e.

(2)
$\mathcal{L}_{exp}=\frac{1}{|t_{0}|}\mathtt{BCE}(p_{0},t_{0})+\frac{1}{|t_{1}|}\mathtt{BCE}(p_{1},t_{1})\mbox{,}$

where $|t_{0}|$, $|t_{1}|$ are the count of the non-evidence/evidence tokens,
correspondingly, while $\mathtt{BCE}(p_{0},t_{0})$,
$\mathtt{BCE}(p_{1},t_{1})$ are the cross entropy for all non-
evidence/evidence tokens.

In the second phase, a separate over-parameterized model is trained for the
prediction task only based on the extracted explanations of the first phase.
After training the first-phase MTL model, we do inference on the
training/validation data once again to emit machine annotated rationales. Then
we screen out all such instances, on which the auxiliary prediction branch
fails to predict the correct labels. The screening prevents the second-phase
model from being polluted by untrustworthy pieces of evidence. Then for the
instances left, all tokens corresponding to evidence identified by the phase 1
model are preserved and all non-evidence tokens are replaced by a wildcard
token (here we use period ’.’). Then the second-phase classifier is trained on
the masked dataset. This scheme is referred to as predict and explaining
(first phase) and then predicting again (second phase). Fig. 2 describes the
overall architecture of ExPred. For example, we want to fact check Emma Watson
was killed in 1990 based on the document Emma Charlotte Duerre Watson (born 15
April 1990) is a French-British actress, model, and activist. Born in Paris …
previously. Watson appeared in all eight Harry Potter films from 2001 to 2011,
earning worldwide fame, critical accolades, and around $60 million... The
first sentence Watson … (born 15 April 1990) …’ suffices to refute the claim.

Figure 2. Overview of ExPred. Explanation generation supervised by task
objectives and explanation generation. Here auxiliary task is the same as the
actual task (the Task Prediction on the right).

In the initialization phase of FaxPlainAC, an ExPred model is pre-trained on
the FEVER dataset(Thorne et al., 2018). In the inference phase, we concatenate
the user input statement and the document following the same manner as
presented in (Zhang et al., 2021), namely [CLS] statement [SEP] document. The
model outputs both justification and explanation.

Figure 3. Prediction results

##  3\. FaxPlainAC as an ExPred Plugin

In this section, we describe the architecture of FaxPlainAC and the training
procedure of ExPred. The web application part is implemented using Python3
with the Flask222https://flask.palletsprojects.com/ framework. ExPred model is
firstly implemented using Pytorch333https://pytorch.org/ 1.8.1 and trained on
an Nvidia A100. Then the trained model is loaded through the CPU in the
Dockers for inference. Figure 1 presents the overall architecture of
FaxPlainAC. This decoupled design guarantees our ExPred model can be
interchangeable with other interpretable fact-checking engines. The step-wise
usage instruction is as follows:

###  3.1. Overview of Search System

The FaxPlainAC architecture receives a query from the user. The task of the
system is to support or to refute the claim based on evidence extracted from
documents.

####  3.1.1. Firing Queries

Standard preprocessing steps (stripping, case folding, etc.) are applied to
the statement and the FaxPlainAC plugin issues a Google Programmable
Search444https://cse.google.com/cse/ to retrieve relevant Wikipedia pages for
the given claim through Google JSON
API555https://developers.google.com/custom-search/v1/introduction.

####  3.1.2. Pruning Search Results

The top-ranking search results are collected, and relevant content is
extracted from those pages with the help of
Mediawiki666https://www.mediawiki.org/wiki/API:Get_the_contents_of_a_page
RESTful tool. The end users can tell the system how many articles to be
returned. Specifically for our running example, we choose the top three
articles since they are the most relevant document for this user input.

###  3.2. Interpretable Fact-checking

Next, the ExPred model is applied over the statement and each of these three
documents. The model either SUPPORTS or REFUTES the claim, along with
providing evidence extracted from documents supporting the prediction. A
sample result page is shown in Figure 3 for the statement ‘Microsoft is a
Chinese company’. Here the first document is misleading for the justification
since it states companies in China instead of the company Microsoft. The
second seems irrelevant to our statement. And the third result states that
Microsoft is an American multinational company, which refutes the claim and of
which both the prediction and evidence are correct.

###  3.3. Collecting User Feedback

As mentioned previously, the pre-trained, well-performing model can be
erroneous during inference and possibly need to be re-trained on human
tutelage: The last column asks users for their feedback. The results agreed by
the users (choosing Yes in the column Agree?) are dumped as trustworthy
machine-generated evidence, while users being asked for their feedback about
the failing results by choosing No and clicking the Next step button. The
outcomes of our example corresponds to three different categories of user
feedback:

  * •

the first outcome is wrong because it is misleading for the justification. The
user can hence choose No for the column Agree? to report the falsely retrieved
document;

  * •

the second outcome is irrelevant to the nationality of Microsoft. Again the
user can choose No and report the irrelevance;

  * •

the third output is correct. The User can simply choose Yes and there is
nothing more to do to this one.

Note that a Wikipedia article is usually too long and might be cumbersome for
the users to read-through and later on fully annotate. Also it exceeds the
input length limit of the interpretable fact-checking model. Fortunately for
the majority of queries, the definition of the related topic is enough for
fact-checking. The definition of concepts usually occurs at the beginning of
its Wiki page. Hence, we take the first 30 sentences at the beginning of each
article by default and display highlighted evidences, together with those non-
evidence tokens, if they occurs at the beginning of the article or closely
after a piece of evidence. Additionally, a ”show more of this document” button
is placed for each article. If the user is sure the article is indeed relevant
to the query, but the evidence does not present in the selected sentences, the
user can acquire the next (e.g.) 30 sentences from the returned article and
update the corresponding outcome.

###  3.4. Token-level User Feedback

In the final step, we ask users for the following types of feedback on a
separate correction page as shown in Figure. 4:

  * •

If the corresponding document either SUPPORTS or REFUTES the statement, users
are asked to toggle up all evidence tokens supporting the judgment from the
document by highlighting them. The machine-annotated rationales are
highlighted by default. All highlighted tokens can be toggled down. Multiple
selections is also viable through pressing-holding and sweeping the mouse
through all evidence tokens.

  * •

Users can also choose if the document misleads the justification. For example,
the first result in our example incites a SUPPORTS result since the claim is
about companies in China. The user therefore selects This document is
misleading regarding my claim.

  * •

The document may be irrelevant (like the second one) to the given claim. In
the end, such feedback is saved into the user annotated evidence database by
clicking the Submit button. Those gathered human annotations are the final
output of our FaxPlainAC and can be used to fine-tune the ExPred model.

Figure 4. User feedback collection

From any of the pages mentioned above, the user can go back to the homepage
and verify another claim by clicking the ’Try another statement (or Go back to
the main page)’ button. If no results are retrieved from Wikipedia.org, an
error page pops out and asks the user to try other statements.

The whole application is at first encapsulated as a docker image and later on
deployed on the Gcloud run777https://cloud.google.com/run. The visualization
technique and cloud platform can also be replaced and tailored according to
the researcher’s demand.

##  4\. Conclusion

In this paper, we present FaxPlainAC, the first open-source framework for
gathering human annotations for interpretable fact-checking systems. We
leverage the recent interpretable-by-design model ExPred as the back-end model
of our fact checking-explanation-feedback web framework. Our decoupled design
enables easy integration with other interpretable search systems. Furthermore,
one can switch explainable models because of the modular design of the web
front-end and the models. The final contribution of our FaxPlainAC is two
folds: Firstly, it identifies errors models made on evidence identification or
prediction and secondly, it gathers human annotation as correction on the
erroneous results. Such human annotations can expand the existing fact-
checking datasets and help models to learn from failures.

Acknowledgement: Funding for this project was in part providedby the European
Union’s Horizon 2020 research and innovation program under grant agreement No
832921 and No 871042.

## References

  * (1)
  * Bastings et al. (2019) Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019.  Interpretable neural predictions with differentiable binary variables.  arXiv preprint arXiv:1905.08160 (2019).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  (2019), 4171–4186.
  * Fernando et al. (2019) Zeon Trevor Fernando, Jaspreet Singh, and Avishek Anand. 2019\.  A study on the Interpretability of Neural Retrieval Models using DeepSHAP. In Proceedings of SIGIR. 1005–1008.
  * Holzmann and Anand (2016) Helge Holzmann and Avishek Anand. 2016.  Tempas: Temporal archive search based on tags. In Proceedings of the 25th International Conference Companion on World Wide Web ’16. 207–210.
  * Holzmann et al. (2016) Helge Holzmann, Wolfgang Nejdl, and Avishek Anand. 2016\.  The Dawn of today’s popular domains: A study of the archived German Web over 18 years. In Proceedings of JCDL ’16. IEEE, 73–82.
  * Kotonya and Toni (2020) Neema Kotonya and Francesca Toni. 2020.  Explainable Automated Fact-Checking: A Survey. In Proceedings of COLING ’20. 5430–5443.
  * Lei et al. (2016) Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016.  Rationalizing neural predictions.  Proceedings of ACL ’16 (2016).
  * Samarinas et al. (2021) Chris Samarinas, Wynne Hsu, and Mong Li Lee. 2021.  Improving Evidence Retrieval for Automated Explainable Fact-Checking. In Proceedings of NAACL. 84–91.
  * Singh and Anand (2019) Jaspreet Singh and Avishek Anand. 2019.  Exs: Explainable search using local model agnostic interpretability. In Proceedings of the WSDM ’19. 770–773.
  * Singh and Anand (2020) Jaspreet Singh and Avishek Anand. 2020.  Model agnostic interpretability of rankers via intent modelling. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 618–628.
  * Singh et al. (2016a) Jaspreet Singh, Johannes Hoffart, and Avishek Anand. 2016a.  Discovering entities with just a little help from you. In Proceedings of CIKM ’16. 1331–1340.
  * Singh et al. (2016b) Jaspreet Singh, Wolfgang Nejdl, and Avishek Anand. 2016b.  Expedition: a time-aware exploratory search system designed for scholars. In Proceedings of SIGIR ’16. 1105–1108.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018.  FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of NAACL ’18. 809–819.
  * Zhang et al. (2021) Zijian Zhang, Koustav Rudra, and Avishek Anand. 2021\.  Explain and Predict, and Then Predict Again. In Proceedings of WSDM ’21. 418–426.
  * Zhang et al. (2019) Zijian Zhang, Jaspreet Singh, Ujwal Gadiraju, and Avishek Anand. 2019.  Dissonance between human and machine understanding.  Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–23.