# Precision Enhancement in Sustained Visual Attention Training Platforms:
Offline EEG Signal Analysis for Classifier Fine-Tuning
††thanks: This project was supported by the Rhode Island INBRE program from
the National Institute of General Medical Sciences of the National Institutes
of Health under grant number P20GM103430 and National Science Foundation under
award ID 2245558. Author 4 is the corresponding author.

1st Maryam Norouzi  Dept. of Electrical, Computer and Biomedical Engineering
University of Rhode Island
Kingston, RI, USA
<EMAIL_ADDRESS>2nd Mohammad Zaeri Amirani  Dept. of Electrical,
Computer and Biomedical Engineering
University of Rhode Island
Kingston, RI, USA
<EMAIL_ADDRESS>3rd Yalda Shahriari  Dept. of Electrical, Computer and
Biomedical Engineering
University of Rhode Island
Kingston, RI, USA
<EMAIL_ADDRESS>
4th Reza Abiri  Dept. of Electrical, Computer and Biomedical Engineering
University of Rhode Island
Kingston, RI, USA
<EMAIL_ADDRESS>

###### Abstract

In this study, a novel open-source brain-computer interface (BCI) platform was
developed to decode scalp electroencephalography (EEG) signals associated with
sustained attention. The EEG signal collection was conducted using a wireless
headset during a sustained visual attention task, where participants were
instructed to discriminate between composite images superimposed with scenes
and faces, responding only to the relevant subcategory while ignoring the
irrelevant ones. Seven volunteers participated in this experiment. The data
collected were subjected to analyses through event-related potential (ERP),
Hilbert Transform, and Wavelet Transform to extract temporal and spectral
features. For each participant, utilizing its extracted features, personalized
Support Vector Machine (SVM) and Random Forest (RF) models with tuned
hyperparameters were developed. The models aimed to decode the participant’s
attentional state towards the face and scene stimuli. The SVM models achieved
a higher average accuracy of 80% and an Area Under the Curve (AUC) of 0.86,
while the RF models showed an average accuracy of 78% and AUC of 0.8. This
work suggests potential applications for the evaluation of visual attention
and the development of closed-loop brainwave regulation systems in the future.

###### Index Terms:

Sustained Visual Attention, Brain-Computer Interface, Electroencephalography,
Composite Image, Classifier.

##  I Introduction

Attention is a fundamental aspect of human cognition and perception [1].
Traditionally, attentional states have been studied using blood-oxygen-level
dependent signals via functional magnetic resonance imaging [2], which,
despite its high spatial resolution, has limitations for real-time
neurofeedback due to the slow vascular response of the brain [3].In contrast,
electroencephalography (EEG), due to its higher temporal resolution is more
suitable for real-time brain-computer interface (BCI) applications [4]. EEG
has been widely used for attention evaluation and training, particularly in
children with attention deficit hyperactivity disorders [5]. Since face-like
visual stimuli undergo specialized processing in the human brain [6], face
images were employed in numerous brain studies, particularly studies on
attention [7]. Among them, the work of Sreenivasan et al. is notable [8]. They
employed event-related potential (ERP) analysis to show that attention to
faces in composite images with different transparency of face can modulate the
perceptual processing of faces. The other recent study explored the
spatiotemporal changes in EEG to classify perceptual states (faces/Gabors) and
analyzed the scope of attention (locally/globally) [9]; nevertheless, it did
not explored the EEG classification results for when subjects focused on both
image categories within a sequence of overlapped images. Based on our
investigation, most previous works focused on identifying participants’
attention levels without examining the visual stimuli that trigger them.
Recently, Abiri et al. focused on decoding EEG-based attentional states
towards faces and scenes. They developed a BCI platform using MATLAB,
incorporating a linear SVM classifier. However, this classifier did not
include subject-specific parameter tuning, which is particularly crucial for
real-time neurofeedback platforms[4]. To the best of the authors’ knowledge,
no studies have yet reported on the development and hyperparameter tuning of
non-linear ML classifiers to assess a participant’s sustained visual attention
and to identify the triggering visual stimulus. To address this, in this
study, we introduced an innovative wireless, portable, and open-source EEG-
based BCI software111https://github.com/AbiriLab/Neurofeedback-Based-BCI, to
explore subjects’ sustained visual attention levels toward two stimulus
categories—faces and scenes. We employed ERP, Morlet continuous Wavelet
Transform (MCWT), and Hilbert Transform (HT) analyses to discern distinctive
neural patterns and extract relevant features. Subsequently, to assess the
level of sustained visual attention in a participant, along with identifying
the specific visual stimulus responsible for capturing that attention, two
classifiers were developed, fine-tuned, and personalized using the extracted
features as input.

##  II Materials and methods

###  II-A EEG-Based BCI Platform: Recording Device and Interface

The BCI platform includes a wireless EEG headset, a workstation equipped with
dual monitors (experimenter and participant screens for control and stimuli),
and a novel developed Python-based software for simultaneous data acquisition,
analysis, and execution of a designed paradigm (Fig. 1). EEG signals were
acquired at 250 Hz using Unicorn Hybrid Black (g.tec medical engineering
GmbH), and transmitted to the PC via Bluetooth.The headset has 8 electrodes at
Fz, C3, Cz, C4, Pz, Po7, Oz, Po8 (10-20 system), plus two snap electrodes for
reference and ground behind the left and right ear, respectively [10].

Figure 1:  Developed BCI Platform (Clinical Trial: NCT05908253). Left: A
sample sequence of composite images during one block and the corresponding
expected responses. Right: Flowchart of the experiment and data analysis

###  II-B Experimental Protocol and Subjects

Seven healthy participants (4 males and 3 females) with a mean age of 28.14
years and a standard deviation of 10.84 took part in this study. All
participants had normal or corrected-to-normal vision. After the Institutional
Review Board (IRB) approval process, all participants gave written informed
consent to the experiment. The participants were asked to sit comfortably in a
fixed chair with one hand resting on their lap and another hand ready to press
a key to give behavioral responses. The participants were instructed to pay
attention to the images displayed on the monitor during the experiment and
limit their body movement. In the present work, we aimed to identify
participants’ attentional states toward two categories of images: face versus
scene; regardless of their subcategories. We assumed that neural response
contains common features for the subcategories within a single category [11].

In our study, we employed the Sustained Attention to Response Task (SART)
paradigm [12]. Our paradigm consists of eight blocks of trials with a
10-second respite between blocks. The experiment starts with a 3-minute EEG
recording for signal stabilization, followed by the blocks of trials in a
random order.(Fig. 1). Each block begins with a five-second texture cue, then
a 7-second grey image (baseline), followed by forty 1-second image stimuli
trials (activity trials). Considering all eight blocks, this paradigm results
in a total of 320 activity trials throughout the experiment. In our
experiment, four subcategories of indoor and outdoor scenes and male and
female face images were chosen as stimuli. The images were all black and white
with equal sizes, 800 × 1000 pixels. Face images were chosen to be neutral and
were centered inside the composite image. The indoor images were chosen from
interior scenes. Outdoor images were natural landscapes and cityscapes.
Brightness and contrast for all face and scene images were adjusted so that
the images have equalized contrast. Each trial includes a composite image with
equal proportions (50%) of the scene and face. (In our future neurofeedback
task, the proportions of images in the composite images will be regulated).
There was no repetition of face or scene images through each block of the
experiment, which helps to prevent any learning mechanism for the participant
[13]. Participants were asked to identify whether the shown image contained a
task-relevant/irrelevant image (e.g., an indoor/outdoor image). They were
asked to press the key on the keyboard for each recognized relevant image and
withhold their responses for irrelevant images. Following the SART paradigm,
which involves the withholding of key presses to rare targets, 90% of the
composite images contained images from the task-relevant subcategory. Each
participant underwent a total of approximately 20 minutes of experimentation,
including EEG cap fitting and electrode impedance checks.

###  II-C Signal Pre-processing

To do pre-processing, each EEG channel underwent a series of preprocessing
steps. Firstly, A 5th-order Butterworth bandpass filter isolated 0.4–40 Hz
frequencies. Subsequently, spikes were detected using the Median Absolute
Deviation technique and interpolated via a cubic spline, maintaining EEG data
integrity. Then, to minimize random noise a K-neighbors regression was used.
Next, baseline correction was implemented by subtracting the average of
baseline signals in each block from the activity signals of that block to
enhance task-specific brain response clarity. Finally, data was normalized
across experiment blocks using Z-scoring.

###  II-D Signal Processing, Feature Extraction, and Classification

Following our initial analysis, we identified several ERPs linked to sustained
visual attention. Additionally, the Hilbert Envelope (HE) revealed distinct
patterns for face and scene during sustained visual attention tasks. The HE
highlights temporal changes in EEG signal amplitude but not detailed
frequency-based oscillations. In contrast, the MCWT excels at extracting
transient features from non-stationary data, making it valuable for EEG data
analysis[14]. Combining MCWT with the HE offers a thorough approach essential
for robust attentional state modeling. So, in this study, ERP, Hilbert
envelop, and the MCWT were employed for the analysis of EEG data. For our
study’s binary classification, SVM was selected due to its proven
effectiveness in face recognition tasks [15]. Additionally, acknowledging
Random Forest’s (RF) capabilities in processing non-linear data, resisting
overfitting, accommodating missing values, and its scalability, efficiency,
and proficiency with imbalanced datasets, we developed an RF model to assess
and compare its classification performance with SVM [16]. The parameters of
both classifiers also have been tuned.

####  II-D1 Feature Extraction

ERP features were extracted through a series of steps: bandpass filtering (1-4
Hz), segmentation into epochs (each block was segmented into 250-sample
epochs, each corresponding to 1000 milliseconds), and downsampling to 50 Hz
(50 samples per epoch to improve the signal-to-noise ratio). The epochs were
then averaged to derive distinct mean signals for baseline and stimulus-evoked
activities per block. Next, the averages for each category were calculated to
identify the distinct ERP waveforms for face and scene stimuli (Fig. 2(a)).
For extracting ERP-related features, key statistical parameters such as mean
amplitude, variance, standard deviation, peak-to-peak amplitude, zero
crossings, and number of peaks were computed within the identified time
windows: early response (0-50 ms), mid-range processing (80-210 ms, 240-350
ms, 400-500 ms, 520-630 ms), and extended analysis periods (650-900 ms), with
an overall assessment across the full epoch (0-1000 ms) leading to 42 (6x7)
ERP features for each channel. Also, utilizing Linear Discriminant Analysis,
the intra-class and inter-class variance have been minimized and maximized,
respectively, yielding a single, feature per EEG channel. Overall, the
analysis led to the extraction of a total of 344 (8x43) ERP-related features
for each trial. To extract time-frequency features, MCWT were generated for
0-40 Hz frequencies in 1 Hz increments, with cycles linearly varying between
0.1 and 10, and convolved with EEG data. Next, epoching and baseline
normalization were performed, quantified using the logarithmic equation
$dB=10\cdot\log_{10}\left(\frac{\text{activity}}{\text{baseline}}\right)$,
where $dB$ represents the power in decibels. Finally, for each epoch of
activity, signal mean, variance, peak frequency, peak magnitude, skewness,
energy, and kurtosis, were extracted which led to 56 (8x7) time-frequency
features for each trial. In addition, the Hilbert transform was applied to
extract the envelope of EEG data in various frequency bands (Delta: [1, 4] Hz,
theta: [4, 8] Hz, alpha: [8, 14] Hz, beta: [14, 30] Hz, and gamma: [30, 40]
Hz). Then the mean, median, standard deviation, skewness, energy, and kurtosis
of each envelope computed led to the extraction of 240 (8x30) features per
trial.

Figure 2: a(1) and a(2): Time-frequency power spectra and b(1) and b(2) ERP
analyses results by channel for one subject, ”*” delta band, ”**” theta and
early alpha band

####  II-D2 Classifier Development

To develop classifiers, initially, all features were combined to form a
comprehensive feature matrix containing 640 features per trial across a total
of 320 trials. Then classifiers were developed and their hyperparameters
systemically optimized for each subject using the well-established Optuna
library [17], ensuring that the selections were not random but systemically
determined. Optuna employs Bayesian optimization with a Tree-structured Parzen
Estimator (TPE) to efficiently explore the hyperparameter space. For each
subject in our study, a separate Optuna experiment was created, and the
hyperparameters were optimized individually. This approach ensures that the
model is finely tuned to each subject’s unique EEG data characteristics,
maximizing the predictive accuracy and generalization capability of the model.
The criteria for selecting the range of hyperparameters were based on
preliminary empirical tests and literature review, ensuring they are suitable
for the complexity and nature of EEG data being analyzed. We aimed to define a
comprehensive yet computationally feasible hyperparameter range with the TPE
algorithm, maximizing cross-validation accuracy and minimizing overfitting to
ensure robust model performance on unseen data. When tuning the RF classifier,
the hyperparameters that were used are as follows: (i) the number of
estimators was set between 2 and 10, (ii) the depth of the tree was between 5
and 20, (iii) the minimum number of samples required to split an internal node
was between 2 and 20, (iv) minimum number of samples to qualify as a leaf node
was between 2 and 5. Moreover, when looking for the best split, the number of
features considered from [‘auto’, ‘sqrt’, ‘log 2’] and [‘gini’, ‘entropy’] was
used to measure the quality of the split (criterion). Hyperparameters for SVM
tuning—misclassification cost ($C$) and Gaussian kernel parameter
($\Gamma$)—were bounded by a predefined logarithmic scale ranging from 0.001
to 1000, to balance the decision boundary and the complexity of the kernel.
Validation of the final model was done based on the Receiver Operating
Characteristic (ROC) curves, AUC scores, and mean accuracy metrics (Table I
and Fig. 3).

##  III Results and discussion

###  III-A Time and Frequency Domain Analysis

The ERP waveforms (Fig. 2 (b1) and (b2)), reveal distinct temporal patterns
between scene and face stimuli across EEG channels (early, mid-range, and
extended response). As it is shown in the wavelet power spectra (Fig. 2 (a1)
and (a2)), there is a prominent power concentration in the lower frequency
bands (Delta and Theta and early Alpha) during the early to mid-time windows
(0-800 ms), particularly for face stimuli. In contrast to this, scene
processing appears more distributed across frequencies and time. The HE also
indicates distinct patterns in brain activity corresponding to the face and
scene conditions, particularly in the Alpha and Beta bands.

TABLE I: Models’ tuned hyperparameter and performance across subjects. Abbreviations: NS: Number of Estimators, MD: Max Depth, MSS: Min Samples Split, MF: Max Features. Model |  Param. | Subjects
---|---|---
|  |  1 |  2 |  3 |  4 |  5 |  6 |  7
RF |  NS |  9 |  9 |  10 |  8 |  10 |  8 |  8
|  MD |  13 |  5 |  7 |  11 |  5 |  10 |  5
|  MSS |  13 |  6 |  8 |  16 |  19 |  7 |  9
|  MF |  auto |  sqrt |  sqrt |  sqrt |  sqrt |  auto |  auto
|  ACC |  75% |  79% |  72% |  80% |  81% |  80% |  78%
|  AUC |  0.72 |  0.83 |  0.70 |  0.82 |  0.84 |  0.87 |  0.84
SVM |  $C$ |  93 |  354 |  672 |  188 |  894 |  20 |  1.5e-2
|  $\Gamma$$e-3$ |  1.8 |  1 |  2 |  1 |  1.1 |  1 |  1
|  ACC |  79% |  82% |  73% |  80% |  85% |  84% |  75%
|  AUC |  0.84 |  0.89 |  0.76 |  0.85 |  0.90 |  0.92 |  0.88
Figure 3: ROC curves of the tuned Scene and Face Classifiers

###  III-B Personalized classifier tuning

Table I summarizes the results of the hyperparameters tuning for RF and SVM
models. Using the tuned parameters, the performance of the models was verified
through 5-fold cross-validation. The RF model resulted in accuracy ranging
from 71.9% to 80.6%. For the SVM model, adjustments in the $C$ and $\Gamma$
resulted in accuracy between 72.8% and 84.7% (Table I). The ROC plot (Fig. 3)
exhibits above-chance classification ability for both models, with SVM showing
superior performance. In our study, we employed the Optuna framework for
hyperparameter optimization, utilizing TPE for adaptive Bayesian optimization.
This method refines the search space dynamically based on feedback from
ongoing trials, enhancing the precision in selecting optimal hyperparameters
and reducing computational overhead. It allows for a nuanced exploration of
hyperparameter effects, significantly improving model performance and
generalization in complex EEG data analysis. This approach significantly
differs from traditional methods like grid search, manual tuning, or nested
cross-validation. The latter, used by Cooney et al. for EEG signal
classification with CNNs, relies on extensive evaluations to optimize
hyperparameters [18]. Our approach introduces a novel aspect of EEG signal
classification by leveraging an advanced, automated hyperparameter tuning
process. This method enhances the precision in selecting optimal
hyperparameters, reducing computational overhead, and leading to improved
model generalization. This systematic and individualized tuning approach
underscores the robustness and reliability of our findings, as each subject’s
model configuration is specifically tuned to derive the best possible outcomes
from their data.

##  IV Conclusion

In conclusion, our study developed a novel BCI platform for interpreting EEG
signals in visual sustained attention tasks. We successfully extracted key
temporal and spectral features. Our tuned SVM and RF models decoded
participants’ attentional state with notable accuracy of 80% and 78%,
respectively. This research study is a foundation for creating a real-time
closed-loop neurofeedback platform that is responsive to immediate neural
pattern changes by employing high-speed open-source language programming and
pipelines.

## References

  * [1] Kwangsun Yoo, Monica D Rosenberg, Young Hye Kwon, Qi Lin, Emily W Avery, Dustin Sheinost, R Todd Constable, and Marvin M Chun.  A brain-based general measure of attention.  Nature human behaviour, 6(6):782–795, 2022.
  * [2] Robert T Thibault, Michael Lifshitz, and Amir Raz.  The self-regulating brain and neurofeedback: Experimental science and clinical promise.  cortex, 74:247–261, 2016.
  * [3] Luis Fernando Nicolas-Alonso and Jaime Gomez-Gil.  Brain computer interfaces, a review.  sensors, 12(2):1211–1279, 2012.
  * [4] Reza Abiri, Soheil Borhani, Eric W Sellers, Yang Jiang, and Xiaopeng Zhao.  A comprehensive review of eeg-based brain–computer interface paradigms.  Journal of neural engineering, 16(1):011001, 2019.
  * [5] Choon Guan Lim, Chui Pin Soh, Shernice Shi Yun Lim, Daniel Shuen Sheng Fung, Cuntai Guan, and Tih-Shih Lee.  Home-based brain–computer interface attention training program for attention deficit hyperactivity disorder: A feasibility trial.  Child and Adolescent Psychiatry and Mental Health, 17(1):1–11, 2023.
  * [6] Bruno Rossion, Corentin Jacques, and Jacques Jonas.  Intracerebral electrophysiological recordings to understand the neural basis of human face recognition.  Brain Sciences, 13(2):354, 2023.
  * [7] Greta Tuckute, Sofie Therese Hansen, Troels Wesenberg Kjaer, and Lars Kai Hansen.  Real-time decoding of attentional states using closed-loop eeg neurofeedback.  Neural Computation, 33(4):967–1004, 2021.
  * [8] Kartik K Sreenivasan, Jonathan M Goldstein, Audrey G Lustig, Luis R Rivas, and Amishi P Jha.  Attention to faces modulates early face processing during low but not high face discriminability.  Attention, Perception, & Psychophysics, 71:837–846, 2009.
  * [9] Alexandra List, Monica D Rosenberg, Aleksandra Sherman, and Michael Esterman.  Pattern classification of eeg signals reveals perceptual and attentional states.  PloS one, 12(4):e0176349, 2017.
  * [10] Unicorn the brain interface.  https://www.unicorn-bi.com.  Accessed: 2024-01-31.
  * [11] Reza Abiri, Soheil Borhani, Yang Jiang, Xiaopeng Zhao, et al.  Decoding attentional state to faces and scenes using eeg brainwaves.  Complexity, 2019, 2019.
  * [12] Ian H Robertson, Tom Manly, Jackie Andrade, Bart T Baddeley, and Jenny Yiend.  Oops!’: performance correlates of everyday attentional failures in traumatic brain injured and normal subjects.  Neuropsychologia, 35(6):747–758, 1997.
  * [13] Deborah E Hannula and Anthony J Greene.  The hippocampus reevaluated in unconscious learning and memory: at a tipping point?  Frontiers in human neuroscience, 6:80, 2012.
  * [14] Ze Deng, Dan Chen, Yangyang Hu, Xiaoming Wu, Weizhou Peng, and Xiaoli Li.  Massively parallel non-stationary eeg data processing on gpgpu platforms with morlet continuous wavelet transform.  Journal of Internet Services and Applications, 3(3):347–357, 2012.
  * [15] Viola Bakiasi Shtino and Markela Muça.  Comparative study of k-nn, naive bayes and svm for face expression classification techniques.  Balkan Journal of Interdisciplinary Research, 9(3):23–32, 2023.
  * [16] Friederike Maite Siemers and Jürgen Bajorath.  Differences in learning characteristics between support vector machine and random forest models for compound classification revealed by shapley value analysis.  Scientific Reports, 13(1):5983, 2023.
  * [17] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A next-generation hyperparameter optimization framework.  In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631, 2019.
  * [18] Ciaran Cooney, Attila Korik, Raffaella Folli, and Damien Coyle.  Evaluation of hyperparameter optimization in machine and deep learning methods for decoding imagined speech eeg.  Sensors, 20(16):4629, 2020.


# Sentiment Analysis for Sinhala Language using Deep Learning Techniques

Lahiru Senevirathne<EMAIL_ADDRESS>0000-0001-7117-8154 ,  Piyumal
Demotte<EMAIL_ADDRESS>0000-0001-8144-580X ,  Binod
Karunanayake<EMAIL_ADDRESS>,  Udyogi Munasinghe
<EMAIL_ADDRESS>and  Surangika Ranathunga<EMAIL_ADDRESS>0000-0003-0701-0204 Department of Computer Science and Engineering, University
of MoratuwaKatubeddaMoratuwaSri Lanka10400

(2020)

###### Abstract.

Due to the high impact of the fast-evolving fields of machine learning and
deep learning, Natural Language Processing (NLP) tasks have further obtained
comprehensive performances for highly resourced languages such as English and
Chinese. However Sinhala, which is an under-resourced language with a rich
morphology, has not experienced these advancements. For sentiment analysis,
there exists only two previous research with deep learning approaches, which
focused only on document-level sentiment analysis for the binary case. They
experimented with only three types of deep learning models. In contrast, this
paper presents a much comprehensive study on the use of standard sequence
models such as RNN, LSTM, Bi-LSTM, as well as more recent state-of-the-art
models such as hierarchical attention hybrid neural networks, and capsule
networks. Classification is done at document-level but with more granularity
by considering POSITIVE, NEGATIVE, NEUTRAL, and CONFLICT classes. A data set
of 15059 Sinhala news comments, annotated with these four classes and a corpus
consists of 9.48 million tokens are publicly released. This is the largest
sentiment annotated data set for Sinhala so far.

sentiment analysis, deep learning, Sinhala language, attention

††copyright: acmcopyright††journalyear: 2020††doi:
10.1145/1122445.1122456††ccs: Computing methodologies Information
extraction††ccs: Computing methodologies Language resources††ccs: Computing
methodologies Natural language processing††ccs: Computing methodologies Neural
networks††ccs: Computing methodologies Regularization††ccs: Computing
methodologies Cross-validation

##  1\. Introduction

With the development of deep learning techniques such as Convolutional Neural
Networks (CNN), Recurrent Neural Networks (RNN) (Zhang et al., 2018b) and
language independent features (Mikolov et al., 2013), the domain of sentiment
analysis has reported impressive results. Over the years, many of these
variants and combinations of deep learning techniques (Wang et al., 2016) and
feature representations have been used for high resourced languages such as
English (Kim, 2014). There also exist certain advancements in sentiment
analysis for languages such as Chinese, Arabic, Spanish (Rosas et al., 2013)
and some Indic languages (Rani and Kumar, 2019).

Sinhala, which is a morphologically rich Indo-Aryan language, has not
experienced these advancements due to its insular and under-resourced nature
(Liyanage, 2018). One of the main challenges is not having large enough
annotated corpora. The data set from Liyanage (2018) is the only publicly
available annotated data set for sentiment analysis. However it includes only
5010 comments extracted from one news source, and contains only POSITIVE and
NEGATIVE samples.

The ﬁrst experiment on using deep learning techniques for Sinhala sentiment
analysis was conducted by Liyanage (2018). Under this research, basic deep
learning techniques such as Long Short-Term Memory (LSTM) network and CNN were
used to categorize news comments as POSITIVE and NEGATIVE. Demotte et al.
(2020) conducted an experiment with the same data set using Sentence-State
LSTM (S-LSTM) (Zhang et al., 2018a), which is a rather advanced technique
where the analysis was further improved considering the n-gram features of
text with word embeddings.

In this paper, we present a more comprehensive empirical study on the use of
deep learning techniques for document-level sentiment analysis for Sinhala
with respect to four sentiment categories as POSITIVE, NEGATIVE, NEUTRAL and
CONFLICT. The experiments were conducted with the commonly used sequence
models such as RNN, LSTM, Bi-LSTM, various improvements on these vanilla
models such as stacking and regularization, as well as more recent ones such
as hierarchical attention hybrid neural networks and capsule networks.
Furthermore, we present a data set of 15059 comments, annotated with these
four classes to be used for sentiment analysis, based on Sinhala news comments
extracted from online newspapers namely
GossipLanka111https://www.gossiplankanews.com/ and
Lankadeepa222http://www.lankadeepa.lk/. This is the only publicly available
multi-class, multi-source dataset for Sinhala sentiment analysis.

Our code implementation, word embedding models, and annotated data set are
publicly
available333https://github.com/LahiruSen/sinhala_sentiment_anlaysis_tallip.git/.

##  2\. Related work

###  2.1. Deep learning for Sentiment Analysis

Recent advancements of Natural Language Processing (NLP) tasks were a direct
result of using deep learning techniques (Zhang et al., 2018b). In these
techniques, text is treated as sequences or spatial patterns, which allowed
the modeling of higher level NLP concepts beyond the boundaries of the meaning
of words in natural language. CNNs (Kim, 2014) and LSTMs (Xu et al., 2016)
were the proper representatives under this paradigm.

With respect to sentiment analysis, linguistic knowledge such as sentiment
lexicons and POS tags has been utilized as auxiliary input for the deep
learning models, to capture deeper levels of language specific features for
greater success (Qian et al., 2016). However, formulating language specific
linguistic knowledge needs considerable human effort. Another approach is to
experiment different combinations and variation of deep learning techniques,
as an end to end solution, considering both the sequential nature and local
n-gram information of text (Wang et al., 2016).

More recent research exploits the attention mechanism (Bahdanau et al., 2014)
in sentiment classification. Abreu et al. (2019) argued that different parts
of a document have no similar or relevant information, thus special attention
should be given to some parts of a document to identify the overall sentiment
correctly. They proposed Hierarchical Attention Hybrid Neural Networks
(HAHNN), which combines convolutional layers, Gated Recurrent units (GRU),
LSTM units and attention mechanism to implement a better document
classification model. It accordingly pays more or less attention to individual
words and sentences when it constructs document representation with the two
levels of attention mechanisms as word-level attention and sentence-level
attention.

The capsule network, which was initially introduced by Sabour et al. (2017) as
an improvement to the CNN strategy, was implemented to be used in NLP tasks
including sentiment analysis. Zhao et al. (2018) implemented different
variations of capsule architectures as capsule-A and capsule-B for binary and
multi-level sentiment analysis with a dynamic routing process. The key feature
of the capsule architecture is the ability to capture context level
information with the exact order or pose of the information with the vector
representation of the capsules. The dynamic routing process of the proposed
architecture could eliminate the disadvantages of CNNs such as high
computational cost and loss of information due to the max pooling strategy
widely used in CNNs.

Moreover, the transformer networks built solely upon the attention mechanism
while neglecting recurrence and convolutions tend to produce promising results
(Vaswani et al., 2017) in the domain of NLP. In particular, Bidirectional
Encoder Representations (BERT) base and BERT large (Munikar et al., 2019)
models have produced state-of-the-art performance for fine-grained sentiment
analysis. However, the high computational cost to build models for low
resource languages hinder the use of BERT towards the NLP tasks more
frequently. Another drawback of using BERT for under resource languages, is
not having enough text to comprehensively learn contextual information as
opposed to English, which has billions of words.

###  2.2. Sentiment Analysis using Deep Learning for Indic Languages

Related approaches for sentiment analysis in Indic languages were
comprehensively investigated by Rani and Kumar (2019). According to the
authors, Indic languages such as Hindi, Bengali, Tamil, Malayalam, Urdu and
Kannada are the languages with major research work for sentiment analysis.

Akhtar et al. (2016) conducted the first experiment on deep learning based
sentiment analysis for Hindi language. There they used a CNN with multi-
objective optimization for both sentence-level and aspect-level sentiment
analysis. Hassan et al. (2016) also introduced deep learning techniques for
sentiment analysis for Bengali language. There they used an LSTM with
different variations of loss functions and regularization techniques. The RNN
based approach for Bengali and Tamil tweets sentiment analysis proposed by
Seshadri et al. (2016) further illustrated the advancements of deep learning
techniques in sentiment analysis for Indic languages. Kumar et al. (2017) also
conducted an experiment on Malayalam tweets using CNN and LSTM. The
comprehensive study conducted by Soumya and Pramod (2019) includes the
experiments based on many deep learning techniques such as CNN, RNN, Bi-LSTM,
GRU on Malayalam tweet sentiment analysis.

###  2.3.  Sinhala Sentiment Analysis

Sinhala is a morphologically rich, but less resourced Indic language when
compared to languages such as English (Liyanage, 2018) or even other major
Indic languages, in the perspective of sentiment analysis, as well as in NLP
in general. Consequently, not many sentiment annotated corpora or sentiment
lexicons are publicly available for Sinhala.

Medagoda (2016) conducted the ﬁrst experiment on sentiment analysis for
Sinhala. A simple feed forward neural network was used with document term
frequencies. Medagoda (2017) experimented with three new techniques to enhance
the sentiment classiﬁcation process. The ﬁrst methodology extracts cross
linguistic features related to sentiment of Sinhala language based on a
bilingual dictionary of English and Sinhala. A further analysis introduced the
linguistic features speciﬁc for Sinhala sentiment analysis. This research then
mainly focused on statistical machine learning algorithms such as Support
Vector Machines (SVM) and Naïve Bayes, where the generated lexicons in
previous steps were used for sentiment analysis. Chathuranga et al. (2019)
also presented a technique based on corpus-based sentiment analysis. The
proposed method could be introduced as a semi-automated method based on
sentiment lexicon generation for sentiment analysis.

Work of Liyanage (2018) can be considered as the ﬁrst to experiment with deep
learning techniques for binary sentiment analysis task in Sinhala language.
These techniques include LSTM and CNN+SVM models for a rather small data set
with POSITIVE and NEGATIVE sentiment categories. These models were trained
using Sinhala word embedding models, thus no language-speciﬁc features were
used. The same features were used to train statistical machine learning
algorithms, which included Naïve Bayes, logistic regression, decision trees,
random forests and SVM. Although these classiﬁers showed a much superior
performance with word embedding features as opposed to sparse features such as
TF-IDF, their results were inferior to that of LSTM. This research carried out
a comprehensive study on using different models, with respect to the
dimensionality of the embeddings, and the effect of punctuation marks.

Demotte et al. (2020) proposed a strategy for sentiment detection of Sinhala
news comments for the same data set used by Liyanage (2018) based on S-LSTM
(Zhang et al., 2018a). This is a rather advanced technique where the sentiment
classification process was further improved considering the n-gram features
with Word2Vec and fastText embeddings. The word level state and sentence level
state with recurrent information exchange between each state of the S-LSTM
network have proven to be able to capture long term dependencies and
outperform the traditional LSTM architecture used by Liyanage (2018).

##  3\. Dataset

###  3.1. Resource Acquisition

As mentioned earlier Liyanage (2018), released the only sentiment annotated
dataset, which includes only 9060 comments. There are few issues with this
data set. One problem is the low inter-annotator agreement for the multi-label
annotation as NEGATIVE, NEUTRAL, and POSITIVE sentiments. Cohen’s kappa value
for this annotation was as low as 0.52, whereas the CONFLICT sentiment was not
even captured. On the other hand, the data was crawled only from one news
source, Lankadeepa, a local news website in Sinhala language. Despite these
issues, this data set covers a wide variety of categories including politics,
sports, crime, economy, society and culture.

Finding another source to extract comments from news articles was truly
challenging. Even though there were multiple news sources in Sinhala, there
were several issues with those sources. Some websites had only a few comments
or no comments at all for most of the articles. Some sources welcomed comments
in both Sinhala as well as Singlish (Sinhala words in English letters). Some
of them have published comments as digital images, which come with an extra
overhead to extract content. Another factor to be considered is the encoding
feasibility. If the content is not in UTF-8 or similar character encoding, it
presents an encoding overhead, due to the non-availability of proper encoding
tools for Sinhala language.

Considering all these factors, comments were crawled from GossipLanka, which
is again a local news website. Even though it does not have a printed version,
it frequently attracts comments from users because of the popularity of the
site. However, this source also has some of the previously mentioned issues.
The pre-processing steps that were followed to overcome these issues are
described in Section 4.1.

In the dataset extracted from GossipLanka, the average number of comments per
article is 8, while some news articles consist of more than 20 comments. 12776
articles containing 30000 comments were extracted in total from this source.
These articles are also from a wide variety of categories including politics,
sports, crime, economy and culture from the date range 2016/06 to 2020/05.

###  3.2. Annotation

For sentiment annotation, three annotators were employed, and the reliability
of their work was monitored continuously. 15059 comments were annotated with 4
labels. This includes 9059 re-annotated comments from Lankadeepa (Liyanage,
2018) and 6000 newly crawled comments from GossipLanka. There are 7665
NEGATIVE comments, 3080 NEUTRAL comments, 2403 POSITIVE comments, and 1911
CONFLICT comments.

Out of 15059 comments, 7000 comments were later re-tagged by authors in order
to calculate inter-annotator agreement, such that each of those 7000 comments
was annotated by two annotators. Calculated Cohen’s kappa value was 0.65.
Following guidelines were strictly followed when annotating comments.

  * •

If the comment has a purely negative opinion or a purely positive opinion, tag
it with NEGATIVE, or POSITIVE (respectively).

  * •

If there is no positive/negative opinion, tag it with NEUTRAL.

  * •

If comment has both negative and positive opinions, tag it with CONFLICT.

##  4\. Methodology

###  4.1. Pre-Processing

Since comments from both sources are generated by news readers, pre-processing
and polishing data was a mandatory step prior to feed to any neural network or
machine learning algorithm. Sinhala language originally did not have any
punctuation marks (Liyanage, 2018). However modern Sinhala has adopted many
punctuation marks from other languages such as English. As a result of this,
punctuation marks may reduce the accuracy of the sentiment analysis task.
However, removing some punctuation marks may adversely affect the model
performance. Liyanage (2018) performed a comprehensive analysis on the effect
of punctuation marks (including question mark (?), full stop (.), comma (,),
and exclamation mark (!)) on sentiment analysis for Sinhala. After those
experiments, his conclusion was to remove all punctuation marks, except the
question mark. Question mark is mostly used with comments with a negative
sentiment. This is a much useful feature to identify negative comments from
the rest.

Both Lankadeepa and GossipLanka allow users to comment in Sinhala, English, or
Sinhala written in English. In Lankadeepa, comments that are not in Sinhala
are converted to Sinhala prior to posting to the website by following a strict
moderation process. However GossipLanka does not impose such restrictions on
comments from users. Therefore, we had to filter punctuation marks and
comments that are not in Sinhala, by considering unicode values. In this
method, all the undefined characters are filtered except the question mark,
space character and decimal numbers. Further, to tokenize words in a sentence,
the Sinhala tokenizer from sinling444https://github.com/ysenarath/sinling was
used.

###  4.2. Feature Selection

Linguistic features such as sentiment lexicons and POS tags are not available,
and language independent features such as bag of words, word n-grams, and TF-
IDF are not performing at an acceptable level (Liyanage, 2018). Thus it was
decided to directly experiment with the two neural word embedding techniques
Word2Vec and fastText as the input features for the deep learning models. As
mentioned earlier, these have shown improvements over the aforementioned
feature representations for Sinhala sentiment analysis (Liyanage, 2018).

###  4.3. Sentiment Analysis with Deep Learning Techniques

As mentioned earlier, Sinhala does not have well-developed linguistic
resources such as sentiment lexicons. Thus, deep learning techniques mentioned
in Section 2.1 that used such auxiliary information could not be applied.
Similarly, BERT based techniques could not be applied as Sinhala does not have
a pre-trained model and we did not have the capacity to build one.
Therefore, to begin with, well-known deep learning models such as RNN, LSTM,
GRU, and BiLSTM were applied to perform sentiment analysis on our data set.
Then regularization techniques such as dropout, L1/L2 regularization, and
early stopping were applied on those vanilla models. After identifying the
best performing baseline models, their performance was further optimized by
combining with CNN and stacking them on top of each other. Specifically, the
combination of CNN with LSTM, GRU and BiLSTM were experimented. In this way,
CNN could extract more coarse-grained features and input to sequential models.
In addition to that, stacked LSTM and stacked BiLSTM were built by stacking up
to 3 layers on top of the base models, LSTM and BiLSTM. Objective of stacking
models is to extract rich contextual information using upper layers of the
network.

Later, more recently introduced deep learning models were employed. First and
foremost, HAHNN model (Abreu et al., 2019) was experimented. Secondly, capsule
networks (Zhao et al., 2018) were used, with the use of two architectures
namely capsule-A and capsule-B, which have been designed to capture different
variations of n-grams features.

Finally, hyper parameter optimization on the above models was performed by
optimizing for the number of units in hidden layers, dropout values, L1 and L2
regularization factors, optimizers, and learning rate. Moreover, the number of
filters, kernel size, and dilation rate were optimized in CNN layers of the
respective models. The hyper-parameter values mentioned in Section 5 are the
optimized values.

##  5\. Experiments and Evaluation

Every experiment under this research was conducted in a Google Colab-
pro555https://colab.research.google.com/ environment with high-end GPUs such
as T4, P100 and 25GB high memory VMs. Moreover, TensorFlow and Keras libraries
were used to build, experiment and evaluate models presented in this paper.
The data set was splitted into train and validation sets with a ratio of $4:1$
for experiments regarding pre-processing and input features. 10-fold cross
validation was used for experiments with deep learning models. All the
evaluation metrics for each experiment were reported as the weighted average
over the four sentiment classes.

###  5.1. Pre-Processing Techniques

Our experiments on different pre-processing techniques reached the same
conclusion of Liyanage (2018) as per section 4.1. Results in Table 1 explain
the impact of different pre-processing techniques towards the sentiment
analysis task. Data set without punctuation marks except the question mark has
outperformed the other two approaches with a remarkable margin, giving a
weighted accuracy of 63.35%, and a weighted F1 score of 61%.

###  5.2. Word embedding models

FastText and Word2Vec models with 50, 150, 200, 250, 300, 350, 400 and 450
dimensions were utilized to analyse the effect of the dimension of the word
embeddings for sentiment analysis. 16840 news articles along with their
comments from both sources were fed to generate these models. Further,
multiple values were experimented for the hyper-parameters in word embedding
models such as window size, minimum word count, number of workers, and down
sampling. Respectively 5, 1, 4, and 0.001 for these hyper-parameters gave
optimal results. As per the results in Table 2, fastText model with 300
dimensions outperformed all other word embedding models, which was thereafter
fixed for the subsequent experiments.

Table 1. Results of Multi-level Sentiment Analysis on LSTM with Different Pre-processing Techniques (Holdout Method) Pre-processing | Features | Evaluation Metrics(%)
---|---|---
|  | Accuracy | Precision | Recall | F1
With punctuation marks | FastText(size=300) | 58.57 | 58.62 | 58.57 | 58.59
Without any punctuation marks | FastText(size=300) | 61.75 | 60.72 | 61.75 | 61.09
Without punctuation marks a | FastText(size=300) | 63.35 | 62.15 | 63.35 | 61.00
aExcept question mark(?). |  |
Table 2. Accuracy of Experiments on Word Embedding Models (Holdout method) Embedding Dimention | LSTM-Word2vec (%) | LSTM-Fasttext (%)
---|---|---
50 | 62.21 | 62.57
100 | 62.12 | 63.64
150 | 61.28 | 63.57
200 | 62.20 | 63.94
250 | 61.83 | 63.79
300 | 62.62 | 64.23
350 | 62.26 | 63.19
400 | 62.33 | 63.07
450 | 62.37 | 63.88

###  5.3. Experiments with Baseline Models

First, the vanilla RNN proposed by Wang et al. (2016) was experimented with.
This network consists of an input layer, followed by an embedding layer, an
RNN layer, a dense layer with the ReLU activation function. Finally, a dense
layer with 4 hidden units with softmax activation function was added to
predict the sentiment class of a given comment. Moreover, dropout technique
was used after the embedding layer to prevent over-fitting. This technique
randomly drops hidden units in dense layers and reduces the number of
trainable parameters.

However, this vanilla RNN suffers from vanishing gradient problem which
happens during the back-propagation. LSTM architecture was introduced to solve
this problem (Hochreiter and Schmidhuber, 1997). Therefore, RNN in the above
network was replaced with an LSTM layer. A dropout layer was added after the
LSTM layer to prevent over fitting. Further, we experimented with GRU (Chung
et al., 2014) by replacing the RNN layer of the above network. GRU uses a
single gating unit, which simultaneously controls the forgetting factor and
the decision to update the state unit, whereas LSTM uses 3 gates to control
the memorizing process as forget gate, input gate and output gate.

For all the above experiments, categorical cross-entropy was applied as the
loss function and Adadelta optimizer with the learning rate as 0.95. In
addition, different hyper-parameters were tuned and identified “He
initialization” for initialization technique, 0.5 for dropout value, and ReLU
activation function as optimal values for hyper-parameters for above models.

####  5.3.1. Bi-Directional LSTM (BiLSTM)

BiLSTM is an improvement to the LSTM architecture, where the model is trained
on both positive and negative time directions (Schuster and Paliwal, 1997).
BiLSTM consists of an input layer, followed by an embedding layer. Then a
bidirectional layer of an LSTM layer was added and the merge mode was
manipulated as ‘concat’ to concatenate outputs of two hidden layers before
being passed on to the next layer. Afterwards, a time distributed dense layer
was applied, followed by a flatten layer. Lastly, a dense layer with 4 hidden
units with softmax activation function was utilized to predict the sentiment
class. As in the previous cases, categorical cross-entropy was applied as the
loss function and Adam optimizer as optimization strategy with hyper-parameter
optimization were used to obtain optimal performance considering the task
specific nature of sentiment analysis as an NLP task.

As per results in Table 3 with 10-fold cross validation BiLSTM produced a
weighted accuracy of 63.81% and a weighted F1 score of 57.71%, thus beating
the vanilla RNN, LSTM, and GRU models. Therefore LSTM and BiLSTM were selected
for further improvements.

###  5.4. CNN + GRU/LSTM/BiLSTM

CNNs are suitable to extract local and deep context level information from
natural languages (Kim, 2014). Even though CNN can extract local features from
the text, it alone cannot handle inputs in a sequential manner. To overcome
this problem, a combination of CNN and other sequential models was utilized as
suggested by Wang et al. (2016). This model consists of a word embedding
layer, 2 convolutional layers followed by 2 max-pooling layers, a
concatenation layer, a LSTM/GRU/BiLSTM layer and a fully connected layer with
softmax output.

The experiments were conducted with different variations of this model as
follows.

  * •

CNN-GRU-fastText: A model with pre-trained vectors from fastText, max pooling
and GRU recurrent unit.

  * •

CNN-LSTM-fastText: A model with pre-trained vectors from fastText, max pooling
and LSTM recurrent unit.

  * •

CNN-BiLSTM-fastText: A model with pre-trained vectors from fastText, max
pooling and bidirectional LSTM.

As per results in Table 3, these experiments did not suggest a noticeable
improvement upon most of the baseline models with the exception of CNN+LSTM
and CNN+BiLSTM. One reason for this low performance could be not having enough
data to learn trainable parameters as a complex model that resulted with the
CNN integration.

###  5.5. Stacked LSTM and BiLSTM

2-layer and 3-layer stacked models for both LSTM and BiLSTM networks were
experimented under this step. These stacked models have more upper layers to
extract rich contextual information from both past and future time sequences
(Zhou et al., 2019). Same as earlier, the stacked LSTM/BiLSTM model consists
of an input layer, followed by an embedding layer. Next, 2 or 3, LSTM/BiLSTM
layers were added sequentially. After obtaining information of the input
sequence by the first hidden layer, it outputs hidden vectors. Then the second
LSTM/BiLSTM layer takes outputs of the first hidden layer as inputs and
extracts further features. If a BiLSTM layer is used, information extraction
happens in both forward and backward directions and then the output layer
combines two upper hidden layers as the output. Same as before, after the
BiLSTM layer, a time distributed dense layer with a flatten layer was applied.
Finally, a dense layer of 4 hidden units with softmax activation function was
further employed regardless of LSTM or BiLSTM layer, to predict the sentiment
class of a given comment.

As per results in Table 3 with 10-fold cross validation, the “Stacked BiLSTM
3” model reached a weighted accuracy of 63.13% and a weighted F1 score of
59.42% by outperforming all the aforementioned approaches. This could be
justified as the ability of the stacked BiLSTM to capture the context level
information in both left and right directions, while considering substantial
amount of neural representation for language modeling based on the stacking
strategy.

###  5.6. Hierarchical Attention Hybrid Neural Networks

The first state-of-the-art deep learning technique we employed was HAHNN
(Abreu et al., 2019). Specifically, Adam optimizer with a learning rate of
0.001 and learning rate decay of 0.0001 were used. Moreover, 0.2 for dropout,
3,4,5 for filter sizes, and 64 for batch size as hyper-parameters gave the
optimal results shown in Table 3. Since granular level attention was given to
important words in a sentence and individual sentences, this approach could
outperform other baseline deep learning models and their improvements to some
extent.

However, when analysing the results of Table 3, HAHNN did not illustrate
greater performance as expected. This could be due to the shorter length of
most of the comments, which hindered the ability to learn deeper neural
representation with the attention mechanism.

###  5.7. Capsule Networks

Finally, the two capsule architectures (capsule-A and capsule-B) proposed by
Zhao et al. (2018) were investigated, using margin loss as loss function with
margin as 0.2. Capsule-A was implemented only to use 3-grams as features,
while capsule-B used 3/4/5-gram features. Each model had 3 capsule layers
initiating with a convolutional layer with 32 filters and ReLU non-linearity
with stride as 1. Each capsule was instantiated with 16 dimensional-
parameters, while each capsule layer had 16 filters. Batch size and learning
rate were set to 50 and 0.001 while using Adam optimizer for optimization
process. The orphan category and leaky ReLU parameters were neglected to
obtain optimal results as suggested by Zhao et al. (2018).

As displayed in the results of Table 3, capsule-B architecture went beyond all
the other experimented models producing a weighted accuracy of 63.23% and a
weighted F1-score of 59.11% with 10-fold cross validation. This observation
could be elaborated considering the neural architecture based on vectors in
capsules that further improves language representation considering the exact
order or pose of the information. Furthermore, capsule-B outperformed
capsule-A due to its sophisticated architecture that is designed to capture
more n-grams.

Table 3. Results of Multi-level Sentiment Analysis (10 Fold Cross Validation) Model | Accuracy(%) | Precision(%) | Recall(%) | F1 Score(%)
---|---|---|---|---
RNN | 58.98 | 42.93 | 54.98 | 42.30
LSTM | 62.88 | 70.95 | 51.93 | 54.50
GRU | 62.78 | 60.93 | 62.78 | 54.83
BiLSTM | 63.81 | 61.17 | 63.81 | 57.71
CNN + GRU | 61.59 | 60.41 | 61.59 | 54.19
CNN + LSTM | 61.89 | 57.82 | 61.89 | 55.30
CNN + BiLSTM | 62.72 | 59.54 | 62.72 | 58.53
Stacked LSTM 2 | 61.92 | 56.92 | 61.92 | 53.17
Stacked LSTM 3 | 62.48 | 54.76 | 62.48 | 53.67
Stacked BiLSTM 2 | 63.18 | 60.50 | 63.18 | 57.78
Stacked BiLSTM 3 | 63.13 | 69.71 | 46.63 | 59.42
HAHNN | 61.16 | 71.08 | 48.54 | 59.25
Capsule-A | 61.89 | 56.12 | 61.89 | 53.55
Capsule-B | 63.23 | 59.84 | 63.23 | 59.11

##  6\. Discussion

The weighted accuracy of each experiment was bounded below the value of 65% as
per the inter-annotator agreement value. This is a direct result of the high
volume of noise in the dataset. As illustrated in Table 4, the CONFLICT and
the NEUTRAL classes seem to be considerably mis-classified as NEGATIVE
comments, due to the impact of a large number of NEGATIVE comments with
respect to the number of CONFLICT and NEUTRAL comments in the training set.
Figure 1 shows few comments where the model was confused while classifying.
The first example illustrates a comment that is negatively classified but
truly a CONFLICT comment. When considering the interpretation of the comment,
the sentence includes two negative sentences with a positive sentence, which
indicates some bias towards the NEGATIVE sentiment. The second and third
comments include NEGATIVE and NEUTRAL comments, which are classified as
POSITIVE and CONFLICT, respectively.

Table 4. Confusion Matrix for Best Model (Holdout Method) | Prediction |
---|---|---
| NEGATIVE | NEUTRAL | POSITIVE | CONFLICT
| NEGATIVE | 1407 | 56 | 41 | 29
| NEUTRAL | 344 | 110 | 35 | 16
Actual | POSITIVE | 162 | 30 | 367 | 31
| CONFLICT | 272 | 15 | 50 | 47

Figure 1. Misclassified Comments

##  7\. Conclusions and Future Work

In this research, a comprehensive analysis was conducted with the use of
state-of-the-art deep learning techniques such as RNN, LSTM, Bi-LSTM,
hierarchical attention hybrid neural networks, and capsule networks for multi-
class sentiment analysis of Sinhala news comments. This research could be
identified as the first experiment to conduct sentiment analysis at a more
granular level with four sentiment categories. Moreover, this research further
established the importance of language-independent word embedding features for
low-resource text classification. The obtained results are not high, owing to
the noisy data set used. This was made evident by the low Kappa value as well.
Despite this, the comparative results we provided, give a clear indication of
the best performing deep learning architectures, input features, as well as
the suitable pre-processing techniques for Sinhala text classification.

As a secondary contribution, a multi-class annotated data set for sentiment
analysis is presented, which consists of 15059 sentiment annotated Sinhala
news comments extracted from two Sinhala online news papers with four
sentiment categories namely POSITIVE, NEGATIVE, NEUTRAL and CONFLICT. Further,
a corpus that includes unannotated comments along with the corresponding news
articles, consisting of 9.48 million tokens was used to generate Word2Vec and
fastText models. Embedding models, source code for the deep learning models,
and all the data are publicly available.

Finally, as further improvements, more sophisticated word embedding techniques
such as BERT could be used for sentiment analysis to capture more syntactic
and semantic information of the language. Language dependent features such as
sentiment lexicons could also be used as auxiliary information to further
optimize deep learning models. It is also important to experiment the
developed models with different data set types. In the absence of customer
reviews written in Sinhala, a possible data source to explore would be Sinhala
Twitter data. Finally, it would be interesting to expand this research into
more fine-grained sentiment analysis tasks such as emotion identification,
sarcasm detection, and hate-speech detection.

## References

  * (1)
  * Abreu et al. (2019) Jader Abreu, Luis Fred, David Macêdo, and Cleber Zanchettin. 2019.  Hierarchical attentional hybrid neural networks for document classification. In _International Conference on Artificial Neural Networks_. Springer, 396–402.
  * Akhtar et al. (2016) Md Shad Akhtar, Ayush Kumar, Asif Ekbal, and Pushpak Bhattacharyya. 2016.  A hybrid deep learning architecture for sentiment analysis. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_. 482–493.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014\.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ (2014).
  * Chathuranga et al. (2019) PDT Chathuranga, SAS Lorensuhewa, and MAL Kalyani. 2019\.  Sinhala Sentiment Analysis using Corpus based Sentiment Lexicon. In _International Conference on Advances in ICT for Emerging Regions (ICTer)_ , Vol. 1. 7\.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * Demotte et al. (2020) Piyumal Demotte, Lahiru Senevirathne, Binod Karunanayake, Udyogi Munasinghe, and Surangika Ranathunga. 2020\.  Sentiment Analysis of Sinhala News Comments Using Sentence-State LSTM Networks. In _2020 Moratuwa Engineering Research Conference (MERCon)_. IEEE.
  * Hassan et al. (2016) Asif Hassan, Mohammad Rashedul Amin, Abul Kalam Al Azad, and Nabeel Mohammed. 2016\.  Sentiment analysis on bangla and romanized bangla text using deep recurrent models. In _2016 International Workshop on Computational Intelligence (IWCI)_. IEEE, 51–56.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  _arXiv preprint arXiv:1408.5882_ (2014).
  * Kumar et al. (2017) S Sachin Kumar, M Anand Kumar, and KP Soman. 2017.  Sentiment analysis of tweets in malayalam using long short-term memory units and convolutional neural nets. In _International Conference on Mining Intelligence and Knowledge Exploration_. Springer, 320–334.
  * Liyanage (2018) Isuru Udara Liyanage. 2018\.  Sentiment Analysis of Sinhala News Comments. (2018).  Unpublished.
  * Medagoda (2016) Nishantha Medagoda. 2016\.  Sentiment Analysis on Morphologically Rich Languages: An Artificial Neural Network (ANN) Approach.  In _Artificial Neural Network Modelling_. Springer, 377–393.
  * Medagoda (2017) Nishantha Medagoda. 2017\.  _Framework for Sentiment Classification for Morphologically Rich Languages: A Case Study for Sinhala_.  Ph.D. Dissertation. Auckland University of Technology.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_. 3111–3119.
  * Munikar et al. (2019) Manish Munikar, Sushil Shakya, and Aakash Shrestha. 2019\.  Fine-grained sentiment classification using BERT. In _2019 Artificial Intelligence for Transforming Business and Society (AITB)_ , Vol. 1. IEEE, 1–5.
  * Qian et al. (2016) Qiao Qian, Minlie Huang, Jinhao Lei, and Xiaoyan Zhu. 2016\.  Linguistically regularized LSTMs for sentiment classification.  _arXiv preprint arXiv:1611.03949_ (2016).
  * Rani and Kumar (2019) Sujata Rani and Parteek Kumar. 2019.  A journey of Indian languages over sentiment analysis: a systematic review.  _Artificial Intelligence Review_ 52, 2 (2019), 1415–1462.
  * Rosas et al. (2013) Verónica Pérez Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013.  Multimodal sentiment analysis of spanish online videos.  _IEEE Intelligent Systems_ 28, 3 (2013), 38–45.
  * Sabour et al. (2017) Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017\.  Dynamic routing between capsules. In _Advances in neural information processing systems_. 3856–3866.
  * Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997.  Bidirectional recurrent neural networks.  _IEEE transactions on Signal Processing_ 45, 11 (1997), 2673–2681.
  * Seshadri et al. (2016) Shriya Seshadri, Anand Kumar Madasamy, Soman Kotti Padannayil, and M Anand Kumar. 2016\.  Analyzing sentiment in indian languages micro text using recurrent neural network.  _IIOAB J_ 7 (2016), 313–318.
  * Soumya and Pramod (2019) S Soumya and KV Pramod. 2019.  Sentiment Analysis of Malayalam Tweets using Different Deep Neural Network Models-Case Study. In _2019 9th International Conference on Advances in Computing and Communication (ICACC)_. IEEE, 163–168.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Wang et al. (2016) Xingyou Wang, Weijie Jiang, and Zhiyong Luo. 2016\.  Combination of convolutional and recurrent neural network for sentiment analysis of short texts. In _Proceedings of COLING 2016, the 26th international conference on computational linguistics: Technical papers_. 2428–2437.
  * Xu et al. (2016) Jiacheng Xu, Danlu Chen, Xipeng Qiu, and Xuangjing Huang. 2016\.  Cached long short-term memory neural networks for document-level sentiment classification.  _arXiv preprint arXiv:1610.04989_ (2016).
  * Zhang et al. (2018b) Lei Zhang, Shuai Wang, and Bing Liu. 2018b.  Deep learning for sentiment analysis: A survey.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ 8, 4 (2018), e1253.
  * Zhang et al. (2018a) Yue Zhang, Qi Liu, and Linfeng Song. 2018a.  Sentence-state LSTM for text representation.  _arXiv preprint arXiv:1805.02474_ (2018).
  * Zhao et al. (2018) Wei Zhao, Jianbo Ye, Min Yang, Zeyang Lei, Suofei Zhang, and Zhou Zhao. 2018\.  Investigating capsule networks with dynamic routing for text classification.  _arXiv preprint arXiv:1804.00538_ (2018).
  * Zhou et al. (2019) Junhao Zhou, Yue Lu, Hong-Ning Dai, Hao Wang, and Hong Xiao. 2019.  Sentiment analysis of Chinese microblog based on stacked bidirectional LSTM.  _IEEE Access_ 7 (2019), 38856–38866.

##  Appendix A Appendix