# The Discovery of 63 Giant Radio Galaxies in the FIRST Survey

Soren Ramdhanie  Department of Astronomy, University of Wisconsin-Madison, 475
N. Charter St., Madison, WI 53703, USA  Department of Physics, University of
the West Indies, St. Augustine, Trinidad and Tobago  Soren Ramdhanie and
Brianna Sampson are summer students at the National Radio Astronomy
Observatory.  Yjan A. Gordon Department of Physics, University of Wisconsin-
Madison, 1150 University Ave, Madison, WI 53706, USA  Heinz Andernach
Thüringer Landessternwarte, Sternwarte 5, D-07778 Tautenburg, Germany
Permanent Address: Depto. de Astronomía, Univ. de Guanajuato, Callejón de
Jalisco s/n, Guanajuato, C.P. 36023, GTO, Mexico  Eric J. Hooper Department of
Astronomy, University of Wisconsin-Madison, 475 N. Charter St., Madison, WI
53703, USA  Brianna Sampson  Department of Astronomy, University of Wisconsin-
Madison, 475 N. Charter St., Madison, WI 53703, USA  Department of Physics,
University of the West Indies, St. Augustine, Trinidad and Tobago  Soren
Ramdhanie and Brianna Sampson are summer students at the National Radio
Astronomy Observatory.

(Received November 14, 2023; Accepted )

###### Abstract

Giant Radio Galaxies (GRGs) are Active Galactic Nuclei (AGN) with radio
emission that extends over projected sizes $>0.7\,$Mpc. The large angular
sizes associated with GRGs complicate their identification in radio survey
images using traditional source finders. In this Note, we use DRAGNhunter, an
algorithm designed to find double-lobed radio galaxies, to search for GRGs in
the Faint Images of the Radio Sky at Twenty cm survey (FIRST). Radio and
optical images of identified candidates are visually inspected to confirm
their authenticity, resulting in the discovery of $63$ previously unreported
GRGs.

Giant radio galaxies (654), Radio Galaxies (1343), Extragalactic radio sources
(508), Active galactic nuclei (16)

††journal: RNAAS††facilities: CDS, VLA, WISE, Sloan††software: Astropy
(Astropy Collaboration et al., 2022), NumPy (Harris et al., 2020), SAO DS9
(Joye & Mandel, 2003), Topcat (Taylor, 2005)

##  1 Introduction

Radio galaxies with a projected “Largest Linear Size” (LLS) over $700\,$kpc
are considered Giant Radio Galaxies (GRGs, e.g., Andernach et al., 2021).
Their jets should be intrinsically powerful (Wiita et al., 1989) and
relatively uninhibited (Dabhade et al., 2020a) to reach such sizes, and can
provide insights into the AGN central engine and host galaxy environment. GRGs
can be difficult to identify in radio surveys: for a classic Double Radio
Source associated with an Active Galactic Nucleus (DRAGN, Leahy, 1993), the
lobes of the radio galaxy may be detected as multiple sources despite
belonging to the same physical object. Consequently, many detectable GRGs may
remain unidentified.

Recently, Gordon et al. (2023, hereafter G23) introduced DRAGNhunter, an
algorithm that groups ‘extended components’ into physical sources, and applied
it to the Very Large Array Sky Survey (VLASS, Lacy et al., 2020), discovering
31 new GRGs in the process. Given the lower frequency ($1.4\,$GHz) and better
sensitivity to extended sources of FIRST (Becker et al., 1995), we used
DRAGNhunter on the FIRST catalog (Helfand et al., 2015) to identify further
unidentified GRGs.

##  2 Identifying Candidate GRGs

DRAGNhunter identifies DRAGNs by pairing cataloged extended radio sources
based on their separation and relative alignment, then uses the likelihood
ratio approach (McAlpine et al., 2012) to search for the probable host galaxy
in the AllWISE catalog (Cutri et al., 2014). Hosts are then cross-matched with
a number of legacy catalogs to obtain redshifts (for details see Sections 2-4
of G23, ). Selecting extended FIRST sources as having deconvolved sizes
$>5.5^{\prime\prime}$, DRAGNhunter finds $>9,000$ DRAGNs with a probable host
and redshift. We determine their LLS from their measured Largest Angular Size
(LAS) assuming a flat $\Lambda$CDM cosmology with $H_{0}=70\,$km/s/Mpc,
$\Omega_{\text{M}}=0.3$, and $\Omega_{\Lambda}=0.7$. We obtain
$\text{LLS}>0.7\,$Mpc for 213 DRAGNs, but because DRAGNhunter is an automated
algorithm, we inspect their radio and optical images to exclude any mistaken
groupings of independent sources, and ensure that the identified host and
redshift are correct. Furthermore, we manually remeasure the LAS that
DRAGNhunter automatically estimates, ensuring these truly are giants. In total
this leaves us with $80$ GRGs identified by DRAGNhunter in FIRST.

##  3 Newly Discovered Giants

We cross match these GRGs with a comprehensive list of the $>3,000$ known GRGs
previously reported by: Kuźmicz et al. (2018), Kozieł-Wierzbowska et al.
(2020), Dabhade et al. (2020a), Dabhade et al. (2020b), Galvin et al. (2020),
Kuźmicz & Jamrozy (2021), Andernach et al. (2021), Mahato et al. (2022), Oei
et al. (2023), and G23. Of our GRGs, $17$ are within $5^{\prime\prime}$ of
known GRGs and are not new discoveries. We list the remaining $63$ newly
discovered GRGs in Table 1.

Only $6$ of our GRGs have spectroscopic redshifts, all from the Sloan Digital
Sky Survey (SDSS-DR16, Ahumada et al., 2020). The remaining GRGs have
photometric redshifts (photo-zs) obtained from the Dark Energy Spectroscopic
Instrument imaging Legacy Surveys (LS-DR8, Duncan, 2022). Alternative photo-zs
are available for many of our GRGs, and the choice of photo-z may impact the
GRG classification for sources with $\text{LLS}\approx 0.7\,$Mpc. However,
Duncan (2022) test the robustness of their photo-zs for radio galaxies,
finding them to be reliable over the redshift range of our sample, thus these
are an appropriate choice here.

Although none of our $63$ sources have been reported as GRGs, they may have
been identified as extended radio galaxies in previous catalogs (e.g.,
Amirkhanyan, 2009; Proctor, 2011). For example, we measure
$\text{LAS}=96^{\prime\prime}$ for SDSS J025551.47-025939.2, whereas Kuźmicz &
Jamrozy (2021) only measure $80^{\prime\prime}$ leading to
$\text{LLS}=0.64\,$Mpc in their work. Here, diffuse structure in the northern
lobe extends beyond the hotspot used to measure the LAS by Kuźmicz & Jamrozy
(2021). Finally, we note that DRAGNhunter is biased toward finding relatively
unbent and symmetric FR II galaxies (G23). Consequently, GRGs with atypical
morphologies will likely be missed. Nonetheless, this Note demonstrates the
usefulness of automated routines in finding GRGs.

Table 1: New GRGs in FIRST

Name | RAJ2000 | DEJ2000 | rmag | $z$ | $z_{\text{type}}$ | $S_{\text{FIRST}}$ | $\log_{10}L_{\text{FIRST}}$ | LAS | LLS
---|---|---|---|---|---|---|---|---|---
| deg | deg | mag |  |  | mJy | $\text{W}\,\text{Hz}^{-1}$ | arcsec | Mpc
SDSS J000133.55$-$105738.0 | 0.3898 | -10.9606 | 20.06 | 0.675 | p | 43.60 | 26.09 | 115 | 0.81
SDSS J001217.46$+$120733.5 | 3.0728 | 12.1260 | 21.93 | 0.913 | p | 42.46 | 26.45 | 101 | 0.79
SDSS J001403.56$+$053008.1 | 3.5148 | 5.5023 | 19.82 | 0.51571 | s | 15.72 | 25.34 | 135 | 0.84
DELS J003912.82$+$002319.6 | 9.8034 | 0.3888 | 24.50 | 1.261 | p | 44.06 | 26.86 | 85 | 0.71
SDSS J005843.94$+$060352.7 | 14.6831 | 6.0646 | 22.02 | 0.778 | p | 17.34 | 25.86 | 95 | 0.71
SDSS J010541.66$-$031554.3 | 16.4236 | -3.2650 | 22.39 | 0.988 | p | 69.28 | 26.75 | 91 | 0.73
SDSS J010931.12$-$023723.8 | 17.3797 | -2.6233 | 22.29 | 1.10045 | s | 15.54 | 26.24 | 114 | 0.93
SDSS J015350.13$+$023539.5 | 28.4589 | 2.5943 | 21.03 | 0.650 | p | 65.22 | 26.22 | 112 | 0.78
SDSS J015438.76$+$120212.3 | 28.6615 | 12.0368 | 21.85 | 0.83175 | s | 75.46 | 26.58 | 146 | 1.11
SDSS J020047.38$-$023629.0 | 30.1975 | -2.6081 | 21.14 | 0.748 | p | 43.10 | 26.21 | 145 | 1.06
SDSS J023303.66$+$012837.1 | 38.2653 | 1.4771 | 22.87 | 0.935 | p | 8.04 | 25.75 | 92 | 0.72
SDSS J025551.47$-$025939.2 | 43.9645 | -2.9942 | 17.74 | 0.96949 | s | 115.62 | 26.95 | 96 | 0.76
SDSS J031516.73$+$004906.3 | 48.8197 | 0.8185 | 21.99 | 0.962 | p | 55.91 | 26.63 | 95 | 0.75
SDSS J083504.48$+$092803.7 | 128.7687 | 9.4677 | 21.09 | 0.698 | p | 48.89 | 26.25 | 114 | 0.81
SDSS J085102.17$+$045844.1 | 132.7591 | 4.9789 | 21.89 | 0.692 | p | 25.99 | 25.90 | 122 | 0.87
DELS J091150.56$-$072648.8 | 137.9607 | -7.4469 | 23.35 | 1.049 | p | 36.94 | 26.55 | 93 | 0.75
DELS J092719.46$+$242027.0 | 141.8309 | 24.3407 | 22.01 | 0.589 | p | 52.22 | 26.01 | 130 | 0.86
DELS J093016.68$+$114241.4 | 142.5695 | 11.7115 | 22.77 | 1.147 | p | 21.59 | 26.43 | 147 | 1.21
DELS J094554.69$+$223020.1 | 146.4779 | 22.5056 | 23.98 | 1.091 | p | 34.05 | 26.57 | 137 | 1.12
SDSS J100302.65$+$174145.6 | 150.7611 | 17.6960 | 22.12 | 0.963 | p | 14.62 | 26.05 | 114 | 0.90
DELS J101056.04$+$023639.2 | 152.7335 | 2.6109 | 24.09 | 1.144 | p | 16.45 | 26.31 | 116 | 0.95
SDSS J101206.08$+$184956.0 | 153.0253 | 18.8322 | 20.75 | 0.673 | p | 34.86 | 25.99 | 104 | 0.73
SDSS J101247.79$+$132703.5 | 153.1991 | 13.4510 | 21.84 | 0.665 | p | 17.04 | 25.67 | 118 | 0.83
DELS J103604.61$+$201103.4 | 159.0192 | 20.1843 | 22.51 | 0.994 | p | 93.22 | 26.89 | 139 | 1.11
DELS J105706.88$+$225621.8 | 164.2787 | 22.9394 | 23.92 | 0.988 | p | 11.78 | 25.99 | 97 | 0.77
DELS J112456.90$+$235242.2 | 171.2371 | 23.8784 | 23.41 | 1.057 | p | 13.23 | 26.12 | 133 | 1.08
DELS J115635.66$+$081655.9 | 179.1486 | 8.2822 | 23.13 | 1.034 | p | 19.86 | 26.27 | 144 | 1.16
SDSS J121008.57$+$184602.5 | 182.5357 | 18.7673 | 21.49 | 0.857 | p | 53.53 | 26.47 | 129 | 0.99
SDSS J121128.84$+$145034.0 | 182.8702 | 14.8428 | 22.92 | 1.068 | p | 41.11 | 26.62 | 105 | 0.85
SDSS J121358.95$+$000650.8 | 183.4956 | 0.1141 | 21.21 | 0.770 | p | 126.77 | 26.71 | 124 | 0.92
DELS J121729.04$+$264126.1 | 184.3710 | 26.6906 | 24.94 | 1.260 | p | 13.70 | 26.14 | 102 | 0.85
DELS J122257.93$+$084449.9 | 185.7414 | 8.7472 | 24.22 | 1.302 | p | 13.75 | 26.39 | 91 | 0.76
SDSS J123831.55$+$305354.2 | 189.6315 | 30.8984 | 21.60 | 0.654 | p | 37.25 | 25.99 | 132 | 0.92
SDSS J124608.79$+$190625.5 | 191.5367 | 19.1071 | 22.71 | 1.059 | p | 28.03 | 26.45 | 130 | 1.05
DELS J125320.32$+$162448.2 | 193.3347 | 16.4134 | 23.48 | 0.909 | p | 16.01 | 26.02 | 107 | 0.84
DELS J130058.08$+$300713.4 | 195.2420 | 30.1204 | 22.95 | 0.881 | p | 46.82 | 26.44 | 97 | 0.75
DELS J130956.06$+$132316.0 | 197.4836 | 13.3878 | 21.40 | 0.833 | p | 21.79 | 26.04 | 114 | 0.87
DELS J131931.48$+$211230.6 | 199.8812 | 21.2085 | 22.70 | 0.979 | p | 20.67 | 25.98 | 104 | 0.83
SDSS J134357.80$+$221813.2 | 205.9909 | 22.3036 | 21.44 | 0.605 | p | 61.82 | 26.12 | 110 | 0.74
SDSS J135435.29$-$042027.3 | 208.6471 | -4.3409 | 20.78 | 0.819 | p | 14.25 | 25.84 | 124 | 0.94
DELS J135703.76$+$243614.0 | 209.2657 | 24.6039 | 22.58 | 0.973 | p | 22.42 | 26.25 | 100 | 0.80
SDSS J141042.41$+$155334.9 | 212.6767 | 15.8930 | 22.49 | 0.878 | p | 46.22 | 26.43 | 92 | 0.71
DELS J142640.10$+$133051.1 | 216.6671 | 13.5142 | 23.61 | 0.805 | p | 18.92 | 25.94 | 93 | 0.70
SDSS J144238.00$+$261326.7 | 220.6583 | 26.2241 | 22.12 | 0.938 | p | 23.79 | 26.23 | 92 | 0.73
SDSS J145528.62$+$104451.1 | 223.8693 | 10.7475 | 19.95 | 0.43883 | s | 42.76 | 25.59 | 149 | 0.85
SDSS J150041.03$-$021511.7 | 225.1710 | -2.2533 | 22.42 | 0.896 | p | 27.77 | 26.24 | 95 | 0.74
DELS J151353.56$+$135919.3 | 228.4732 | 13.9887 | 23.55 | 1.003 | p | 15.06 | 26.11 | 95 | 0.76
SDSS J151751.03$+$095936.8 | 229.4627 | 9.9936 | 20.86 | 0.689 | p | 337.85 | 27.01 | 104 | 0.74
SDSS J154114.86$+$300257.5 | 235.3119 | 30.0493 | 21.64 | 0.692 | p | 7.14 | 25.34 | 122 | 0.87
SDSS J154905.62$+$264331.4 | 237.2734 | 26.7254 | 20.89 | 0.779 | p | 188.68 | 26.90 | 105 | 0.78
SDSS J155254.45$+$312656.5 | 238.2269 | 31.4491 | 21.17 | 0.738 | p | 7.74 | 25.45 | 98 | 0.71
DELS J160431.82$+$304625.3 | 241.1326 | 30.7737 | 23.09 | 1.112 | p | 13.86 | 26.20 | 100 | 0.82
DELS J162417.37$+$153937.8 | 246.0724 | 15.6605 | 23.54 | 0.921 | p | 86.42 | 26.76 | 90 | 0.71
SDSS J163051.31$+$532518.7 | 247.7138 | 53.4219 | 21.23 | 0.78579 | s | 12.20 | 25.72 | 98 | 0.73
DELS J170158.23$+$225940.6 | 255.4925 | 22.9948 | 22.94 | 0.748 | p | 12.83 | 25.69 | 121 | 0.89
SDSS J210537.99$+$101851.7 | 316.4083 | 10.3144 | 22.02 | 0.607 | p | 76.47 | 26.21 | 121 | 0.81
SDSS J212953.43$-$002404.3 | 322.4727 | -0.4012 | 20.81 | 0.762 | p | 126.52 | 26.65 | 128 | 0.94
SDSS J221607.43$-$044752.4 | 334.0310 | -4.7979 | 22.38 | 0.756 | p | 18.95 | 25.87 | 118 | 0.87
DELS J225123.88$-$103459.1 | 342.8495 | -10.5831 | 22.15 | 1.011 | p | 132.26 | 27.06 | 113 | 0.91
DELS J225125.27$-$025451.8 | 342.8553 | -2.9144 | 20.19 | 1.324 | p | 7.37 | 26.14 | 110 | 0.92
DELS J232804.71$+$001933.4 | 352.0196 | 0.3260 | 24.54 | 1.195 | p | 12.42 | 26.24 | 104 | 0.86
DELS J234027.85$+$003057.4 | 355.1161 | 0.5160 | 23.60 | 1.010 | p | 139.88 | 27.09 | 87 | 0.70
SDSS J235527.42$+$010853.5 | 358.8643 | 1.1482 | 22.05 | 0.783 | p | 10.54 | 25.65 | 113 | 0.84

Note. — For these 63 GRGs, we list the host name (1), R.A. (2), declination
(3), $r$-band magnitude from LS-DR8 (4), redshift (5), photometric (p) or
spectroscopic (s) redshift (6), flux density in FIRST (7), $1.4\,$GHz
luminosity (8), LAS (9) and LLS (10). The full table is available in machine
readable format in the online article and from
https://doi.org/10.5281/zenodo.10094175.

##

S.R. and B.S. acknowledge support from the NSF under Cooperative Agreements
No. 1647375 and 1647378, including the Radio Astronomy Data Imaging and
Analysis Lab (RADIAL) Research & Training Experience program. S.R. and B.S.
further acknowledge support from the Alfred P. Sloan Foundation’s Creating
Equitable Pathways to STEM Graduate Education program and NSF grant AST
21-50222.

## References

  * Ahumada et al. (2020) Ahumada, R., Allende Prieto, C., Almeida, A., et al. 2020, ApJS, 249, 3, doi: 10.3847/1538-4365/ab929e
  * Amirkhanyan (2009) Amirkhanyan, V. R. 2009, Astrophysical Bulletin, 64, 325, doi: 10.1134/S1990341309040026
  * Andernach et al. (2021) Andernach, H., Jiménez-Andrade, E. F., & Willis, A. G. 2021, Galaxies, 9, 99, doi: 10.3390/galaxies9040099
  * Astropy Collaboration et al. (2022) Astropy Collaboration, Price-Whelan, A. M., Lim, P. L., et al. 2022, ApJ, 935, 167, doi: 10.3847/1538-4357/ac7c74
  * Becker et al. (1995) Becker, R. H., White, R. L., & Helfand, D. J. 1995, ApJ, 450, 559, doi: 10.1086/176166
  * Cutri et al. (2014) Cutri, R. M., Wright, E. L., Conrow, T., et al. 2014, VizieR Online Data Catalog, II/328
  * Dabhade et al. (2020a) Dabhade, P., Mahato, M., Bagchi, J., et al. 2020a, A&A, 642, A153, doi: 10.1051/0004-6361/202038344
  * Dabhade et al. (2020b) Dabhade, P., Röttgering, H. J. A., Bagchi, J., et al. 2020b, A&A, 635, A5, doi: 10.1051/0004-6361/201935589
  * Duncan (2022) Duncan, K. J. 2022, MNRAS, 512, 3662, doi: 10.1093/mnras/stac608
  * Galvin et al. (2020) Galvin, T. J., Huynh, M. T., Norris, R. P., et al. 2020, MNRAS, 497, 2730, doi: 10.1093/mnras/staa1890
  * Gordon et al. (2023) Gordon, Y. A., Rudnick, L., Andernach, H., et al. 2023, ApJS, 267, 37, doi: 10.3847/1538-4365/acda30
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357, doi: 10.1038/s41586-020-2649-2
  * Helfand et al. (2015) Helfand, D. J., White, R. L., & Becker, R. H. 2015, ApJ, 801, 26, doi: 10.1088/0004-637X/801/1/26
  * Joye & Mandel (2003) Joye, W. A., & Mandel, E. 2003, in Astronomical Society of the Pacific Conference Series, Vol. 295, Astronomical Data Analysis Software and Systems XII, ed. H. E. Payne, R. I. Jedrzejewski, & R. N. Hook, 489
  * Kozieł-Wierzbowska et al. (2020) Kozieł-Wierzbowska, D., Goyal, A., & Żywucka, N. 2020, ApJS, 247, 53, doi: 10.3847/1538-4365/ab63d3
  * Kuźmicz & Jamrozy (2021) Kuźmicz, A., & Jamrozy, M. 2021, ApJS, 253, 25, doi: 10.3847/1538-4365/abd483
  * Kuźmicz et al. (2018) Kuźmicz, A., Jamrozy, M., Bronarska, K., Janda-Boczar, K., & Saikia, D. J. 2018, ApJS, 238, 9, doi: 10.3847/1538-4365/aad9ff
  * Lacy et al. (2020) Lacy, M., Baum, S. A., Chandler, C. J., et al. 2020, PASP, 132, 035001, doi: 10.1088/1538-3873/ab63eb
  * Leahy (1993) Leahy, J. P. 1993, in Jets in Extragalactic Radio Sources, ed. H.-J. Röser & K. Meisenheimer, Vol. 421, 1, doi: 10.1007/3-540-57164-7_74
  * Mahato et al. (2022) Mahato, M., Dabhade, P., Saikia, D. J., et al. 2022, A&A, 660, A59, doi: 10.1051/0004-6361/202141928
  * McAlpine et al. (2012) McAlpine, K., Smith, D. J. B., Jarvis, M. J., Bonfield, D. G., & Fleuren, S. 2012, MNRAS, 423, 132, doi: 10.1111/j.1365-2966.2012.20715.x
  * Oei et al. (2023) Oei, M. S. S. L., van Weeren, R. J., Gast, A., et al. 2023, A&A, 672, A163, doi: 10.1051/0004-6361/202243572
  * Proctor (2011) Proctor, D. D. 2011, ApJS, 194, 31, doi: 10.1088/0067-0049/194/2/31
  * Taylor (2005) Taylor, M. B. 2005, in Astronomical Society of the Pacific Conference Series, Vol. 347, Astronomical Data Analysis Software and Systems XIV, ed. P. Shopbell, M. Britton, & R. Ebert, 29
  * Wiita et al. (1989) Wiita, P. J., Rosen, A., Gopal-Krishna, & Saripalli, L. 1989, in Hot Spots in Extragalactic Radio Sources, ed. K. Meisenheimer & H.-J. Roeser, Vol. 327, 173, doi: 10.1007/BFb0036027


# Effect of Text Processing Steps on Twitter Sentiment Classification using
Word Embedding

Manar D. Samad and Nalin Khounviengxay
Department of Computer Science
Tennessee State University
Nashville, TN, USA
<EMAIL_ADDRESS>
Megan A. Witherow
Department of Electrical and Computer Engineering
Old Dominion University
Norfolk, VA, USA

###### Abstract

Processing of raw text is the crucial first step in text classification and
sentiment analysis. However, text processing steps are often performed using
off-the-shelf routines and pre-built word dictionaries without optimizing for
domain, application, and context. This paper investigates the effect of seven
text processing scenarios on a particular text domain (Twitter) and
application (sentiment classification). Skip gram-based word embeddings are
developed to include Twitter colloquial words, emojis, and hashtag keywords
that are often removed for being unavailable in conventional literature
corpora. Our experiments reveal negative effects on sentiment classification
of two common text processing steps: 1) stop word removal and 2) averaging of
word vectors to represent individual tweets. New effective steps for 1)
including non-ASCII emoji characters, 2) measuring word importance from word
embedding, 3) aggregating word vectors into a tweet embedding, and 4)
developing linearly separable feature space have been proposed to optimize the
sentiment classification pipeline. The best combination of text processing
steps yields the highest average area under the curve (AUC) of 88.4 ($\pm$0.4)
in classifying 14,640 tweets with three sentiment labels. Word selection from
context-driven word embedding reveals that only the ten most important words
in Tweets cumulatively yield over 98% of the maximum accuracy. Results
demonstrate a means for data-driven selection of important words in tweet
classification as opposed to using pre-built word dictionaries. The proposed
tweet embedding is robust to and alleviates the need for several text
processing steps.

_K_ eywords Sentiment analysis, Twitter, Stop words, Regular expression,
Emoji, Skip gram, Word2vec, Word importance

##  1 Introduction

The last decade has seen an unprecedented surge in the growth and
proliferation of online social media platforms such as Twitter, Facebook, and
Tumblr. For individuals, social media is a uniquely powerful platform for
communication, self-expression, microblogging, social networking, image and
video sharing, and opinion sharing through online word-of-mouth. As of January
2020, the world has more than 3.8 billion active social media users [Kepios
Pte. Ltd. et al., 2020] generating prodigious volumes of public opinion data.
For businesses and organizations, social media provides access to consumer
bases for targeted advertising and public opinion mining on products and
services to inform strategic business plans. An important task of public
opinion mining is sentiment analysis (SA) to disambiguate sentiment from data
shared on social media platforms. SA performs sentiment classification to
stratify social media posts based upon the polarity (positive, negative, or
neutral) of the expressed opinion in the textual data [Kharde and Sonawane,
2016, Wang et al., 2019].

A mass-producer of textual data is the popular microblogging platform Twitter,
which reports 166 million average daily active users in 2020 Q1 [Twitter,
Inc., 2020]. Twitter’s defining characteristic is its character-limited
microblog posts, called ’tweets’. Tweets have been limited to 140 Unicode
characters until 2017 when the limit has doubled to 280 characters [Boot et
al., 2019]. Due to the character limitation, Twitter users prioritize
efficiency when composing tweets by employing informal language,
abbreviations, unconventional word spellings, symbols and numerals, as well as
omitting parts-of-speech (POS) to convey the desired message [Boot et al.,
2019]. Twitter’s condensed format and associated usage characteristics make it
a challenging domain for SA unlike mining standard literature and documents.
This paper investigates the efficacy of common text processing and
classification steps on Twitter-specific SA. We hypothesize that efficient use
of word embedding with data-driven optimization can be robust to the selection
of standard text processing routines in Twitter sentiment classification.

###  1.1 Background review

The two primary approaches to SA are: 1) lexicon-based and 2) machine learning
(ML) based methods [MARTÍNEZ-CÁMARA et al., 2014, Kolchyna et al., 2015].
Lexicon-based methods use a sentiment dictionary (i.e. lexicon) that contains
words labeled with their polarity to determine the overall sentiment of the
social media post [Kolchyna et al., 2015]. Some popular lexicons include
WordNet [Princeton University, 2010], SentiWordNet [Esuli, 2013], and opinion
lexicon [Liu, 2017]. Since lexicon-based methods depend on the words used in
the social media post being present in the sentiment dictionary, these methods
suffer from low recall [MARTÍNEZ-CÁMARA et al., 2014]. Two words of different
polarity (good and bad) can appear in the same tweet that complicates SA in a
particular context. Furthermore, the noisy nature of tweets and ambiguous
usage of words across contexts makes the automatic generation of lexicons from
Twitter data unreliable [Kolchyna et al., 2015]. On the other hand, the ML
approach uses supervised optimization to learn a function that performs the
classification task based upon labeled training examples. However, the primary
limitation of ML methods is the availability of labeled social media posts for
training SA models.

SA of social media posts, especially from character-limited tweets, remains a
nontrivial task. For example, sentiment dictionaries used in lexicon-based
methods do not fully capture new domain and context specific terminologies
[Kharde and Sonawane, 2016]. Abbreviated expressions such as ’b4’, ’plz’,
’sry’, and ’gr8’ are common in Tweets. The informal linguistic style and
variety of irregular abbreviations prompted by the character-limited Twitter
environment exacerbate the limitations of the lexicon-based approach. ML-based
methods offer a favorable alternative to lexicon-based methods for Twitter SA.
However, ML models require all text to be converted to representative numeric
data. This text-to-numeric data conversion is not straightforward and requires
interdisciplinary efforts involving linguistics.

Early ML approaches to Twitter SA rely on raw word representations called
n-grams as the primary input features [MARTÍNEZ-CÁMARA et al., 2014]. An
n-gram is a sequence of n consecutive words such that single words and pairs
of words are referred to as unigrams and bigrams, respectively. A tweet is
represented in a feature vector of length equal to the number of n-grams such
that each n-gram has its unique position in the vector. The vector positions
contain frequency of corresponding n-grams or simply hold ’1’ or ’0’ depending
on the presence or absence of the n-gram in the tweet, respectively. This
construction is heavily reliant on substantial text processing and careful
feature selection steps to optimize the classifier model performance. For
example, Jianqiang et al. have extensively studied the effect of six well
known text processing steps on Twitter sentiment classification using n-gram
representations [Jianqiang and Xiaolin, 2017] as summarized in Table 1. The
reported effects are mixed across n-gram models, classification types (binary
versus three-way) and classification algorithms. Many studies routinely employ
standard text processing steps, including stemming [Singh and Gupta, 2016],
lemmatization [Dereza, 2018], removal of stop words [Ahamed and Samad, 2020]
and non-ASCII characters [Jianqiang et al., 2018] assuming that these steps
are essential for all text analysis applications. In contrast, both Bao et al.
[Bao et al., 2014] and Kolchyna et al. [Kolchyna et al., 2015] report that
stemming and lemmatization reduce Twitter sentiment classification accuracy.
Zhao et al. report no effect of removing numbers, URL, and stop words on
sentiment classification [Jianqiang, 2015].

Recently, ML-derived contextual vector representations of text (text
embedding) have created a new paradigm for ML-based SA research [Tang et al.,
2016]. Several highly efficient text-to-numeric word embedding algorithms have
been developed by field experts and industrial giants, including skip gram
[Balikas and Amini, 2016a], GloVe [Pennington et al., 2014a], and fastText of
facebook [Balikas and Amini, 2016b]. For example, the popular word2vec
routine, developed by Mikolov et al. [Mikolov et al., 2013, CHURCH, 2017], is
a highly optimized Python package that implements the skip gram model for
obtaining a contextually meaningful word embedding that enables subsequent
text analysis. In contrast to word embeddings trained for a particular
application, there are publicly-available word embeddings pre-trained from
large corpora, such as the GloVe database [Pennington et al., 2014b]. However,
such pre-trained word embeddings may not optimally capture the underlying
context (sentiment towards a subject matter) and the domain (tweets) [Yu et
al., 2017, Liu et al., 2015]. While training word embeddings directly from the
domain text is important, it requires careful selection of text processing
steps.

To the best of our knowledge, the effects of different text processing steps
have not been well studied in word embedding-based models for Twitter SA. We
hypothesize that the importance of these text processing steps depends on the
text domain (Tweets) and the contextual (sentiment analysis) representation of
words in the feature vector obtained via word embedding. A robust word
embedding is expected to be less sensitive to the selection of text processing
steps and classifier models.

Table 1: Effects of text processing steps on sentiment classification performance reported in [Jianqiang and Xiaolin, 2017] using Bag-of-Words (BoW) n-gram features. Text processing | Effects on sentiment classification
---|---
steps | classification performance
Removing URL | Barely affects the performance
Removing stop words | Useful in general. No effects on 3-way
classification using n-gram features.
positive effect on binary
classification with random forest.
Removing random words | Severely affects the performance
Removing numbers | Useful with mixed findings
Reverting repetition letter | Useful except in binary classification
using Support vector machine.
No effects on 3-way classification.
Expanding acronym | Positive effects in general

A common text processing step is stop word removal using off-the-shelf stop
word dictionaries. For example, the several decades old Van stop list [van,
1979] is used in recent Twitter SA literature [Jianqiang et al., 2018]. In
contrast, a pre-compiled list of stop words has shown negative effect on
Twitter sentiment classification [Saif et al., 2014]. We hypothesize that the
selection of stop words can be domain-specific to avoid discarding potentially
informative keywords. Random removal of words has been reported to
significantly deteriorate text classification performance [Jianqiang and
Xiaolin, 2017]. We propose that, prior to removing words, the importance of
individual words can be learned from context-derived word embeddings. Thus,
word selection or removal can be optimally made based on such data-driven
decision instead of using a default dictionary of stop words.

Furthermore, there are many non-dictionary words, symbols, and short and
colloquial language expressions on social media that can complicate SA, and
hence, are typically discarded from analysis. For example, ’wow’, ’lol’,
’omg’, emoticons, and many other non-dictionary expressions frequently appear
in tweets. The treatment of these non-dictionary but potentially informative
words requires special attention. Previous studies have used emoticon
dictionary [DCN, ] to replace emoticons (composed of ASCII characters, such as
:-) represents a happy expression) with their textual names (happy). Kolchyna
et al. manually construct a lexicon of emoticons and social media
abbreviations with associated polarities [Kolchyna et al., 2015].

The successors to emoticons, Unicode ideograms called emoji, such as , are
used by Twitter users as multimodal markers to convey affect, gestures, or
repetition, and/or to function semantically in a sentence by replacing parts
of speech [Na’aman et al., 2017, Bai et al., 2019]. However, since text
analysis is commonly performed only on ASCII characters, Unicode emojis are
often filtered out. In Redmond et al., emojis have been shown to outperform
emoticons in sentiment classification [Redmond et al., 2017]. Manual [Kralj
Novak et al., 2015] and automatic [Kimura and Katsurai, 2017] methods have
been proposed to build emoji lexicons by labeling emojis with their
corresponding sentiment. However, emoji lexicons are limited by the corpora
without learning their contextual representations in word embeddings. We
postulate that emojis may be directly represented using their individual UTF-8
encodings for learning the word embedding. In general, the use of off-the-
shelf dictionaries or lexicons, including those used for stop word removal,
emoji mapping, and obtaining pre-trained word embeddings may not provide
flexibility for custom learning and domain-specific SA optimization.

Last but not least, the strategy for aggregating multiple word vectors from
word embeddings into a constant length tweet vector is an open research
problem. In machine learning, the ordering of constituent words in the tweet
vector is important and informative. A simple concatenation of word vectors
with zero padding [Jianqiang et al., 2018] or averaging of word vectors
[Sarkar, 2016] is used in practice without identifying the best strategy. The
effect of different word vector aggregation strategies needs to be evaluated
on Twitter SA.

###  1.2 Contributions

Given the limitations in the current literature, this paper aims to
investigate the effects of text processing and classification steps on Twitter
SA. The contributions of this paper are as follows.

  * •

Stop words and other non-ASCII or Unicode characters (Table 2) are routinely
removed from corpora to emphasize more informative words. This exclusion may
inadvertently leave out elements that can be informative for SA. More
importantly, aggressive removal of words and characters can substantially
reduce a character-limited tweet to only several or even no words. This
substantial loss of data can adversely affect the performance of machine
learning models. We hypothesize that retaining and properly encoding
conventionally overlooked words and non-ASCII emojis can improve sentiment
classification performance.

  * •

Documents are often converted to feature vectors using an off-the-shelf word-
embedding dictionary. These word-embeddings are developed using large corpora
that may not capture atypical and domain-specific expressions of social media.
Instead of using a global word-embedding dictionary, we propose to develop and
analyze the word-embedding directly from the tweet data set to capture domain-
specific contexts and trending keywords.

  * •

For tweet classification purposes, machine learning models require all tweets
to be represented in equal length vectors. However, each tweet constitutes
varying numbers of words. Therefore, the challenge is to represent all tweets
in an equal length feature vector by optimally aggregating the constituent
word vectors. We compare several alternative methods of aggregating word
vectors into a constant length tweet vector to identify the best strategy.

  * •

We investigate if word embeddings can provide information about the word
importance without the supervision of classification labels. We propose a
data-driven method for estimating word importance to aid context informed
selection or removal of words in Twitter SA.

The remainder of this paper is organized as follows. Section II describes the
methodology towards our investigative study of text processing and
classification steps in Twitter SA. Section 3 reports our results on the
effect of several combinations of text processing steps, the dimensionality of
the word vectors and word importance, and different tweet embedding methods.
Section IV discusses the key findings and limitations of this paper. Section V
provides concluding remarks.

##  2 Methods

Our proposed models are evaluated using a public data set of tweets on
customer experience with several U.S. airlines [kag, ]. The data set is
comprised of tweets from February 2015 when tweets are restricted to
challenging 140-character limit. Each tweet is labeled by one of three
sentiments: 1) positive, 2) negative, and 3) neutral for three-class
classification of customer sentiments. The proposed processing and
classification steps of tweets are discussed below.

Table 2: Four processing steps for text analysis: 1) removing undesired portions of text by matching regular expression patterns, 2) stop words, 3) removing single or isolated characters using regular expression matching, 4) representing each emojis by its Unicode transformation format (UTF-8) Step | Pattern | Description
---|---|---
1 | ”htt$\backslash$S+$\backslash$ w*”, | Sub-string starting with (htt, ww)
| ”ww$\backslash$ S+$\backslash$w*” | followed by other characters.
|  | Examples: http, https, www, www.
| ”[’ ’ .]” | Remove ’apostrophe’, ’period’
| [”?”, “!”] | Isolate ’?’ and ’!’ using a space
|  | in between the word and character.
| ”&$\backslash$w+”, ”@$\backslash$w+” | Remove entire words starting
|  | with ’&’ and ’@’.
| ”[0-9]$\backslash$w+”, ”[0-9]+” | Remove numeric strings
| ”$\backslash$w[0-9]+”, | starting with a number or a letter
|  | examples: ’8Goal’, ’Go8al’, ’4567’.
| , $/$, =, $\hat{}$, $\\}$, :, $\\{$, $\|$ | List of non-alphabetic characters
| +, $\ast$, ” , $\sim$, ], [, $\hat{a}$, $\hat{e}$, $\tilde{a}$ | to be removed using regular
| -, #, …, $,;,\acute{a},\acute{e},\\_,$, ’ | expressions.
2 | Sample stop words to | [’it’, ’its’, ’this’, ’that’,
| be removed | ’are’, ’was’, ’were’,
| from corpus | ’for’, ’by’, ’so’, ’on’, ’have’, ’has’,
|  | ’had’, ’an’, ’out’, ’with’, ’as’, ’in’
|  | ’be’, ’been’, ’to’, ’the’, ’and’, ’of’,
|  | ’am’, ’is’, ’at’, ’by’, ’into’]
3 | tk = | Remove single characters
| [x for x in tk if len(x)$>$ 1] | from tokenized list of words (tk)
4 | Emojis | UTF-8 encoding of Emoji
|  |  is encoded and replaced by
|  | (b’\xf0\x9f\x98\x8a’)

###  2.1 Text processing

The text processing and cleaning steps are often arbitrarily chosen in the
literature without systematic investigation of their effects on model
performance. We study the effect of seven combinations of four text processing
steps (Table 1) prior to obtaining the word embedding. First, regular
expressions are used to remove undesired characters and words from tweets.
Examples include hyperlinks, webpages, words starting with ’@’ and ’&’,
numeric values, and special characters. Second, we create a list of stop words
including auxiliary verbs, prepositions, and pronouns that are known to bear
little or no semantic information. Note that this is an experimental list to
study the effect of excluding stop words on tweet classification. Third, we
exclude single characters from tweets. Fourth, we use the Unicode
transformation format (UTF) of individual emojis in place of the Unicode
character.

Figure 1: Classifier model training, validation, and testing steps for one of
the ten folds in 10$\times$2 nested cross-validation.

###  2.2 Word embedding

The first step of text classification is to encode semantic and contextual
properties each word into a numeric vector. We use the skip gram model,
developed within the highly optimized $word2vec$ package, to produce Twitter
context-specific word embeddings. The word-embedding is a dictionary of
numeric vectors for individual words in the corpus. The skip gram model is a
neural network with a single hidden layer that uses a word vector in the input
layer to predict a set of neighboring words in the output layer. This
prediction learns the vector representation of the entire corpus vocabulary in
the hidden layer as the word embedding. Given a vocabulary of N unique words
in a Twitter corpus, each word can be represented in a N-dimensional one-hot
encoded vector. The one-hot encoded vector has a designated position for each
word that contains ’1’ to represent the presence of the word and ’0’
otherwise. An autoencoder maps each N-dimensional input vector $X\in\Re^{N}$
(representing a single word) to a lower K-dimensional encoded space that can
be used to reproduce the N-dimensional target output vector $Y\in\Re^{N}$
(representing one-hot encoded presence of multiple neighboring words in
tweet). The encoding in the hidden layer, $Z\in\Re^{K}$, can be obtained by
learning the word embedding matrix $W\in\Re^{KXN}$ as below,

$Z=WX+Q1,$ (1)

such that Z can be mapped back to the known target Y vector as below.

$\displaystyle Y$ $\displaystyle=$ $\displaystyle W^{T}Z+Q2,$ $\displaystyle
Y$ $\displaystyle=$ $\displaystyle W^{T}(WX+Q1)+Q2$ (2)

Here, Q1 and Q2 are bias vectors of K and N dimensions, respectively. Thus,
the word embedding matrix $W^{T}$ has a K-dimensional vector representation
for each of N words corresponding to N rows.

We do not use an off-the-shelf pre-trained word-embedding or a global corpus
to obtain the word embedding. Instead, the proposed Twitter data set is used
to create the word-embedding at each training cycle to capture the
classification relevant context (airline sentiment) in tweets. This approach
is also useful for text domains that may not have large training corpus or
pre-built dictionary. For instance, predicting patient condition from
physician’s generated text reports deals with many rare and domain-specific
clinical terminologies. To strictly separate the training and test data, only
the training data set (90%) is used to yield the word embedding from the skip
gram model. This word-embedding is used to extract feature vectors for both
training (90%) and test (10%) data such that the skip gram model never sees
the test data fold. We further investigate the effect of skip gram window size
and word vector dimensionality on sentiment classification accuracy.

Algorithm 1 Measuring word importance for word selection

Input: Word matrix, W = $[x]_{mx1}$, $x\in\Re^{D}$: a word vector

Output:

  * •

Set of m L2-norms, SL2 = $\\{a\\}^{m}_{i=1}$

  * •

Set of m variances, Svar = $\\{b\\}^{m}_{i=1}$

  * •

Sorted SL2 and Svar in descending order as Ssort

  * •

Select top k word in matrix, $W_{k}=[x]_{kx1}$, (k$\leq$m).

m $\leftarrow$ length (W)

for t =1 $\rightarrow$ m do

x = W[t] and then a = $\bf{xx^{T}}$

b = variance (x)

SL2 [t] = a and then Svar [t] = b

end for

Repeat below separately for

Svar and SL2 in place of S

for r = 1 $\rightarrow$ m do

index [r] = argmax (S)

maxValue = S [argmax (S)]

S$\leftarrow$ remove maxValue from S

end for

Wk $\leftarrow$ W [index [1:k]]

###  2.3 Tweet-embedding and word importance

Each tweet of variable word-length is required to be represented in a Tweet
vector of constant length. The baseline approach is to average (Eq. 4) across
n-dimensional m word vectors to yield an n-dimensional tweet vector. We
compare the performance of this baseline approach with three other methods of
aggregation: 1) sum of vectors (Eq. 3), 2) weighted average of vectors (Eq.
5), and 3) selecting and adding the $k$ most important word vectors based on
word importance. Let $m$ number of words in a tweet form a tweet matrix
$[w_{ij}]\in\Re^{m\times D}$ where each row is a word vector. We can obtain
the tweet vector as below.

$\displaystyle Tw$ $\displaystyle=$ $\displaystyle\sum_{i=1}^{i=m}[wij],$ (3)
$\displaystyle Tw_{avg}$ $\displaystyle=$ $\displaystyle\frac{Tw}{m},$ (4)
$\displaystyle Tw_{weighted}$ $\displaystyle=$
$\displaystyle\frac{\sum_{i=1}^{i=m}\frac{1}{1+\sigma^{2}_{j}}[wij]}{\sum\frac{1}{\sigma^{2}_{j}}}$
(5)

where, $Tw\in\Re^{D}$ is a D-dimensional tweet vector and $\sigma^{2}_{j}$
represents the column variance of the $[w_{ij}]$ word matrix. Next, we sort
the words in descending order of importance to subsequently select the $k$
most important words. The cumulative contribution of the top $k$ most
important words is investigated in a sentiment classification task. The word
importance is measured using two metrics in an unsupervised way, i.e., being
unaware of the relationship between the tweet and its sentiment class. The two
metrics, the variance and L2 norm of individual word vector, are used to
determine the relative importance of individual words. Algorithm 1 shows the
computational steps for selecting $k$ most important words.

###  2.4 Model training and validation

We use a 10$\times$2 fold nested cross-validation scheme [Varma and Simon,
2006] where the data samples are split 90%-10% (training-testing) in the
10-fold outer cross-validation as shown in Figure 1. The inner 2-fold cross-
validation takes the 90% data (outer nine folds) and splits it 50%-50% for
model training and validation, respectively. The inner cross-validation yields
the best hyperparameter values for the classifier model. Using the best
hyperparameter values and the 90% data, we train the model and then test on
the left-out test data fold (10%) in the outer cross-validation. This train-
validate-test process is repeated for all ten data folds to obtain ten area
under the receiver operating characteristic curve (AUC) values for
classification. We use three one-versus-all classification models to perform
the proposed three-class sentiment classification. For each sentiment class,
the resultant AUC is obtained by averaging across ten AUC values corresponding
to ten test data folds.

Figure 2: Effect of Skip gram parameters: (a) word vector dimensionality and
(b) n-gram window size on Twitter sentiment classification performance.

##  3 Results

The Airline Sentiment Twitter data set includes 3,099, 2,363, and 9,178 tweets
labeled as neutral, positive, and negative tweets, respectively. The label
imbalance in the data set can bias the 3-way classification performance due to
the higher proportion of tweets with negative sentiment. With an imbalanced
data set, the accuracy metric appears inflated and misleading as the
classifier can identify most neutral tweets with negative sentiment. Both our
study and [Jeni et al., 2013] suggest similar observation on F1-score and the
receiver operating characteristic curve (ROC) as follows. F1-score and its
constituent precision-recall values are significantly affected by data
imbalance. The ROC includes sensitivity and specificity values for varying
thresholds on classification scores and is not affected by data imbalance. An
F1-score or a confusion matrix can be obtained for each point on the ROC.
Thus, the ROC captures the trade-off between sensitivity and specificity in
area under the curve (AUC) scores. Hence, we use AUC scores to report and
compare the classification performances. To train an unbiased classifier
model, the training data set (90%) is adjusted for imbalance using the SMOTE
algorithm [Chawla et al., 2002], which is equivalent to setting class_weight =
’balanced’ in scikit learn classifier modules. However, the left-out test data
fold (10%) is unadjusted for imbalance. Adjusting test samples for imbalance
will inflate the test accuracy due to the presence of additional
representative class samples from oversampling. For all classification tasks,
we have used the logistic regression classifier model with hyperparameters :
the c value (inverse of regularization weight) and regularization type (L1 and
L2 norms). Note that more sophisticated nonlinear classifier models, such as
random forest or gradient boosting trees, do not improve the performance over
the linear classifier model at the cost of higher computational steps as
discussed later. We discuss our findings in the following sections.

Figure 3: Effect of selecting N most important words on sentiment analysis.
Cumulative contribution of top N words selected from descending or ascending
rank order of variance and L2-norm of word vectors. Superior performance of
descending order suggests higher variance and dot products infer more
informative words.

###  3.1 Effect of text processing steps

We identify eight text processing cases (Table 3) using four text processing
steps that are shown in Table 2.

Table 3: Example results of eight Tweet processing cases. Stop word and Signal char denote stop word and single character removal, respectively. The ’#’ character is removed keeping the keyword to retain the context of hashtag. Case 3 appears to be the cleanest with most informative tokens. Number | Definition | Tokenized list of strings
---|---|---
Case 0 | No processing | [’&amp’, ’#gofundme’, ’`@`united’, ’the’, ’refund’, ’request?’,
(raw tweet) | ”functionality’s”, ’w’, ’on’, ’http://t.co/5IMDckODfx’, ’is’,
| ’broken.i’, ’get’, ’a’, ’time’, ’out’, ’error.’, ]
Case 1 | RegEx | [’gofundme’, ’the’, ’refund’, ’request’, ’?’, ’functionality’, ’s’, ’w’, ’on’,
’is’, ’broken’, ’i’, ’get’, ’a’, ’time’, ’out’, ’error’, ]
Case 2 | RegEx + Stop word | [’gofundme’, ’refund’, ’request’, ’?’, ’functionality’,’s’, ’w’, ’broken’,
|  | ’i’, ’get’, ’a’, ’time’, ’error’, ’’]
Case 3 | RegEx + Stop word | [’gofundme’, ’refund’, ’request’, ’functionality’, ’broken’, ’get’,
\+ Single Char | ’time’, ’error’]
Case 4 | RegEx+Single Char | [’gofundme’, ’the’, ’refund’, ’request’, ’functionality’, ’on’, ’is’,
\+ Encode Emoji | ’broken’, ’get’, ’time’,’out’, ’error’, b’ `\\`xf0`\\`x9f`\\`x98`\\`x8a’ ]
Case 5 | RegEx + Stop word + Single | [’gofundme’, ’refund’, ’request’, ’functionality’, ’broken’,’get’, ’
Char + Encode Emoji | time’, ’error’, b’`\\`xf0`\\`x9f`\\`x98`\\`x8a’]
Case 6 | RegEx + Encode Emoji | [’gofundme’, ’the’, ’refund’, ’request’, ’?’, ’functionality’, ’s’, ’w’, ’on’, ’is’,
’broken’, ’i’, ’get’, ’a’, time’, ’out’, ’error’,
”b’`\\`xf0`\\`x9f`\\`x98`\\`x8a’”]
Case 7 | RegEx + Stop word | [’gofundme’, ’refund’, ’request’, ’?’, ’functionality’, ’s’, ’w’, ’broken’,
\+ Encode Emoji | ’i’, ’get’, ’a’, ’time’, ’error’, ”b’`\\`xf0`\\`x9f`\\`x98`\\`x8a’”]

Case 0 generates the word-embedding and performs the classification without
involving any text processing, which equivalent to simple tokenization of the
raw tweet. Case 1 cleans raw tweets using regular expression (regEx) based
matching patterns as shown in Table 1. This RegEx-based cleaning isolates
punctuation and punctuation-spliced words such as ’broken.i’ while retaining
hashtag words and single characters including the Unicode emoji character
showing a happy face. Case 2 additionally removes stop words like ‘on’, ‘out’,
and ‘the’. Case 3 builds on Case 2 by additionally removing single characters
such as ’a’, ’i’, ’w’, ’s’ including Unicode characters and emojis. Case 4
includes the stop words (excluded in Case 3) and additionally encodes the
Unicode emoji character into its UTF-8 format
(”b’`\\`xf0`\\`x9f`\\`x98`\\`x8a’”). Case 5 combines the Case 4 processing
steps with stop word removal. Case 6 pairs the procedure from Case 1 with
emoji encoding. Case 7 follows the procedure of Case 2 and additionally
encodes the emoji character.

###  3.2 Word importance and effect of word embedding

The general intuition is that increasing data dimensionality deteriorates
classifier performance due to the ’curse-of-dimensionality’ effect. We study
the effect of word vector dimensionality on classification accuracy. Figure
2(a) shows that higher word vector dimensionality gradually adds more
predictive information until it reaches a plateau. There appears to be no
significant improvement following the word vector dimensionality of 40. Hence,
we report our subsequent analysis results using 40-dimensional word vector. In
skip gram model, the number of neighboring words to be predicted from the
input word (discussed with Eq. 2) also affects the performance of word
embedding. Figure 2(b) shows the effect of increasing the window size of
neighboring words on Twitter sentiment classification performance. A window
size of seven appears to be the best choice for our particular Twitter
classification task. The importance of individual words is calculated from the
variance and L2 norm of the word vectors. The words with the highest and
lowest variance of word vectors yield mean AUCs of 70 and 60, respectively.
This suggests that word vectors with higher variance are more informative than
those with low variance. The cumulative performance gains after including the
top N most important words and the N least important words are shown in Figure
3. The top 10 most important words in a tweet yield 98.33% of the maximum
sentiment classification accuracy that is obtained using all available words.
The L2 norm and variance of word vectors result in similar accuracy trends as
shown in Figure 3.

###  3.3 Effect of tweet embedding

We compare the performance of three proposed tweet embeddings with that of the
baseline vector averaging method in Twitter sentiment classification. Table 4
shows 10$\times$2 nested cross-validation AUC-ROC for four tweet embeddings
and eight tweet processing cases. The baseline averaging of word vectors
performs the worst among all tweet embeddings. Case 0, without any text
processing, yields low accuracy with the highest standard deviation. The sum
of the word vectors yields the most accurate classification of sentiment.
However, weighted average slightly outperforms the summation method in several
cases and is a superior choice over baseline averaging. The best
classification performance is obtained using text processing Case 6, which is
regular expression-based processing followed by encoding of emojis. This
suggests the value of encoding and incorporating emojis in Twitter SA. The
worst classification performance is obtained using averaging of word vectors
in Case 3. Case 3 excludes all the stop words, single characters, and emojis
to yield cleanest tokens in Table 3. Case 3 text processing scheme invariably
yields the worst performance for all types of tweet embedding. This
observation suggests that aggressive removal of stop words and characters has
negative effects on word embedding, classifier model training and performance.
Using the 10 most important words (based on word vector variance) and
summation of these word vectors, the classification performance is comparable
to other tweet embedding methods.

.  Case SUM AVG WAVG VECSEL Best 0 87.71(1.0) 86.21(1.2) 87.47(1.1) 86.80(1.3)
SUM 1 88.21(0.5) 86.48(0.4) 88.13(0.4) 86.35(0.6) SUM 2 88.15(0.4) 86.39(0.4)
88.18(0.3) 86.81(0.6) WAVG 3 87.31(0.4) 85.32(0.4) 87.32(0.4) 86.32(0.6) WAVG
4 87.44(0.5) 85.54(0.6) 87.40(0.6) 86.46(0.8) SUM 5 87.52(0.4) 85.51(0.5)
87.37(0.4) 86.49(0.7) SUM 6 88.37(0.4) 86.66(0.4) 88.33(0.4) 86.88(0.5) SUM 7
88.21(0.5) 86.65(0.5) 88.28(0.4) 86.98(0.5) WAVG Best Case 6 Case 6 Case 6
Case 7

Table 4: Effects of eight text processing cases and four tweet embeddings on
sentiment classification performance. Values represent average area under the
receiver operating characteristic curve (AUC) across 10 cross-validation folds
with standard deviation in the parenthesis. AVG = average, WAVG = weighted
average, VECSEL = selecting top 10 words from the descending order of word
vector variance. The best and worst AUC scores are highlighted for comparison.

We further compare classification performance for individual sentiment
classes. For the best (sum + case 5) and worst (average + case 3) scenarios,
we show their corresponding confusion matrices in Table 5. The average
accuracies of the best and worst scenarios are 73.33% and 69%, respectively.
The confusion matrices show that negative tweets are confused as positive
tweets much less than the positive tweets are confused as negatives. The worst
model heavily confuses neutral tweets as negative tweets. Classifying neutral
tweets is the most difficult task as it is a gray zone between positive and
negative sentiment classes. Hence, the accuracy of neutral tweet
classification accounts for most deviations between the best scenario (71%)
and the worst scenario (60%).

.  Best Actual Predicted As F1-score Negative Neutral Positive Negative 76%
17% 7% 0.82 Model Neutral 17% 71% 12% 0.60 Positive 11% 16% 73% 0.67 Worst
Actual Predicted As F1-score Negative Neutral Positive Negative 78% 16% 6%
0.83 Model Neutral 26% 60% 14% 0.56 Positive 14% 17% 69% 0.66

Table 5: Confusion matrices and F1-scores for three sentiment classification.
The best (sum + case 6) and worst (avg + case 3) models (Table 3) are used as
examples.

.  SVM-Linear Logistic SVM-RBF Random Gradient Boosting Trees Regression
Forest Hyper C = [0.01, 0.5, [’L1’, ’L2’] C = [0.01, 0.5, 0.05, Num. of trees
Num. of trees=[50, 80, 110] para- 0.05, 0.8, 1, 5] C = [0.01, 0.5,
$0.8,1,5],~{}\gamma$ = [0.1, 0.3, =[10, 30, 50, learning rate = [0.1, 0.5],
meter 0.05, 0.8, 1, 5] 0.5, 0.01, 0.02, 0.05] 70, 90, 110] max_depth = [2, 5,
10] Time 70 minutes 6 minutes 756 minutes 13 minutes 312 minutes 88.4 (0.8)
88.4 (0.4) 88.6 (0.6) 87.7 (0.5) 87.7 (0.6) AUC

Table 6: Comparison of classifier model performances in Twitter sentiment
classification. Time is in minutes required for the nested cross-validation
that includes generation of the word embedding at each fold iteration. SVM-RBF
stands for support vector machine with radial basis function kernel.

.  Tokenized Tweet [’i’, ’am’, ’appalled’, ’at’, ’your’, ’lack’, ’of’,
’communication’, ’during’, ’our’, ’awful’, ’journey’, ’on’, ’ewr’, ’to’,
’bos’, ’hr’, ’delays’, ’need’, ’to’, ’be’, ’explained’, ’better’] Ordered list
of words [journey (0.49), appalled (0.46), explained (0.42), communication
(0.31), lack (0.31), with importance scores BOS (0.24), awful (0.24), hr
(0.24), EWR (0.21),during (0.20), delays (0.18), better (0.16), our (0.16),
need (0.13),am (0.13), be (0.09), at (0.09), of (0.08), your (0.07), I (0.07),
on (0.05), to (0.05)]

Table 7:  An example of tokenized tweet and ordered list of words based on
importance score derived from word-embedding. The trivial words appear with
low importance scores.

##  4 Discussion

This paper compares the performance of different text processing strategies to
investigate an optimal pipeline for Twitter sentiment classification. The key
findings of this paper are: 1) Reg-Ex based text cleaning is sufficient since
removal of stop words and characters negatively affects model training and
classification performance, 2) retaining and encoding non-ASCII emoji and
previously overlooked characters are important to improve Twitter sentiment
classification, 3) sum of word vectors is preferred over averaging to yield
tweet vectors for classification, 4) the skip gram model-derived word
embedding provides with the information about word importance for data-driven
and domain-specific selection of words. In the following subsections, we
provide further discussion of our observed results.

#### Off-the-shelf text cleaning routines affect sentiment classification
performance:

Our results show that skip gram model-derived word embeddings are rich in
predictive information and robust to noise in several ways. Even the stop
words that are routinely removed in text analysis show low but useful additive
predictive information. This is evident in the performance of text processing
Case 3. In Table 2, Case 3 removes all stop words and single characters to
reduce the original tokenized list of 18 tokens (Case 0) to presumably 8 most
informative tokens. In fact, Case 3 with the least number of tokens performs
the worst among all the cases. Therefore, a basic regular expression-based
text cleaning is sufficient (Case 1) without targeting stop words and special
characters in character-limited tweet classification. Leveraging special
characters like emoji is the key to improving the Twitter sentiment
classification performance. This explains why Case 6 (Case 1 plus emoji
encoding) appears as the best performing case. On the other hand, averaging
the word vectors lowers the variance in the resultant tweet vector. The
reduction in data variance may negatively affect sentiment classification
performance. This effect is improved by using the weighted average of word
vectors where the variance estimate is considered in the average calculation.
Overall, the sum of word vectors retains the most predictive information
within a tweet vector. This is particularly useful when no vector is available
in the word embedding for a test keyword.

The dimensionality of word vectors is often set default as large as 200
[Jianqiang and Xiaolin, 2017] without validating if such large vector
dimensionality is the optimal choice. We show that higher vector
dimensionality gradually adds more predictive information up to 40 dimensions
beyond which computational cost is increased with no added value to the model
performance (Figure 2(a)). Note that the performance difference in Table 4
between the cases may appear low in terms of percentage. This is true for the
VECSEL and average (avg) based tweet embedding. However, for sum and weighted
average (wavg) based tweet vector representations, the standard deviation
appears low (0.4%) with difference in AUC between 1% and 3%. This suggests a
significant improvement in classification performance across the text
processing cases (Case 0 to Case 7). For example, the accuracy difference
between the best and worst models (Table 5) is 4.33%. This implies that the
best model is able to correctly classify 4.33% of total 14,640 tweets or 634
tweets more than those of the worst model. This improvement is substantial in
terms of sample size and important for real-world application.

#### Linear classifier models are preferable:

The skip gram model obtains quite informative word embedding that efficiently
captures domain contexts. The aggregation (sum or averaging) of these word
vectors results in tweet embeddings that yield linearly separable sentiment
classes for which a linear classifier model is sufficient. Unlike linear
models, nonlinear classifier models are not only demanding in time and
computational resources but also prone to overfitting for this particular
application. This is evident in the comparative results obtained using several
well known classifier models in Table 6. Both linear models (SVM-Linear and
Logistic Regression) yield the same accuracy and slightly outperform the
random forest and gradient boosting tree models. The nonlinear SVM-RBF yields
slight improvement at the cost of very high computational time.

#### Word-embedding for word selection:

The skip gram model derived word vectors (word embedding) contain useful
information that can be leveraged to measure individual word importance in a
tweet. Since the word embedding is obtained using domain-specific text instead
of a global corpus, the word vectors are expected to learn the domain contexts
and importance. Individual words in tweets cumulatively add predictive
information to the sentiment classification performance as observed in Figure
3. Figure 3 suggests that every word bears a contribution (that may be more or
less informative) to the overall classification performance. Therefore, using
a default dictionary of stop words for all applications, without recognizing
contextual importance of individual words, can limit the overall sentiment
classification performance. Instead, we recommend the learning of domain
(Twitter) and application (sentiment classification) specific stop words from
word embeddings. Table 7 shows the ordering of all words in a tweet example
based on proposed importance scores from word embedding. Intuitively, it shows
that conventional stop words (am, be, at, I, your, on, to) are bearing low
importance scores based on their word embedding. Thus, a data-driven approach
for stop word selection may better guide for context-specific text
classification. Even in the case when some test words are unavailable in the
trained word embedding dictionary, existing test words can still cumulatively
contribute to the final tweet classification.

###  4.1 Limitations

The Skip gram model, implemented under the word2vec package, is a powerful
model for learning the word embedding that ultimately sets the upper limit on
accuracy of our Twitter sentiment classification. The goal of this paper is
not to propose a new word embedding algorithm or to compare several word
embedding algorithms but rather to study the effect and contributions of
different text processing schemes when using an efficient word embedding.
Indeed, there are more recent word embeddings that may outperform the skip
gram model at the cost of higher computational needs. However, the effects of
text processing steps are likely to impact all other word embedding models in
more or less a similar fashion that we do not investigate in this paper. The
paper compares simple statistics (sum, average, weighted average) to aggregate
word vectors into a tweet vector. While this simple strategy works, there are
more sophisticated ways to aggregate word vectors into a sentence or tweet
vector that are not commonly used in the literature and practice.

Our results cannot be fairly compared with those reported using the same data
set for sentiment analysis. A single 80-20 split of samples may yield biased
accuracy if the 20% data include easy-to-classify samples. A mere 10-fold
cross validation may not yield separate train-validate-test phases for
unbiased model selection and accuracy reporting. Hence, we have proposed 10x2
nested cross-validation to yield an unbiased estimate of classification
performance. The overall F1-score, across all three classes, is heavily
affected by data imbalance. The existing literature does not report how the
imbalance is adjusted or if the training or entire data set is subjected to
imbalance adjustment. The AUC of ROC is unaffected by data imbalance and
provides flexibility to choose an appropriate classifier model considering
sensitivity and specificity trade-offs as required for a particular
application.

##  5 Conclusions

In this paper, we conduct an investigative study of text processing and
classification steps to identify optimal strategies for Twitter SA. We
conclude that the most aggressive text cleaning routine that removes tokens
through stop word and single character removal yields the worst performance.
This observation is supported by the finding that all word embedding vectors,
including those of stop words, contain additive predictive information. We
also find that the inclusion of emoji improves sentiment classification
performance. Further analysis finds an optimal dimensionality of word vectors
and identifies that the sum of word vectors is superior to averaging for tweet
vector formation. These observations recommend a data-driven approach to text
processing and word selection as opposed to using off-the-shelf routines and
dictionary-based mapping. The skip gram-based word embeddings not only learn
domain-specific contexts but also facilitate data-driven informed decision
making and alleviate the need for many standard text processing steps. The
robustness of skip gram word embedding is evident even in cases when certain
words are not in the training corpus. The word embedding also yields a
linearly separable feature space that is more suitable for developing fast and
generalizable sentiment classifier models.

## References

  * [1] Wikipedia. List of Emoticons.  URL https://en.wikipedia.org/wiki/List_of_emoticons.
  * [2] Twitter US Airline Sentiment — Kaggle.  URL https://www.kaggle.com/crowdflower/twitter-airline-sentiment.
  * van [1979] Information Retrieval - C. J. Van Rijsbergen, C. J. Keith Van Rijsbergen - Google Books, 1979.  URL https://books.google.com/books/about/Information_ Retrieval.html?id=t-pTAAAAMAAJ.
  * Ahamed and Samad [2020] Sabber Ahamed and Manar Samad.  Information Mining for COVID-19 Research From a Large Volume of Scientific Literature.  apr 2020.  URL http://arxiv.org/abs/2004.02085.
  * Bai et al. [2019] Qiyu Bai, Qi Dan, Zhe Mu, and Maokun Yang.  A Systematic Review of Emoji: Current Research and Future Perspectives.  _Frontiers in Psychology_ , 10:2221, 2019.  ISSN 1664-1078.  doi: 10.3389/fpsyg.2019.02221.  URL https://www.frontiersin.org/article/10.3389/fpsyg.2019.02221.
  * Balikas and Amini [2016a] Georgios Balikas and Massih-Reza Amini.  An empirical study on large scale text classification with skip-gram embeddings.  jun 2016a.  URL http://arxiv.org/abs/1606.06623.
  * Balikas and Amini [2016b] Georgios Balikas and Massih-Reza Amini.  An empirical study on large scale text classification with skip-gram embeddings.  jun 2016b.  URL http://arxiv.org/abs/1606.06623.
  * Bao et al. [2014] Yanwei Bao, Changqin Quan, Lijuan Wang, and Fuji Ren.  The role of pre-processing in twitter sentiment analysis.  In _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , volume 8589 LNAI, pages 615–624. Springer Verlag, 2014.  ISBN 9783319093383.  doi: 10.1007/978-3-319-09339-0-62.  URL https://link.springer.com/chapter/10.1007/978-3-319-09339-0_62.
  * Boot et al. [2019] Arnout B Boot, Erik Tjong Kim Sang, Katinka Dijkstra, and Rolf A Zwaan.  How character limit affects language usage in tweets.  _Palgrave Communications_ , 5(1):76, 2019.  ISSN 2055-1045.  doi: 10.1057/s41599-019-0280-3.  URL https://doi.org/10.1057/s41599-019-0280-3.
  * Chawla et al. [2002] Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer.  SMOTE: Synthetic minority over-sampling technique.  _Journal of Artificial Intelligence Research_ , 16:321–357, jan 2002.  ISSN 10769757.  doi: 10.1613/jair.953.  URL https://www.jair.org/index.php/jair/article/view/10302.
  * CHURCH [2017] KENNETH WARD CHURCH.  Word2vec.  _Natural Language Engineering_ , 23(1):155–162, 2017.  doi: 10.1017/S1351324916000334.
  * Dereza [2018] Oksana Dereza.  Lemmatization for ancient languages: Rules or neural networks?  In _Communications in Computer and Information Science_ , volume 930, pages 35–47. Springer Verlag, oct 2018.  ISBN 9783030012038.  doi: 10.1007/978-3-030-01204-5-4.  URL https://link.springer.com/chapter/10.1007/978-3-030-01204-5_4.
  * Esuli [2013] Andrea Esuli.  The User Feedback on SentiWordNet.  pages 1–7, 2013.  URL http://arxiv.org/abs/1306.1343.
  * Jeni et al. [2013] László A. Jeni, Jeffrey F. Cohn, and Fernando De La Torre.  Facing imbalanced data - Recommendations for the use of performance metrics.  In _Proceedings - 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, ACII 2013_ , volume 2013, pages 245–251. NIH Public Access, 2013.  ISBN 9780769550480.  doi: 10.1109/ACII.2013.47.  URL /pmc/articles/PMC4285355/?report=abstract https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4285355/.
  * Jianqiang [2015] Z. Jianqiang.  Pre-processing boosting twitter sentiment analysis?  In _2015 IEEE International Conference on Smart City/SocialCom/SustainCom (SmartCity)_ , pages 748–753, 2015.
  * Jianqiang and Xiaolin [2017] Z Jianqiang and G Xiaolin.  Comparison Research on Text Pre-processing Methods on Twitter Sentiment Analysis.  _IEEE Access_ , 5:2870–2879, 2017.  ISSN 2169-3536 VO - 5.  doi: 10.1109/ACCESS.2017.2672677.
  * Jianqiang et al. [2018] Z Jianqiang, G Xiaolin, and Z Xuejun.  Deep Convolution Neural Networks for Twitter Sentiment Analysis.  _IEEE Access_ , 6:23253–23260, 2018.  ISSN 2169-3536 VO - 6.  doi: 10.1109/ACCESS.2017.2776930.
  * Kepios Pte. Ltd. et al. [2020] Kepios Pte. Ltd., We Are Social Ltd., and Hootsuite Inc.  Digital 2020: Global Digital Yearbook.  Technical report, 2020.  URL https://datareportal.com/reports/digital-2020-global-digital-yearbook.
  * Kharde and Sonawane [2016] Vishal A. Kharde and S.S. Sonawane.  Sentiment Analysis of Twitter Data: A Survey of Techniques.  _International Journal of Computer Applications_ , 139(11):5–15, 2016.  doi: 10.5120/ijca2016908625.
  * Kimura and Katsurai [2017] Mayu Kimura and Marie Katsurai.  Automatic Construction of an Emoji Sentiment Lexicon.  In _Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017_ , ASONAM ’17, pages 1033–1036, New York, NY, USA, 2017. Association for Computing Machinery.  ISBN 9781450349932.  doi: 10.1145/3110025.3110139.  URL https://doi.org/10.1145/3110025.3110139.
  * Kolchyna et al. [2015] Olga Kolchyna, Tharsis T P Souza, Philip Treleaven, and Tomaso Aste.  Twitter Sentiment Analysis: Lexicon Method, Machine Learning Method and Their Combination, 2015.
  * Kralj Novak et al. [2015] Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič.  Sentiment of Emojis.  _PLOS ONE_ , 10(12):1–22, 2015.  doi: 10.1371/journal.pone.0144296.  URL https://doi.org/10.1371/journal.pone.0144296.
  * Liu [2017] Bing Liu.  A list of positive and negative opinion words or sentiment words for English, 2017.  URL http://www.cs.uic.edu/ liub/FBS/sentiment-analysis.html#datasets.
  * Liu et al. [2015] Shenghua Liu, Xueqi Cheng, Fuxin Li, and Fangtao Li.  TASC: Topic-adaptive sentiment classification on dynamic tweets.  _IEEE Transactions on Knowledge and Data Engineering_ , 27(6):1696–1709, jun 2015.  ISSN 10414347.  doi: 10.1109/TKDE.2014.2382600.
  * MARTÍNEZ-CÁMARA et al. [2014] EUGENIO MARTÍNEZ-CÁMARA, M. TERESA MARTÍN-VALDIVIA, L. ALFONSO UREÑA-LÓPEZ, and A RTURO MONTEJO-RÁEZ.  Sentiment analysis in Twitter.  _Natural Language Engineering_ , 20(1):1–28, jan 2014.  ISSN 1351-3249.  doi: 10.1017/S1351324912000332.
  * Mikolov et al. [2013] Tomas Mikolov, Kai Chen, Gregory S Corrado, and Jeffrey Dean.  Efficient Estimation of Word Representations in Vector Space.  _CoRR_ , abs/1301.3, 2013.
  * Na’aman et al. [2017] Noa Na’aman, Hannah Provenza, and Orion Montoya.  MojiSem: Varying linguistic purposes of emoji in (Twitter) context.  In _Proceedings of ACL 2017, Student Research Workshop_ , pages 136–141, 2017.  doi: 10.18653/v1/P17-3022.  URL https://doi.org/10.18653/v1/P17-3022.
  * Pennington et al. [2014a] Jeffrey Pennington, Richard Socher, and Christopher D. Manning.  GloVe: Global Vectors for Word Representation.  pages 1532–1543, 2014a.  doi: 10.3115/V1/D14-1162.  URL https://www.aclweb.org/anthology/D14-1162.
  * Pennington et al. [2014b] Jeffrey Pennington, Richard Socher, and Christopher D. Manning.  Pre-trained Word Vectors, 2014b.  URL https://nlp.stanford.edu/projects/glove/.
  * Princeton University [2010] Princeton University.  About WordNet, 2010.  URL https://wordnet.princeton.edu/citing-wordnet.
  * Redmond et al. [2017] M Redmond, S Salesi, and G Cosma.  A novel approach based on an extended cuckoo search algorithm for the classification of tweets which contain Emoticon and Emoji.  In _2017 2nd International Conference on Knowledge Engineering and Applications (ICKEA)_ , pages 13–19, oct 2017.  doi: 10.1109/ICKEA.2017.8169894.
  * Saif et al. [2014] Hassan Saif, Miriam Fernandez, Yulan He, and Harith Alani.  On Stopwords, Filtering and Data Sparsity for Sentiment Analysis of Twitter.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation_ , pages 810–817. European Language Resources Association, may 2014.  URL http://www.lrec-conf.org/proceedings/lrec2014/pdf/292_Paper.pdf.
  * Sarkar [2016] Dipanjan Sarkar.  _Text Analytics with Python_.  Apress, 2016.  doi: 10.1007/978-1-4842-2388-8.
  * Singh and Gupta [2016] Jasmeet Singh and Vishal Gupta.  Text stemming: Approaches, applications, and challenges.  _ACM Computing Surveys_ , 49(3):1–46, sep 2016\.  ISSN 15577341.  doi: 10.1145/2975608.  URL https://dl.acm.org/doi/10.1145/2975608.
  * Tang et al. [2016] Duyu Tang, Furu Wei, Bing Qin, Nan Yang, Ting Liu, and Ming Zhou.  Sentiment Embeddings with Applications to Sentiment Analysis.  In _IEEE Transactions on Knowledge and Data Engineering_ , volume 28, pages 496–509. IEEE Computer Society, feb 2016.  doi: 10.1109/TKDE.2015.2489653.
  * Twitter, Inc. [2020] Twitter, Inc.  Twitter Announces First Quarter 2020 Results.  Technical report, San Francisco, CA, 2020.
  * Varma and Simon [2006] Sudhir Varma and Richard Simon.  Bias in error estimation when using cross-validation for model selection.  _BMC Bioinformatics_ , 7(1):91, 2006.  ISSN 14712105.  doi: 10.1186/1471-2105-7-91.
  * Wang et al. [2019] Lei Wang, Jianwei Niu, and Shui Yu.  SentiDiff: Combining Textual Information and Sentiment Diffusion Patterns for Twitter Sentiment Analysis.  _IEEE Transactions on Knowledge and Data Engineering_ , apr 2019.  ISSN 1041-4347.  doi: 10.1109/tkde.2019.2913641.
  * Yu et al. [2017] Liang-Chih Yu, Jin Wang, K Robert Lai, and Xuejie Zhang.  Refining Word Embeddings for Sentiment Analysis.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 534–539, Copenhagen, Denmark, sep 2017. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1056.  URL https://www.aclweb.org/anthology/D17-1056.