# #REval: A Semantic Evaluation Framework for Hashtag Recommendation

Areej Alsini
Umm Al-Qura University
<EMAIL_ADDRESS>
&Du Q. Huynh
The University of Western Australia
<EMAIL_ADDRESS>
&Amitava Datta
The University of Western Australia
<EMAIL_ADDRESS>

###### Abstract

Automatic evaluation of hashtag recommendation models is a fundamental task in
many online social network systems. In the traditional evaluation method, the
recommended hashtags from an algorithm are firstly compared with the ground
truth hashtags for exact correspondences. The number of exact matches is then
used to calculate the hit rate, hit ratio, precision, recall, or F1-score.
This way of evaluating hashtag similarities is inadequate as it ignores the
semantic correlation between the recommended and ground truth hashtags. To
tackle this problem, we propose a novel semantic evaluation framework for
hashtag recommendation, called #REval. This framework includes an internal
module referred to as BERTag, which automatically learns the hashtag
embeddings. We investigate on how the #REval framework performs under
different word embedding methods and different numbers of synonyms and
hashtags in the recommendation using our proposed #REval-hit-ratio measure.
Our experiments of the proposed framework on three large datasets show that
#REval gave more meaningful hashtag synonyms for hashtag recommendation
evaluation. Our analysis also highlights the sensitivity of the framework to
the word embedding technique, with #REval based on BERTag more superior over
#REval based on FastText and Word2Vec.

##  1 Introduction

It is important to have frameworks that can automatically and accurately
evaluate the performances of algorithms. In the hashtag recommendation
research area, algorithms that can help recommend hashtags to the user while
the tweet is being written have received a great amount of attention in recent
years. This is due to the increased numbers of online messaging systems and
online users every year. Hashtags are innovative social identifiers prefixed
with the # sign. They work as annotations to their corresponding tweets. They
enable users to quickly find tweets of a topic in mind and they help to
promote engagement among users. Although hashtag recommendation models have
received considerable interest due to the increased popularity of social media
platforms Alsini et al. (2021), studies on their evaluation methods are still
in their infancy.

Current evaluation methods for hashtag recommendation mainly focus on the
accuracy-based metrics Kywe et al. (2012); Zhao et al. (2016); Li et al.
(2016); Alsini et al. (2017); Kowald et al. (2017); Zhang et al. (2017); Shi
et al. (2018); Alsini et al. (2020a, b) such as hit rate, hit ratio Alsini et
al. (2020b), precision, recall, and F1 scores. These metrics are dependent on
the exact matching of the recommended hashtags with the actual list of
hashtags used in the tweets of the user. For evaluation purpose, we refer to
this actual list of hashtags as the ground truth hashtags. Some other
researchers used human evaluation Mazzia and Juett (2011) to assess their
hashtag recommendation models because the suggested hashtags did not match the
ground truth hashtags even though they were relevant.

Evaluating hashtag recommendations is a very challenging task, as illustrated
in the following tweets taken from the #Covid19 thread in Twitter:

  1. (1)

Know the facts about #Covid19…Don’t panic #StayHomeSA #StaySafe
#21DayLockdown.

  2. (2)

#Covid19: A Crucial Impact On #Defence #Industry.

  3. (3)

#Covid19 Together Let’s #FightCovid19 #FlattenTheCurve #StopTheSpread.

  4. (4)

Please list #covid19 helpline numbers various states in the U.S. #covid2019
#corona #coron…

These tweets use 2-4 hashtags. For example in tweet (1), the ground truth
hashtags are #Covid19, #StayHomeSA, #StaySafe, and #21DayLockdown. It is clear
from these examples that the hashtags used in all these tweets share similar
meaning with the hashtags #Covid_19 and #coronavirus. However, if a
recommendation model recommended these two hashtags, then the traditional
method of evaluation, which looks for exact matches of the character strings,
would deem them as incorrect.

Another challenge is that these hashtags are user-generated contents which
make them prone to spelling mistakes. Some hashtags might be connected
mistakenly with other words or punctuation if no space is left after writing
the hashtag. This is shown clearly in the hashtags #Covid19…Don’t, #Covid19:
and #coron… used in tweets (1), (2) and (4), respectively. With these hashtags
in the ground truth and the use of the traditional method of evaluation,
hashtag recommendation models cannot be evaluated properly.

Thus, there is a need for a semantic evaluation method that can evaluate
hashtag recommendation models based on the semantic correlation of the
recommended hashtags with their corresponding ground truth ones. Our research
contributions for tackling the challenges described above are summarised
below:

  * •

We propose #REval, a semantic evaluation framework that evaluates the
performance of hashtag recommendation models by incorporating hashtag
synonyms. Based on the hit ratio used in Alsini et al. (2020b), we introduce a
new measure referred to as #REval-hit-ratio for synonym evaluation.

  * •

We propose a method that automatically constructs a thesaurus of hashtags
where every hashtag is associated with a set of hashtag synonyms that share
similar meanings.

  * •

We propose a hashtag embedding module called BERTag, which encodes hashtags
using the tweet embeddings generated by the transformer-based model BERTweet
Nguyen et al. (2020). Our encoded hashtags are represented together with
tweets in the same semantic space.

  * •

To evaluate the performance of our BERTag embeddings, we compare them with two
popular word embedding techniques, namely Word2Vec and FastText, on three
large datasets, two of which are general and the remaining one is domain-
specific. Our evaluation results show that the BERTag embeddings give more
relevant hashtag synonyms.

The paper is organised as follows. Section 2 reviews studies related to
semantic evaluation methods in text-based problems, domain modelling and
synonym identification, hashtag representations, and transformer-based models.
Our proposed framework #REval is detailed in Section 3. Experimental setups,
details of the datasets, results, and discussion are presented in Section 4.
Finally, concluding remarks and future work are given in Section 5.

##  2 Literature Review

###  2.1 Semantic Evaluation Methods for Text Analysis

There are various automatic language-based evaluation metrics. In the context
of machine translation, BiLingual Evaluation Understudy (BLEU) Papineni et al.
(2002) is an automatic evaluation method. It matches the n-gram vectors of a
given translated sentence (known as candidate) with the n-gram vectors of
multiple human translations (known as references) to yield a weighted
geometric mean of all the n-gram precisions. Thus, the more is the number of
matches, the higher is the BLEU score. METEOR Banerjee and Lavie (2005) is
another automatic method for evaluating the outputs of machine translation
algorithms. It gives further improvement to BLEU by using additional elements,
such as stemming and synonym matching, in the evaluation process. While BLEU
scores are defined based on precisions only, METEOR scores are defined as the
harmonic means of unigram precisions and recalls, with a heavier weight on the
latter. In the context of text summarization, Recall-Oriented Understudy for
Gisting Evaluation (ROUGE) Lin (2004) is introduced as a set of automatic
metrics, which evaluate the quality of a candidate summary by matching it to
the references created by humans using the n-grams, word sequences, and word
pairs.

Evaluation metrics have also gained increasing interest in the image
captioning research area. CIDEr Vedantam et al. (2015) and SPICE Anderson et
al. (2016) are image captioning evaluation methods that incorporate human
consensus in the description of an image. Precision and recall are considered
when estimating the similarity between the candidate sentence and the
sentences annotated by humans. CIDEr measures the similarity between the
n-gram of words occurring in the candidate sentence and all the reference
sentences. SPICE Anderson et al. (2016), on the other hand, constructs a
semantic parsing that converts the captions into scene graphs. It has been
shown to outperform all previously mentioned metrics. An interesting finding
is that the more is the number of reference sentences, the higher is the SPICE
score.

In all the evaluation methods mentioned above, human references were
considered as the base of evaluation. For hashtag recommendation, the ground
truth hashtags, which are the hashtags present in the test tweets, are used as
references for evaluation. In this regard, the only available method is to
perform exact matching between the recommended hashtags and the hashtags from
the ground truth. In the literature, accuracy-based metrics such as hit rate,
precision, recall, hit ratio, and F1-score are commonly used for evaluating
hashtag recommendation models Alsini et al. (2020b): Hit rate considers the
recommendation as a hit if at least a single recommended hashtag has an
identical match with a hashtag from the ground truth; precision and recall
measure the proportion of correctly matched hashtags in the recommended
hashtags and in the ground truth hashtags, respectively; hit ratio calculates
the ratio of the hits against the minimum number of hashtags in the
recommendation and in the ground truth; F1-score is a harmonic mean of
precision and recall. A shortcoming of these metrics is that they do not take
into consideration the semantic correlation in the evaluation of hashtag
recommendation. The aim of our proposed framework is to overcome this
shortcoming.

###  2.2 Domain Modeling and Synonym Identification

Domain modelling schemes represent data in one of the following structured
form Gilchrist (2003): Taxonomy, ontology, and thesaurus. In taxonomy,
multiple levels of concept generality are represented using a hierarchical
tree structure, with generic terms at the top of the tree to specific terms at
the bottom; in ontology of a given domain, of interest is to retrieve a formal
and explicit specification of the shared conceptualisation; in thesaurus
Schwarz (2005), every term has a reference to all the other terms that have
similar meanings to it. The relationship of a concept term with its synonyms
in a thesaurus can be hierarchy, association, or equivalence Schwarz (2005).
Synonym expansion has been used in many applications, such as information
retrieval Al-Khateeb et al. (2017) and search-related domains Boteanu et al.
(2019). Botenanu et al. Boteanu et al. (2019) enhanced the search for Amazon
products via building large taxonomies of shopping items.

In the literature, three main methods have been used to identify synonyms:
WordNet Boteanu et al. (2019), Named Entity Recognition Bøhn and Nørvåg
(2010); Sotomayor and Veloz (2017), and word embedding Wang et al. (2015);
Leeuwenberg et al. (2016); Landthaler et al. (2017); Ali et al. (2019); Li et
al. (2019a). Handler et al. Handler (2014) highlighted that word embedding
generates a larger number of synonyms compared to other methods. Hazem et al.
Hazem and Daille (2018) used word embedding to identify synonyms for multi-
word terms and Ali et al. Ali et al. (2019) used word embeddings to
distinguish between synonyms and antonyms of the embeddings. Both papers used
Word2Vec as their word embedding technique.

In our paper, our interest is to associate hashtags that have similar
meanings. So we investigate various word embedding techniques to represent our
hashtags and to build our thesaurus.

###  2.3 Hashtag Representations

Hashtags have been used to promote TV-shows De Michele et al. (2019) and
utilised by health care professionals to share ideas and policies Ojo et al.
(2021). Being able to semantically compare hashtags help to quickly identify
social behaviours in the digital world, such as bullying Calvin et al. (2015)
and activism Byrne et al. (2021), and aggregate disaster responses Chowdhury
et al. (2020). Generally, machine learning tasks require the hashtags to be
represented as feature vectors (embeddings or representations). However,
learning the embeddings of hashtags can be very challenging. As hashtags are
user-generated, problems such as sparsity, polysemy and synonymy are common
Liu et al. (2018). Hashtags can be written as acronyms and can include
numbers, misspelled words, connected words, and connected words separated by
an underscore and/or full stops. Besides, a hashtag can have multiple
meanings, and multiple hashtags can share the same meaning.

Hashtags have been dealt with differently in the literature. Some researchers
assume that hashtags are independent components and should not be treated as
words Weston et al. (2014); Li et al. (2019b); Zhu et al. (2020). Thus, they
preserve hashtags as independent tokens without pre-processing them. Some
other researchers assume that the same peculiarities are shared between words
in tweets and hashtags. Accordingly, they treat words and hashtags similarly
after removing the punctuation, including the # sign. Hashtags are then
tokenised into words and sub-words using a tokenisation method.

Various methods have been proposed to learn hashtag representations, such as
Bag of Words methods (BOW) Tsur et al. (2012, 2013); Javed and Suk Lee (2018),
topic models Zhao et al. (2016), graph models Liu et al. (2018), and word
embedding techniques Dey et al. (2017). Tsur et al. Tsur et al. (2012, 2013)
and Javed et al. Javed and Suk Lee (2018) aggregated tweets containing the
same hashtag as a document. They then represented each hashtag using BOW of
the most frequent words in each document. Zhao et al. Zhao et al. (2016)
proposed Hashtag-LDA to generate joint representations of hashtags and words
in tweets to discover latent topics and global hashtags. Dey et al. Dey et al.
(2017) proposed EmTaggeR that treats hashtags as words in the tweets. They
pre-trained a skip-gram Word2Vec model over tweets to derive the embeddings of
hashtags. Liu et al. Liu et al. (2018) proposed Hashtag2Vec that creates a
heterogeneous hierarchical graph with hashtags, tweets, and words of tweets to
derive the embeddings of hashtags.

As hashtags are user generated and the free style of creating hashtags change
over time, applying the previous methods can be difficult. BOW methods focus
on the presence and absence of words in a set of tweets containing hashtags
and ignore the contextual meaning. LDA-based models adopt statistical methods
where latent parameters need to be recalculated with new data. Using the graph
models, the network graph structure can change significantly with the dynamic
change of topics over time. Word embedding techniques such as Word2Vec and
FastText are trained on the word-level, and pre-trained Word2Vec and FastText
models can perform poorly on Twitter data Stewart et al. (2019). Most
researchers re-trained their word embedding models, especially when working on
domain-specific Twitter data Alsini et al. (2019, 2020a).

###  2.4 Transformer-based Models

The context vector of a word can be encoded with respect to its surrounding
words using an attention mechanism such as the transformer Vaswani et al.
(2017). In 2018, Google’s Bidirectional Encoder Representations from
Transformers (BERT) Devlin et al. (2019) was a breakthrough in the NLP domain.
BERT is trained on extensive corpora using the masked words for predicting the
next sentence. BERT comes with two model sizes:

  * •

BERT-base, which contains 12 layers (transformer blocks), 12 attention heads,
and 110 million parameters;

  * •

BERT-Large, which includes 24 layers, 16 attention heads and 340 million
parameters.

Recently, various transformer-based models, such as RoBERTa Liu et al. (2019),
AlBERT Lan et al. (2020), and BERTweet Nguyen et al. (2020), have been
proposed. RoBERTa optimizes BERT by training it on larger corpora and more
training iterations with different masking patterns. AlBERT optimizes BERT by
using fewer parameters and thus shortens the training time. BERTweet, on the
other hand, uses a similar architecture as BERT-base but was trained using
RoBERTa’s pre-training procedure on Twitter data. BERTweet outperformed
RoBERTa and XLM-R on two classification datasets.

BERTweet-base was pre-trained on 850 million English tweets collected from
01/2012 to 08/2019. Pre-trained transformer-based models enable researchers to
achieve better results with minimal fine-tuning on various NLP tasks such as
classification Antoun et al. (2020), Named Entity Recognition Liang et al.
(2020), and Question-Answering Kayesh et al. (2020). All the previously
mentioned transformer-based models generate embeddings at the sentence level.
Unlike all the above transformer-based models which generate word embeddings
at the sentence level, our module, BERTag, encodes hashtags using a pre-
trained BERTweet model.

##  3 The #REval Framework

Given a dataset $\mathscr{D}\\!=\\!\\{(t_{1},H_{1}),(t_{2},H_{2}),\cdots\\}$
that contains a list of tweets and their corresponding hashtags where
$H_{i}\\!=\\!\\{h_{1},h_{2},\cdots\\}$, let
$\\{(\mathscr{R},\mathscr{G})\\}\\!=\\!\\{(R_{1},G_{1}),(R_{2},G_{2}),\cdots\\}$
be the collection of tuples representing the test set obtained from a hashtag
recommendation model. Here,
$R_{i}=\\{\hat{h}^{\prime}_{1},\hat{h}^{\prime}_{2},...\\}$ is the list of the
top-$r$ recommended hashtags (for a user-defined $r$ value) of the
$i^{\text{th}}$ test tweet and $G_{i}=\\{h^{\prime}_{1},h^{\prime}_{2},...\\}$
is the corresponding ground truth hashtags. Our goal is to measure the
performance of a given hashtag recommendation model by computing the average
#REval-hit-ratio (described later in Section 3.3.1), taking into account the
semantic correlation between hashtags. The notations used in the rest of the
paper are listed in Table 1.

Our #REval framework is designed to detect the semantic correspondences of the
lists of hashtags between $\mathscr{R}$ and $\mathscr{G}$. It comprises the
following modules: BERTag, synonym and thesaurus construction, and semantic
evaluation. These modules are detailed in the subsections below.

Table 1: Notations used in the paper.

$\mathscr{D}\\!=\\!\\{(t_{1},H_{1}),(t_{2},H_{2}),...\\}$ |  the processed dataset that contains a list of hashtagged tweets $t_{i}$, each of which has a corresponding list of hashtags $H_{i}$.
---|---
$T\\!=\\!\\{t_{1},t_{2},...\\}$ |  the set of all tweets.
$h$; $h_{i}$ |  an arbitrary hashtag; the $i^{\text{th}}$ hashtag in a set.
${(\eta_{t},h)}$ |  the index $\eta_{t}$ of a tweet $t$ for the hashtag $h$.
$m$ |  total number of clusters (i.e., total number of unique hashtags in the dataset).
${\bf v}_{t}$; ${\bf v}_{h}$ |  embedding for tweet $t$; embedding for hashtag $h$
$n_{h}$ |  total number of tweets having $h$ as a hashtag.
$R_{i}\\!=\\!\\{h_{1},h_{2},...,h_{r}\\}$ |  the list of top-$r$ (for some positive integer $r$) recommended hashtags for the $i^{\text{th}}$ test tweet.
$G_{i}\\!=\\!\\{h^{\prime}_{1},h^{\prime}_{2},...,h^{\prime}_{n_{i}}\\}$ |  the ground truth list of hashtags for the $i^{\text{th}}$ test tweet; the length $n_{i}$ of the list varies from tweet to tweet.
$\langle\mathscr{R},\mathscr{G}\rangle\\!=\\!\\{(R_{1},G_{1}),...,(R_{n},G_{n})\\}$ |  the test set for evaluation, where $R_{i}$ is to be verified against $G_{i}$.
$\text{Syn}_{k}(h)$ |  the set containing hashtag $h$ and the $k$ closest synonyms of $h$, i.e., the set has $k+1$ elements.
$\mathscr{E}\\!=\\!\\{(h,{\bf v}_{h})\,|\,\forall h\\}$ |  a dictionary of hashtags and their embeddings.
$\mathscr{T}\\!=\\!\\{(h,\text{Syn}(h))\,|\,\forall h\\}$ |  a thesaurus of hashtags and their synonyms.

###  3.1 BERTag

Let $T=\\{t_{i}\,|\,\forall i\\}$ be the set containing all the tweets in
$\mathscr{D}$. We consider the hashtags as labels for tweets, where every
tweet has a single hashtag label. For a tweet with multiple hashtags, we
duplicate the tweet a sufficient number of times to match the number of
hashtags. For example, if a tweet $t_{j}\in T$ contains two hashtags
$\\{h_{a},h_{b}\\}$, for some integers $a$ and $b$, we create the pairs
$(t_{j}^{(1)},h_{a})$ and $(t_{j}^{(2)},h_{b})$. The superscript of each
$t_{j}$ denotes the duplication of the tweet to match the number of hashtags.
The absolute ordering of tweets for a given hashtag is not important. The
subscripts under $h$ correspond to the unique ID of each distinct hashtag.
Retweets with the same label are removed. For each hashtag $h$, we also keep
the index $\eta_{t}$ of the corresponding tweet $t$, i.e., we store
$\\{(\eta_{t},h)\,|\,\forall t\\}$, for future retrieval of all the tweets for
any given hashtag $h$.

Tweets with their hashtags are the input to our module BERTag. The output of
BERTag is a dictionary $\mathscr{E}=\\{(h,{\bf v}_{h})\,|\,\forall h\\}$,
where each dictionary item is a hashtag $h$ and its semantic representation
${\bf v}_{h}$. To generate $\mathscr{E}$, three stages are involved in BERTag:
pre-processing, extracting tweet embeddings, and computing hashtag embeddings.

The architecture of BERTag comprising these three stages is shown in Fig. 1.

####  3.1.1 Pre-processing

The pre-processing stage of the BERTag module consists of two steps.

The first step is to perform data cleaning on the tweets. This step involves
removing stop words, mentions, URLs, and punctuation except for the # sign.
Tweets are transformed into lower case letters and non-English tweets are
removed.

The second step is to load the pre-trained BERTweet models from Hugging
Face111https://huggingface.co/vinai/bertweet-base. These pre-trained models
contains at least 16 billion word tokens generated via the TweetTokenizer from
NLTK222https://www.nltk.org. The tokenization process removes all the mentions
from tweets and truncates the length of repeated characters to 3 to reduce the
number of unique words (e.g., ‘Heeeeelllllllo’ becomes ‘Heeelllo’).
TweetTokenizer keeps hashtags as they are and considers them as independent
tokens. We use two versions of BERTweet in the evaluation of our framework:

  * •

BERTweet-base, and

  * •

BERTweet-covid19-base-uncased.

Both versions of BERTweet accept any tweet in the following form: The tweet
has been tokenized; the ‘[CLS]’ token has been appended to the start of the
tweet; the ‘[SEP]’ token has been appended at the end of the tweet; each token
for the tweet has been mapped to a unique token ID; the tweet has been
truncated to a pre-defined length; and attention masks for the ‘[PAD]’ tokens
have been added to differentiate between padding and non-padding.

Pre-processingPre-trained tokenizerTransformer block Transformer block
Transformer block Pre-trained BERTweet Semantic space (hashtag embedding
space) [CLS] reading nice books today [SEP] … $(t_{1}^{(1)},h_{a})$
$(t_{1}^{(2)},h_{b})$ $(t_{1}^{(3)},h_{c})$ $\vdots$ $(t_{i}^{(j)},h_{d})$
$\vdots$ $\\{(\eta_{t},h)\,|\,\forall h\\}$
$t_{1}$$t_{2}$$t_{3}$$t_{i}$$\vdots$$\vdots$${\bf v}_{t_{1}}$${\bf
v}_{t_{2}}$${\bf v}_{t_{3}}$${\bf
v}_{t_{i}}$$\vdots$$\vdots$$\mathscr{E}$$h_{1}$$h_{4}$$h_{3}$$h_{2}$$h_{m}$

Figure 1: The BERTag Architecture. The inputs are the tweets and their
hashtags; the output is the dictionary $\mathscr{E}=\\{(h,{\bf
v}_{h})\,|\,\forall h\\}$ containing all the hashtags and their embeddings
expressed in the same semantic space as the tweet embeddings.

####  3.1.2 Extracting tweet embeddings

At the end of the second pre-processing step above, each tweet is represented
by a BERTweet-base or a BERTweet-covid19-base-uncased embedding vector.
Following the dimensions specified in BERTweet, the tweet embedding vectors in
BERTag are also 768 dimensional. For any given tweet $t$, we use ${\bf v}_{t}$
to denote its tweet embedding vector.

Since the tweet embeddings in BERTag are learned from a large corpus of
tweets, tweets with similar textual contents are closer together in the
semantic space. The cosine distance function (i.e., $1-\cos(\theta)$, where
$\theta$ is the angle between the two embedding vectors being compared) can
therefore be used as a distance measure between tweets.

####  3.1.3 Computing hashtag embeddings

Since every tweet $t$ is now paired with a single hashtag label $h$, through
the index $\\{(\eta_{t},h)\,|\,\forall t\\}$, all tweets associated with the
hashtag $h$ can be retrieved. As described in the previous subsection, the way
that the BERTag tweet embeddings are learned ensures that tweets which share
the same hashtag are near each other and form a cluster, because these tweets
have similar textual contents. Thus, the number of clusters can be set to $m$,
the number of unique hashtags in the dataset.

Let $\\{{\bf v}_{t}\,|\,\forall t\\}$ be the set of embedding vectors of
tweets sharing $h$ as a common hashtag, the embedding ${\bf v}_{h}$ for
hashtag $h$ can be easily calculated by the arithmetic mean: ${\bf
v}_{h}=\frac{1}{s}\sum_{t}{\bf v}_{t}/n_{h}$, where $n_{h}$ is the total
number of tweets that use hashtag $h$ and $s$ is a normalisation term to
ensure that ${\bf v}_{h}$ is a unit vector. The centroid of each cluster thus
represents a hashtag embedding in the semantic space alongside its tweets.
Putting all the embedding vectors ${\bf v}_{h},\forall h$ together yields a
dictionary $\mathscr{E}$ of hashtags with their embeddings, i.e.,
$\mathscr{E}=\\{(h,{\bf v}_{h})\,|\,\forall h\\}$. From hereon, we refer to
the hashtag embeddings computed from BERTweet-based and BERTweet-covid-base-
uncased (see Section 3.1.1) as BERTag-base and BERTag-covid.

####  3.1.4 BERTag hashtag updates

Our BERTag module considers the dynamic nature of social media platforms where
new tweets and hashtags are continuously posted over time. So, it is designed
to avoid recalculating the hashtags’ embeddings from scratch when new tweets
are added. It is straightforward to incorporate a new tweet embedding and
calculate the cumulative average of all the tweet embeddings to get the
resultant hashtag embedding. Let $n_{h}$ be the number of existing tweets that
have been found to use hashtag $h$ and ${\bf v}_{h}$ be the corresponding
hashtag embedding. Let ${\bf v}_{t},$ be the embedding vector of the new tweet
$t$. Then the inclusion of this new tweet $t$ results in the following
updates:

$\displaystyle{\bf v}_{h}$
$\displaystyle\leftarrow\frac{1}{s}\left(\dfrac{n_{h}}{n_{h}+1}{\bf
v}_{h}+\dfrac{1}{n_{h}+1}{\bf v}_{t}\right),$ (1) $\displaystyle n_{h}$
$\displaystyle\leftarrow n_{h}+1,$ (2)

where $s$ is a scale factor to normalise ${\bf v}_{h}$ to a unit vector.

###  3.2 Synonym and Thesaurus Construction

The process of building a thesaurus starts with identifying the synonyms of
each existing hashtag. For every hashtag $h\in H$, the set of its $k$ closest
synonyms are identified from $\mathscr{E}=\\{(h,{\bf v}_{h})\\}$ using the
$k$-nearest neighbours (kNN) algorithm. Here, the value of $k$ is the number
of desired synonyms. As kNN includes the hashtag $h$ itself as one of its own
synonyms, to get, for instance, $n$ synonyms for $h$, we set the $k$ value in
kNN to $n+1$. We use the cosine distance as a measure of closeness of hashtags
in the hashtag embedding space. This process gives a list of synonyms
$\text{Syn}(h)$ for each hashtag $h$. These lists of synonyms together form
the thesaurus $\mathscr{T}$. That is,
$\mathscr{T}=\\{(h,\text{Syn}(h))\,|\,\forall h\\}$. Algorithm 1 shows the
pseudocode for constructing the list of $k$ synonyms for a given hashtag.

Algorithm 1 Construct_Synonyms

INPUT:

$\hat{h}$: a recommended hashtag.

$k$: an integer denoting the number of desired synonyms.

$\mathscr{E}=\\{(h,{\bf v}_{h})\\}$: the dictionary from BERTag training.

OUTPUT: $\hat{S}$: the set of synonyms for $\hat{h}$.

1: /* Get the embedding for $\hat{h}$ */

2: for each hashtag $h\in\mathscr{E}$ do

3: if $\hat{h}$ matches $h$ then

4: ${\bf v}_{\hat{h}}\leftarrow{\bf v}_{h}$

5: break

6: end if

7: end for

8: Construct $S_{v}$ using kNN and $\mathscr{E}$ to find $k+1$ hashtags having
shortest cosine distances from ${\bf v}_{\hat{h}}$

9: Construct $\hat{S}$ by searching $\mathscr{E}$ for all hashtags in $S_{v}$

10: return $\hat{S}$

Throughout the paper, we use $\text{Syn}(h)$ to denote the set of synonyms
with $h$ itself included. Where appropriate, we use a subscript to denote the
number of synonyms (other than the hashtag itself) in the set. For instance,

  1. $\bullet$

$\text{Syn}_{0}(h)=\\{h\\}$;

  2. $\bullet$

$\text{Syn}_{4}(h)$ is a list containing five hashtags, including $h$.

When the argument to $\text{Syn}()$ is a set of hashtags, e.g., if
$S=\\{h_{1},h_{2},\cdots\\}$, then $\text{Syn}(S)\triangleq\underset{h_{i}\in
S}{\cup}\text{Syn}(h_{i})$.

It should be noted that the synonym relationship between hashtags is not
symmetrical. In other words, suppose that a hashtag $h_{i}$ is in the synonym
set of another hashtag $h_{j}$, then it is not necessary that $h_{j}$ is in
the synonym set of $h_{i}$. For instance, using the example shown in Table 2,
#sport is one of the three closest synonyms of #swim; however, #swim is
outside the three closest synonyms of #sport.

###  3.3 Semantic Evaluation

The third and last module of the #REval framework is the #REval-hit-ratio
calculation.

####  3.3.1 #REval-hit-ratio

Let $R$ and $G$ be, respectively, the sets of recommended hashtags and ground
truth hashtags. In our previous work Alsini et al. (2020b), the hit ratio is
defined as:

$\text{hit-ratio}(R,G)=\frac{|R\cap G|}{\min(|R|,|G|)},$ (3)

where $|\\!\cdot\\!|$ denotes the cardinality of the set. In the #REval
framework, we modify this formula to incorporate the sets of synonyms
extracted from $R$ as follows:

$\displaystyle\text{\\#REval-hit-ratio}(R,G)=$
$\displaystyle\frac{1}{\min(|R|,|G|)}\begin{cases}\sum\limits_{\hat{h}\in
R}\rho\left(\text{Syn}(\hat{h})\cap G\neq\emptyset\right)&\text{if
}|R|\leq|G|\\\ \sum\limits_{h\in
G}\rho\bigg{(}h\in\text{Syn}(R)\bigg{)}&\text{otherwise},\end{cases}$ (4)

where

$\rho(s)=\begin{cases}1&\text{if $s$ = True}\\\
0&\text{otherwise}.\end{cases}$

We can see the original formula for hit ratio in Eq. (3) is just a special
case of the formula in Eq. (4) with the number of synonyms being zero.

Notice that synonyms are only constructed for the set of recommended hashtags
$R$, and not for the set of ground truth hashtags $G$. By doing so, we
consider each recommended hashtag $\hat{h}$ to be a hit only if a ground truth
hashtag $h$ in $G$ is synonymous to it, but not the other way round. Consider
the case where the recommended hashtag is a more specific term but the ground
truth hashtag is a more general term, such as $R=\\{\text{\\#hockey}\\}$ and
$G=\\{\text{\\#sport}\\}$. In this case, hockey is a kind of sport and it is
more likely that $\text{Syn}(\text{\\#hockey})\ni\text{\\#sport}$ and the
#REval-hit-ratio formula will correctly count the recommendation as a hit. On
the other hand, if $R=\\{\text{\\#sport}\\}$ and $G=\\{\text{\\#hockey}\\}$,
then it is unlikely that $\text{Syn}(\text{\\#sport})\ni\text{\\#hockey}$ as
sport is not necessarily about hockey. In this case, the formula will
correctly not count the recommendation as a hit. In contrary, if the synonym
lists were allowed to grow for both the recommendation set $R$ and ground
truth set $G$, then for extremely large $k$ values, the two synonym lists will
overlap even though the starting hashtags in the two sets are very different.
By restricting $G$ to remain as is, the above scenario will not occur. It
should be noted that $k$ is usually required to be a large number due to the
noise in hashtags; however, one should set the value of $k$ sensibly. If we
were to let $k\rightarrow\infty$, then every hashtag in the dataset would be
in the list of synonyms of any given hashtag, and it would not be meaningful
to use the measure for hashtag recommendation evaluation.

####  3.3.2 Synonym matching

The pseudo code for computing the number of matches between a set of
recommended hashtags and a set of ground truth hashtags is given in Algorithms
2. It is easy to verify that $\rho$, the number of matches between the two
sets $R$ and $G$ computed using Eq. (4), satisfies the condition:
$\rho\leq\min(|R|,|G|)$. The #REval-hit-ratio is then computed as
$\rho/\min(|R|,|G|)$. So, like the hit ratio defined in Eq. (3), #REval-hit-
ratio values are also bounded between 0 and 1, and the higher they are (closer
to 1), the better.

Algorithm 2 Match_Synonyms

INPUT:

$R$: a set containing the recommended hashtags,

$G$: a set containing the groundtruth hashtags,

$k$: the number of desired synonyms,

$\mathscr{T}=\\{(h,\text{Syn}(h))\,|\,\forall h\text{ in the test set}\\}$: a
thesaurus.

OUTPUT: ${\rho}$: the number of matches.

1: Initialize: $\rho\leftarrow 0$

2: if $|R|\leq|G|$ then

3: for $\hat{h}\in R$ do

4: $\text{Syn}(\hat{h})\leftarrow$ Look up $\hat{h}$ in $\mathscr{T}$

5: if $\text{Syn}(\hat{h})\cap G\neq\emptyset$ then

6: $\rho\leftarrow\rho+1$

7: end if

8: end for

9: else

10: /* construct the list of synonyms for $R$ */

11: $\text{Syn}(R)\leftarrow\emptyset$

12: for $\hat{h}\in R$ do

13: $\text{Syn}(\hat{h})\leftarrow$ Look up $\hat{h}$ in $\mathscr{T}$

14: $\text{Syn}(R)\leftarrow\text{Syn}(R)\cup\text{Syn}(\hat{h})$

15: end for

16: for $h\in G$ do

17: if $h\in\text{Syn}(R)$ then

18: $\rho\leftarrow\rho+1$

19: end if

20: end for

21: end if

22: return $\rho$

Four small examples showing how synonym matching works are given in Table 2,
for different cardinalities of $R$ and $G$. The last column of the table
contains the #REval-hit-ratio values, expressed as a fraction. Note that both
the recommended and ground truth hashtags can contain spelling mistakes, such
as #keeepfit shown in the examples. Except for the third example where
$R=\\{\text{\\#hockey}\\}$, all the other three examples show that the
recommended hashtags meaningfully match with the ground truth hashtags;
however, an exact matching of the hashtags in $R$ and $G$ will give a hit-
ratio of zero for all the examples.

Table 2: Four examples showing how $\rho$ and #REval-hit-ratio are computed
for different sizes of $R$ and $G$.

$R$ |  $G$ |  $\rho$ |  #REval-hit-ratio
---|---|---|---
$\\{\text{\\#hockey},\text{\\#championship}\\}$ |  $\\{\text{\\#football},\text{\\#sport}\\}$ |  1 |  1/2
$\\{\text{\\#football},\text{\\#sport}\\}$ |  $\\{\text{\\#hockey},\text{\\#sports}\\}$ |  1 |  1/2
$\\{\text{\\#hockey}\\}$ |  $\\{\text{\\#football},\text{\\#rugby}\\}$ |  0 |  0/1
$\\{\text{\\#swim},\text{\\#exercise}\\}$ |  $\\{\text{\\#sport}\\}$ |  1 |  1/1
$\text{Syn}(\text{\\#hockey})\\!=\\!\\{\text{\\#hockey},\text{\\#bowling},\text{\\#golf},\text{\\#sport}\\}$
$\text{Syn}(\text{\\#championship})\\!=\\!\\{\text{\\#championship},\text{\\#champion},\text{\\#winner},\text{\\#tournament}\\}$
$\text{Syn}(\text{\\#football})=\\{\text{\\#football},\text{\\#soccer},\text{\\#footy},\text{\\#rugby}\\}$
$\text{Syn}(\text{\\#sport})=\\{\text{\\#sport},\text{\\#sports},\text{\\#exercise},\text{\\#keeepfit}\\}$
$\text{Syn}(\text{\\#swim})=\\{\text{\\#swim},\text{\\#dive},\text{\\#paddle},\text{\\#sport}\\}$
$\text{Syn}(\text{\\#exercise})=\\{\text{\\#exercise},\text{\\#keeepfit},\text{\\#yoga},\text{\\#walking}\\}$

The #REval framework comprising all the modules explained above is shown in
Figure 2.

#REval Learning Hashtag Embeddings BERTag Synonym and Thesaurus Construction
Semantic Evaluation Synonym Matching and #REval-Hit-ratio Calculation Dataset
$\mathscr{D}=\\{(t_{i},H_{i})\\}_{i=1}^{n}$ A Hashtag Recommendation Model
$\mathscr{E}=\\{(h,\mathbf{v}_{h})\,|\,\forall
h\\}$$\mathscr{T}=\\{(\hat{h}^{\prime},\text{Syn}_{k}(\hat{h}^{\prime})\,|\,\forall\hat{h}^{\prime}\in\hat{H}^{\prime}\\}$Recommendation
$\mathscr{R}=\\{(t^{\prime},\hat{H}^{\prime})\,|\,\forall
t^{\prime},\hat{H}^{\prime}\\}$Average #REval-hit-ratio$k$Training set
$\\{(t,H)\,|\,\forall t,H\\}$; Test set $\\{t^{\prime}\,|\,\forall
t^{\prime}\\}$Ground truth for the test set:
$\mathscr{G}=\\{(t^{\prime},H^{\prime})\,|\,\forall t^{\prime},H^{\prime}\\}$

Figure 2: The #REval framework comprises three major modules: the BERTag
module, the Synonym and Thesaurus Construction module, and the Semantic
Evaluation module.

##  4 Experiments

We have evaluated the effectiveness of our #REval framework using three
datasets, with the last dataset being domain-specific (in health). We have
also tested the impact on the average #REval-hit-ratio when embeddings other
than BERTag-base and BERTag-covid are used in the first BERTag module of the
framework.

###  4.1 Datasets

####  4.1.1 The AU Trends dataset

We used 16 Australian trending hashtags (see Table 3) as seeds for crawling
for tweets posted on March $13^{\text{th}}$, 2020. As other hashtags were also
used in these tweets, the resultant number of unique hashtags is a lot larger
than the number of trending hashtags. After the data cleaning step, 272,686
tweets containing one or more hashtags remain. The number of unique hashtags
in these tweets is 15,375. As many hashtags were repetitively used multiple
times, the total number of hashtags (including the repetitions) is 547,227.
Table 4 summarises the statistics of this dataset and the remaining two
datasets described below.

Table 3: Trending hashtags in Australia and US on March $13^{\text{th}},2020$

AU Trends | US Trends
---|---
#AustralianGP #Covid_19 #HeartbreakWeather #yiayhi #Formula1 #SeizetheCSWmoment #australiangrandprix #WorldSleepDay #Ride2School #FlattenTheCurve #loona1stwin #SocialDistancing #MelbourneGP #ScottyFromMarketting #tomhanks #My750 |  #HeartbreakWeather #FridayThe13th #QuarantineAndChill #BernieIsRight #panicbuying #WeGotThisSeattle #EverythingsGonnaBeOkay #selfisolating #MyQuirkiestQuirkReasonsToLeaveTheToiletSeatUp #Restore4GinKashmir #peteonkimmel #LUVVsTheWorld2 #NiallCarpool #foreverfletcher

Table 4: Statistics summary of the four datasets used in our experiments.

|  AU Trends (Mar 2020) |  US Trends (Mar 2020) |  Health (2015)
---|---|---|---
No. of hashtagged tweets |  272,686 |  146,963 |  11,210
No. of unique hashtags |  15,375 |  5,608 |  2,710
No. of hashtags (including repetitions) |  547,227 |  206,663 |  15,548
Min. no. of hashtags per tweet |  1 |  1 |  1
Max. no. of hashtags per tweet |  14 |  13 |  8
Avg. no. of hashtags per tweet |  2.1 |  1.3 |  1.2
No. of training tweets (after pre-processing) |  242,032 |  125,781 |  9,900
No. of testing tweets (after pre-processing) |  26,893 |  13,976 |  1,101

####  4.1.2 The US Trends dataset

Following the same tweet crawling process above, we used 14 US trending
hashtags (see Table 3) to collect tweets, also on March $13^{\text{th}}$,
2020.

####  4.1.3 The Health dataset

This dataset Karami et al. (2018) contains health-related tweets collected in
2015 from 16 news agencies such as BBC, CBC, CNN, FoxNews, NBC.

###  4.2 Experimental setup

####  4.2.1 Variants of the #REval framework

As shown in Fig. 2, the BERTag module (see Section 3.1) outputs BERTag-base
embeddings or BERTag-covid embeddings for the dictionary $\mathscr{E}$. Apart
from using these embeddings, a different way to learn the hashtag embeddings
for building the dictionary $\mathscr{E}$ can be evaluated. In our
experiments, we tested the Word2Vec and FastText models. The training
procedure is summarised below.

We firstly trained Word2Vec (similarly for FastText) model using the gensim
library333https://radimrehurek.com/gensim/index.html, where every tweet is
considered as a document. Knowing that tweets are short texts, we set the
window_size hyperparameter to 2 to determine the dependency of a word to its
neighbouring words. We took into account every word in the tweets because some
hashtags occur only once. Thus, we set the min_count hyperparameter to 1. The
number of training epochs was set to 30. Finally, a dictionary
$\mathscr{E}=\\{(h,{\bf v}_{h})\\}$ containing all the hashtags and their
embeddings was generated. The hashtag embeddings were finally obtained by
taking the arithmetic means of the tweet embeddings obtained from the training
process.

So, in our experiments, we compared four different variants for the first
module of the #REval framework shown in Fig. 2. Each version uses:

  * •

hashtag embeddings defined in terms of Word2Vec;

  * •

hashtag embeddings defined in terms of FastText;

  * •

BERTag-base embeddings; or

  * •

BERTag-covid embeddings.

####  4.2.2 Generation of $\mathscr{R}$ for the test set

As shown in Fig. 2, #REval is agnostic to the hashtag recommendation model
that produces the recommendation set $\mathscr{R}$. This is evident from the
figure that the orange rectangular box is outside the light blue shaded region
for #REval. However, in order to test the Semantic Evaluation module of
#RERval, we need an example hashtag recommendation model. In this paper, we
chose the hashtag recommendation method based on tweet similarity proposed in
Alsini et al. (2020a) to generate $\mathscr{R}$. This method encoded the
tweets as Mean of Word Embedding (MOWE) vectors computed from words that were
trained using the Word2Vec model. For a given test tweet, the algorithm
computed its MOWE vector and compared it with the training tweets’ MOWE
vectors based on their cosine similarity measure. A threshold value of $0.5$
was used as the cut off for selecting similar tweets from the training set.
Hashtags that were used in these selected training tweets were ranked based on
their hashtag popularity, and the top-$r$ hashtags, for a pre-defined positive
integer $r$, were put forwarded as the recommended hashtags for the test
tweet. Because of the threshold value ($0.5$) used above, the number of
recommended hashtags for any test tweet may be fewer than $r$.

####  4.2.3 Top-$r$ recommendations and number of synonyms

For each dataset mentioned above, only those tweets having at least one
hashtag are retained. After the pre-processing stage (see Section 3.1.1), the
tweets are split into 90/10% to form the training set and the test set. The
training set is used to train the hashtag recommendation method Alsini et al.
(2020a) described above to generate $\mathscr{R}$ for the test set. For each
tweet in the testing set, the top-$r$ recommended hashtags are extracted for
the following $r$ values: $1$, $5$, and $10$. This gives three separate
$\mathscr{R}$ for the three top-$r$ recommendations for comparison with the
ground truth test set $\mathscr{G}$ (see Fig. 2).

For each recommended hashtag, we set the number of synonyms (the value of $k$
in Algorithms 1 and 2) to a range of values: $\\{0,5,10,20,\cdots,70\\}$. The
reason for setting $k$ to a large value such as $70$ is because hashtags are
noisy text strings and can contain spelling mistakes, numbers, etc. For
instance, the hashtags #covid, #covid0, #covid19, #covid2019, #coronavirus,
and #coronav are all synonymous words and most of them are not in the standard
English dictionary. So, when considering a specific target hashtag, we need to
enlarge the number of synonyms in the hashtag embedding space. It should be
noted that, for the case where $k=0$, the #REval-hit-ratio becomes the hit-
ratio (Eq. (3)) and it can be considered to be the output from the baseline
exact-matching method of hashtags.

###  4.3 Results and Discussions

For each test tweet $t^{\prime}$ in the test set, we computed the #REval-hit-
ratio between the recommended hashtag set $R$ and the ground truth hashtag set
$G$ for $t^{\prime}$. The average #REval-hit-ratio was finally computed for
the entire recommendation set $\mathscr{R}$. Figure 3 shows the average
#REval-hit-ratios for the four variants of embeddings computed using different
numbers of synonyms (the value of $k$ on the horizontal axis) for the three
datasets. The top, middle, and bottom rows of the figure correspond to the
top-1, top-5, and top-10 recommendations. The #REval-hit-ratio at $k=0$ in
each subplot denotes the case where no synonyms were used, i.e.,
$\text{Syn}_{0}(h)=\\{h\\}$ for each hashtag $h$, and effectively the hit-
ratio (Eq. (3)) for the baseline exact-matching of hashtags was computed
instead.

(a)  (b)  (c)

Figure 3: A comparison of the average #REval-hit-ratios for top-$1$, top-$5$,
and top-$10$ recommendations from the hashtag recommendation model of Alsini
et al. (2020a) on the three datasets, with the hashtag embeddings represented
as Word2Vec, FastText, BERTag-base, and BERTag-covid. The number of synonyms
($k$) varies from $0,5,10,20,\cdots,70$. (figure best viewed in colour)

Comparing the results shown in Figure 3, we summarize the following four
observations.

(i) The exact-matching method underrates the performance of the hashtag
recommendation model. As expected, the baseline exact-matching method (where
$k=0$) has the lowest average #REval-hit-ratio in each subplot, as no synonyms
were included in the calculation. For the AU Trends and Health datasets (Fig.
3, columns 1 and 3) we can see that, by increasing $k$ slightly to 5 or 10,
the average #REval-hit-ratio values are almost double when the BERTag-base
(blue curves) or BERTag-covid (magenta curves) embeddings were used to
represent the hashtags. This improvement is consistent for all the top-1,
top-5, and top-10 recommendations. For the US Trends dataset, we also observe
increases in the average #REval-hit-ratio values when $k$ grows, although the
increases are smaller when $k$ is under 10. When Word2Vec or FastText was used
as hashtag embeddings, the increase in average #REval-hit-ratio with respect
to $k$ is small.

Comparing across the three datasets, the hashtag recommendation model Alsini
et al. (2017) performed the best on the US Trends dataset. Even when $k=0$,
its average #REval-hit-ratio on the US Trends dataset is much higher than
those for the other two datasets. It should be noted that hashtag
recommendation is not an easy task because of noise in the data and large
number of hashtags and variations of tweet contents. With synonyms included,
its average #REval-hit-ratio is just under 0.8 for the top-10 recommendations.

(ii) BERTag-base and BERTag-covid outperform Word2Vec and FastText. Since the
same recommendation test set $\mathscr{R}$ was used for all the four variants,
the only difference is what embeddings were used to represent the hashtags
and, consequently, the quality of the generated synonyms for the #REval-hit-
ratio calculation. We manually inspected many hashtags and their synonyms
extracted from the embedding spaces of these four variants and we summarise
six hashtags (two from each dataset) and their five closest synonyms in Table
5. We can see from the table that the synonyms from BERTag-base and BERTag-
covid are very similar in most examples. Also, compared to those from Word2Vec
and FastText, these synonyms are more related to the original hashtags in the
first column. For the hashtag #stockmarket, both BERTag-base and BERTag-covid
even picked up the names of two specific shares, #sensex and #nifty, as
synonyms. Looking at the Word2Vec column in the table, we see some strange
words, such as #lasagna and #grilling as the two closest synonyms of #aging;
and #ask as the first synonym of #quinoa. FastText, on the other hand, seemed
to group words that have similar substrings together in the hashtag space,
e.g., the substring “market” appears in all the five synonyms of #stockmarket;
the postfix “ing” appears in four of the five synonyms of #aging. While these
synonymous hashtags from FastText are sometimes quite relevant, there are many
cases where FastText did not produce good synonyms, e.g., the five synonyms
from FastText for #quinoa are quite poor.

Table 5: The five closest synonyms of a few example hashtags from the three
datasets.

Hashtag | Synonyms from thesaurus using
---|---
Word2Vec | FastText | BERTag-base | BERTag-covid
#stockmarket (AU Trends) |  #fearnoevil, #mkonoh, #awareness, #sooryavanshion24thmarch, #faculty |  #stockmarkets, #stockmarketcrash, #stockmarket2020, #trumpstockmarket, #stockmarketcrash2020 |  #stockmarketcrash, #stockmarketcrash2020, #sensex, #nifty, #stocks |  #stockmarketcrash2020, #stockmarketcrash, #sensex, #nifty, #commodities
#selfisolation (AU Trends) |  #onthepulse, #coronahitsnoida, #koreandrama, #julialang, #mohfw |  #selfisolationhelp, #selfisolating, #selfisolate, #italyselfisolation, #selfisolationlevel100 |  #socialdistancing, #selfquarantine, #selfisolate, #quarantine, #isolation |  #socialdistancing, #selfquarantine, #selfisolate, #quarantine, #isolation
#panicbuying (US Trends) |  #aiea, #petebuttigieg, #powernap, #florida, #ebooks |  #panicbuyinguk, #panicbuy, #stoppanicbuying, #panickbuying, #costcopanicbuying |  #cronavirus, #selfisolating, #blessed, #superstore, #foodbank |  #hoarding, #cronavirus, #panickbuying, #superstore, #asda
#selfisolation (US Trends) |  #lka, #stop, #cinderella, #gaslightingofamerica, #melekoout |  #selfisolationhelp, #selfisolating, #selfisolate, #selfisolationhelpdrogheda, #isolation |  #selfquarantine, #selfisolationhelp, #socialdistancing, #lockdown, #quarantine |  #socialdistancing, #selfquarantine, #selfisolationhelp, #selfisolating, #quarantined
#aging (Health) |  #lasagna, #grilling, #healthforall, #healthyliving, #pet |  #antiaging, #healthyaging, #agility, #jogging, #breaking |  #memory, #dementia, #stroke, #autism, #health |  #memory, #stroke, #autism, #dementia, #health
#quinoa (Health) |  #ask, #polio, #hulahooping, #brunch, #cancerfighting |  #quit, #quiz, #quesadillas, #mojo, #smokers |  #healthy, #recipes, #health, #diet, #food |  #healthy, #recipes, #health, #diet, #food

(iii) Both BERTag-base and BERTag-covid have similar performance and are
better embeddings for hashtags. Comparing the #REval-hit-ratios when these
embeddings are used in the BERTag module (Fig. 2), we see that BERTag-base
(blue curve) outperforms BERTag-covid (magenta curve) slightly for the US
Trends dataset, but the results are the other way round for the AU Trends
dataset. For the Health dataset, the two curves almost completely overlap,
except when $k=20$, the magenta curve is a little bit higher.

As discussed above, compared to Word2Vec and FastText, when BERTag-base or
BERTag-covid embeddings are used to represent hashtags, the average #REval-
hit-ratios are consistently higher for all the datasets. To illustrate how the
clusters of tweets sharing the same hashtags look like, we pick the ten most
popular hashtags in each dataset (see Table 6) and project all the tweets for
these hashtags to a 2D space using the t-SNE algorithm. Due to space
limitation, we only show the t-SNE projection for the BERTag-covid embeddings.
Some details about Fig. 4 are listed below.

Table 6: The ten most popular hashtags and their frequencies Dataset | Popular hashtags and their frequencies
---|---
AU Trends |  #covid19 (81900), #coronavirusupdate (59514), #coronaviruspandemic (52163), #coronavirus (46290), #heabreakweather (27885), #covid2019 (20797), #coronaoutbreak (17403), #corona (9801), #coronav (9150), #flattenthecurve (6595)
US Trends |  #heabreakweather (66262), #niallcarpool (16328), #coronavirusinkenya (10203), #1 (10089), #fridaythe13th (8442), #2 (5849), #traderjoes (4671), #bernieisright (4578), #latelateniall (4352), #covid19 (4345)
Health |  #healthtalk (873), #nhs (768), #ebola (392), #getfit (261), #latfit (241), #obamacare (237), #weightloss (235), #health (228), #pharma (209), #fitness (209)

t-SNE projection of 85646 labelled tweets

t-SNE projection of 52929 labelled tweets

t-SNE projection of 3510 labelled tweets

(a) AU Trends dataset

(b) US Trends dataset

(c) Health dataset

Figure 4: 2-D Visualization of the tweets for the ten most popular hashtags
for the three datasets. The tweets are represented by the 768-dimensional
BERTag-covid embeddings. Note that the hashtags in the legend of each plot are
sorted in alphabetical order rather than decreasing order of frequency as in
Table 6. (figure best viewed in colour)

  * •

For the AU Trends dataset, as the tweets were crawled in the period when
Covid-19 was widely discussed on Twitter, nine out of these ten hashtags are
related to Covid-19, except for #heabreakweather, which is related to the
singer Niall Horan’s studio album Heartbreak Weather released on the same
date. It is clear in Fig. 4(a) that the tweets for this hashtag form a cluster
on their own (the deep red dots) while the tweets for the other nine clusters
all lump together as they are related to the same topic.

  * •

For the US Trends dataset (Fig. 4(b)), the largest cluster is the tweets for
the first hashtag #heabreakweather. There is a great overlap between the two
clusters for #1 and #2, which are common hashtags that Twitter users like to
use when they want to claim they were the first person achieving certain
endeavours. As expected, the clusters for the two Covid-related hashtags,
#coronavirusinkenya and #covid19, overlap significantly, while the tweets for
#bernieisright and #fridaythe13th form two separate clusters. Many tweets for
the hashtag #traderjoes (about Trader Joe’s grocery store) had cross-topic
discussions and they form scattered points, overlapping with other clusters.

  * •

Different from the previous two datasets, the Health Dataset is a domain-
specific dataset crawled in 2015. For this dataset (Fig. 4(c)), the tweet
clusters for most hashtags are well separated, e.g., #ebola, #obamacare, #nhs,
and #pharma. However, due to the similarities of the topics discussed in the
tweets, the #health cluster has some overlap with #healthtalk and #fitness;
the tweet cluster for #weightloss overlaps with those for #latfit and
#fitness, and it is also close to the cluster for #getfit.

We can see from the t-SNE projection that the hashtags represented using
BERTag-covid (similarly for BERTag-base) are more closely related to the
topics discussed in the tweets. Consequently, these embeddings give more
relevant synonyms for hashtag recommendation evaluation, as confirmed also
from the average #REval-hit-ratios shown in Fig. 3. Our #REval framework
therefore uses either BERTag-base or BERTag-covid as the default embeddings
for the BERTag module.

(iv) Effectiveness and correctness of the #REval-hit-ratio formula. We
discussed in Section 3.3 why the list of synonyms should be constructed for
the hashtags in the recommendation set $R$ only, as formulated in Eq. (4). It
is clear that all the curves in Fig. 3 have a steeper increase of the average
#REval-hit-ratio when the number of synonyms is small and only a gentle
increase when more synonyms are included in the #REval-hit-ratio calculation.
This is what we expect to see in a good hashtag recommendation model. Of
course, if we keep growing $k$ indefinitely, we will continue to see a gradual
increase of #REval-hit-ratio. So, when comparing the performance of two
hashtag recommendation models, one should consider the model that achieves the
same #REval-hit-ratio but with a smaller $k$ value as a better model. Our
current framework allows the $k$ value to be fed to the Synonym and Thesaurus
Construction module (Fig. 2). To impose the criterion that the synonyms must
meet a certain level of quality, a threshold value on the cosine distance used
in kNN (Section 3.2) can be easily added to the module.

##  5 Conclusion and Future Work

We have presented #REval, a novel evaluation framework that measures the
performance of hashtag recommendation models by incorporating hashtag
synonyms. The framework automatically constructs thesauri of hashtags with
their synonyms based on their similarity in the hashtag (and tweet) embedding
space. In the paper, we have introduced BERTag, an internal module of #REval
that learns hashtag representations from clusters of tweets encoded using a
pre-trained transformer-based model. Our experiments show that #REval with
BERTag-base and BERTag-covid hashtag embeddings give better and more relevant
synonyms than the Word2Vec and FastText embeddings do. We have shown that the
exact-matching method is inadequate for matching hashtags and therefore not
suitable for evaluating hashtag recommendation models. Our proposed #REval-
hit-ratio formula helps to overcome this short-coming.

While we use BERTag embeddings in our #REval framework because of the superior
performance of the parent BERTweet embeddings, other improved hashtag
embeddings in the future can be used in the BERTag module. Our #REval
framework can be trained for different datasets to generate different
dictionaries. Once the training is done, the framework can be put in action
for evaluation, i.e., the second and third modules do not need any fine-
tuning. We think that, with a large number of social media users today, having
an evaluation framework such as #REval is important. Our framework can also be
adopted for other applications such as product recommendation extracted from
messages posted by users. Besides, the resulting hashtags representations
encoded using BERTag can be used in many applications such as topic
identification, event detection, and information diffusion.

Currently, the #REval-hit-ratio formula considers the synonym list for any
given hashtag as a set. Our future work is to extend the formula by
considering it as an ordered list. By doing so, higher weights can be given to
those synonyms appearing near the beginning of list, as they are closer to the
hashtag. Different ways of ranking hashtags in the synonym list can also be
investigated and compared.

## References

  * Alsini et al. [2021] Areej Alsini, Du Q. Huynh, and Amitava Datta.  Hashtag recommendation methods for twitter and sina weibo: A review.  _Future Internet_ , 13(5), 2021.  ISSN 1999-5903.  doi:10.3390/fi13050129.  URL https://www.mdpi.com/1999-5903/13/5/129.
  * Kywe et al. [2012] Su Mon Kywe, Tuan-Anh Hoang, Ee-Peng Lim, and Feida Zhu.  _On Recommending Hashtags in Twitter Networks_ , pages 337–350.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2012.  ISBN 978-3-642-35386-4.  doi:10.1007/978-3-642-35386-4_25.
  * Zhao et al. [2016] Feng Zhao, Yajun Zhu, Hai Jin, and Laurence T. Yang.  A personalized hashtag recommendation approach using lda-based topic model in microblog environment.  _Future Gener. Comput. Syst._ , 65(C):196–206, December 2016.  ISSN 0167-739X.  doi:10.1016/j.future.2015.10.012.  URL http://dx.doi.org/10.1016/j.future.2015.10.012.
  * Li et al. [2016] Jia Li, Hua Xu, Xingwei He, Junhui Deng, and Xiaomin Sun.  Tweet modeling with lstm recurrent neural networks for hashtag recommendation.  In _IJCNN_ , pages 1570–1577. IEEE, 2016.  ISBN 978-1-5090-0620-5.  URL http://dblp.uni-trier.de/db/conf/ijcnn/ijcnn2016.html.
  * Alsini et al. [2017] Areej Alsini, Amitava Datta, Jianxin Li, and Du Huynh.  Empirical analysis of factors influencing twitter hashtag recommendation on detected communities.  In _ADMA - 13th International Conference, ADMA, Singapore_ , pages 119–131, 2017.  doi:10.1007/978-3-319-69179-4_9.  URL https://doi.org/10.1007/978-3-319-69179-4_9.
  * Kowald et al. [2017] Dominik Kowald, Subhash Chandra Pujari, and Elisabeth Lex.  Temporal Effects on Hashtag Reuse in Twitter: A Cognitive-Inspired Hashtag Recommendation Approach.  In _Proceedings of the 26th International Conference on WWW_ , pages 1401–1410, Republic and Canton of Geneva, Switzerland, 2017. International World Wide Web Conferences Steering Committee.  ISBN 978-1-4503-4913-0.  doi:10.1145/3038912.3052605.
  * Zhang et al. [2017] Qi Zhang, Jiawen Wang, Haoran Huang, Xuanjing Huang, and Yeyun Gong.  Hashtag recommendation for multimodal microblog using co-attention network.  In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17_ , pages 3420–3426, 2017.  doi:10.24963/ijcai.2017/478.  URL https://doi.org/10.24963/ijcai.2017/478.
  * Shi et al. [2018] Bichen Shi, Gevorg Poghosyan, Georgiana Ifrim, and Neil Hurley.  Hashtagger+: Efficient high-coverage social tagging of streaming news.  _IEEE Transactions on Knowledge and Data Engineering_ , 30(1):43–58, 2018.  doi:10.1109/TKDE.2017.2754253.
  * Alsini et al. [2020a] Areej Alsini, Amitava Datta, and Du Q. Huynh.  On utilizing communities detected from social networks in hashtag recommendation.  _IEEE Transactions on Computational Social Systems_ , pages 1–12, 2020a.
  * Alsini et al. [2020b] Areej Alsini, Du Q. Huynh, and Amitava Datta.  Hit ratio: An evaluation metric for hashtag recommendation.  _CoRR_ , abs/2010.01258, 2020b.  URL https://arxiv.org/abs/2010.01258.
  * Mazzia and Juett [2011] Allie Mazzia and James Juett.  Suggesting hashtags on twitter.  In _EECS 545 Project, Winter Term, 2011. URL http://www-personal.umich.edu/ amazzia/pubs/545-final.pdf_ , 2011.
  * Nguyen et al. [2020] Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen.  BERTweet: A pre-trained language model for English tweets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 9–14, Online, October 2020. Association for Computational Linguistics.  doi:10.18653/v1/2020.emnlp-demos.2.  URL https://www.aclweb.org/anthology/2020.emnlp-demos.2.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.  doi:10.3115/1073083.1073135.  URL https://www.aclweb.org/anthology/P02-1040.
  * Banerjee and Lavie [2005] Satanjeev Banerjee and Alon Lavie.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/W05-0909.
  * Lin [2004] Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/W04-1013.
  * Vedantam et al. [2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In _CVPR_ , pages 4566–4575. IEEE Computer Society, 2015.  ISBN 978-1-4673-6964-0.  URL http://dblp.uni-trier.de/db/conf/cvpr/cvpr2015.html#VedantamZP15.
  * Anderson et al. [2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  Spice: Semantic propositional image caption evaluation.  _CoRR_ , abs/1607.08822, 2016.  URL http://dblp.uni-trier.de/db/journals/corr/corr1607.html#AndersonFJG16.
  * Gilchrist [2003] Alan Gilchrist.  Thesauri, taxonomies and ontologies - an etymological note.  _Journal of Documentation - J DOC_ , 59:7–18, 02 2003.  doi:10.1108/00220410310457984.
  * Schwarz [2005] Katharina Schwarz.  _Domain model enhanced search- A comparison of taxonomy, thesaurus and ontology_.  PhD thesis, University of Utrecht, 2005.
  * Al-Khateeb et al. [2017] B. Al-Khateeb, A. J. Al-Kubaisi, and S. T. Al-Janabi.  Query reformulation using wordnet and genetic algorithm.  In _2017 Annual Conference on New Trends in Information Communications Technology Applications (NTICT)_ , pages 91–96, 2017.
  * Boteanu et al. [2019] Adrian Boteanu, Adam Kiezun, and Shay Artzi.  Synonym expansion for large shopping taxonomies.  In _Automated Knowledge Base Construction (AKBC)_ , 2019.  URL https://openreview.net/forum?id=rJx2g-qaTm.
  * Bøhn and Nørvåg [2010] C. Bøhn and K. Nørvåg.  Extracting named entities and synonyms from wikipedia.  In _2010 24th IEEE International Conference on Advanced Information Networking and Applications_ , pages 1300–1307, 2010.
  * Sotomayor and Veloz [2017] M. Sotomayor and F. Veloz.  Thesaurus-based named entity recognition system for detecting spatio-temporal crime events in spanish language from twitter.  In _2017 IEEE Second Ecuador Technical Chapters Meeting (ETCM)_ , pages 1–5, 2017.
  * Wang et al. [2015] Chang Wang, Liangliang Cao, and Bowen Zhou.  Medical synonym extraction with concept space models.  In _Proceedings of the 24th International Conference on Artificial Intelligence_ , IJCAI’15, page 989–995. AAAI Press, 2015.  ISBN 9781577357384.
  * Leeuwenberg et al. [2016] Artuur Leeuwenberg, Mihaela Vela, Jon Dehdari, and Josef Genabith.  A minimally supervised approach for synonym extraction with word embeddings.  _The Prague Bulletin of Mathematical Linguistics_ , 105:111–142, 04 2016.  doi:10.1515/pralin-2016-0006.
  * Landthaler et al. [2017] Jörg Landthaler, Bernhard Waltl, Dominik Huth, Daniel Braun, and Florian Matthes.  Extending thesauri using word embeddings and the intersection method.  _Proceedings of ASAIL 2017_ , 2143(6), 2017.
  * Ali et al. [2019] Muhammad Asif Ali, Yifang Sun, Xiaoling Zhou, Wei Wang, and Xiang Zhao.  Antonym-synonym classification based on new sub-space embeddings.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 6204–6211. AAAI Press, 2019\.  doi:10.1609/aaai.v33i01.33016204.  URL https://doi.org/10.1609/aaai.v33i01.33016204.
  * Li et al. [2019a] Cheng Li, Mingyang Zhang, Michael Bendersky, Hongbo Deng, Donald Metzler, and Marc Najork.  Multi-view embedding-based synonyms for email search.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR’19, page 575–584, New York, NY, USA, 2019a. Association for Computing Machinery.  ISBN 9781450361729.  doi:10.1145/3331184.3331250.  URL https://doi.org/10.1145/3331184.3331250.
  * Handler [2014] Abram Handler.  An empirical study of semantic similarity in wordnet and word2vec.  Master’s thesis, University of New Orleans, 2014.
  * Hazem and Daille [2018] Amir Hazem and Béatrice Daille.  Word embedding approach for synonym extraction of multi-word terms.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan, May 2018. European Language Resources Association (ELRA).  URL https://www.aclweb.org/anthology/L18-1045.
  * De Michele et al. [2019] Roberta De Michele, Stefano Ferretti, and Marco Furini.  On helping broadcasters to promote tv-shows through hashtags.  _Multimedia Tools and Applications_ , 78, 02 2019.  doi:10.1007/s11042-018-6510-7.
  * Ojo et al. [2021] Ayotomiwa Ojo, Sharath Chandra Guntuku, Margaret Zheng, Rinad S Beidas, and Megan L Ranney.  How health care workers wield influence through twitter hashtags: Retrospective cross-sectional study of the gun violence and covid-19 public health crises.  _JMIR Public Health Surveill_ , 7(1):e24562, Jan 2021.  ISSN 2369-2960.  doi:10.2196/24562.  URL https://publichealth.jmir.org/2021/1/e24562.
  * Calvin et al. [2015] Angela Calvin, Amy Bellmore, Jun-Ming Xu, and Xiaojin Zhu.  #bully: Uses of hashtags in posts about bullying on twitter.  _Journal of School Violence_ , 14:133–153, 01 2015.  doi:10.1080/15388220.2014.966828.
  * Byrne et al. [2021] Virginia L. Byrne, Bridget L. Higginbotham, Alice E. Donlan, and Terah J. Stewart.  An online occupation of the university hashtag: Exploring how student activists use social media to engage in protest.  _Journal of College and Character_ , 22(1):13–30, 2021.  doi:10.1080/2194587X.2020.1860775.  URL https://doi.org/10.1080/2194587X.2020.1860775.
  * Chowdhury et al. [2020] Jishnu Ray Chowdhury, Cornelia Caragea, and Doina Caragea.  On identifying hashtags in disaster twitter data.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 498–506. AAAI Press, 2020.  URL https://aaai.org/ojs/index.php/AAAI/article/view/5387.
  * Liu et al. [2018] Jie Liu, Zhicheng He, and Yalou Huang.  Hashtag2vec: Learning hashtag representation with relational hierarchical embedding model.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , pages 3456–3462. International Joint Conferences on Artificial Intelligence Organization, 7 2018\.  doi:10.24963/ijcai.2018/480.  URL https://doi.org/10.24963/ijcai.2018/480.
  * Weston et al. [2014] Jason Weston, Sumit Chopra, and Keith Adams.  #TagSpace: semantic embeddings from hashtags.  In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, _EMNLP_ , pages 1822–1827. ACL, 2014.  ISBN 978-1-937284-96-1.
  * Li et al. [2019b] Yang Li, Ting Liu, Jingwen Hu, and Jing Jiang.  Topical co-attention networks for hashtag recommendation on microblogs.  _Neurocomputing_ , pages 356–365, 2019b.
  * Zhu et al. [2020] Rui Zhu, Delu Yang, and Yang Li.  Learning improved semantic representations with tree-structured lstm for hashtag recommendation: An experimental study.  _Information_ , 10 2020.  doi:https://doi.org/10.3390/info10040127.
  * Tsur et al. [2012] Oren Tsur, Adi Littman, and Ari Rappoport.  Scalable multi stage clustering of tagged micro-messages.  In _Proceedings of the 21st International Conference on World Wide Web_ , WWW ’12 Companion, page 621–622, New York, NY, USA, 2012. Association for Computing Machinery.  ISBN 9781450312301.  doi:10.1145/2187980.2188157.  URL https://doi.org/10.1145/2187980.2188157.
  * Tsur et al. [2013] O. Tsur, A. Littman, and A. Rappoport.  Efficient clustering of short messages into general domains.  _Proceedings of the 7th International Conference on Weblogs and Social Media, ICWSM 2013_ , pages 621–630, 01 2013.
  * Javed and Suk Lee [2018] Ali Javed and Byung Suk Lee.  Hybrid semantic clustering of hashtags.  _Online Social Networks and Media_ , 5:23–36, 03 2018.  doi:10.1016/j.osnem.2017.10.004.
  * Dey et al. [2017] Kuntal Dey, Ritvik Shrivastava, Saroj Kaushik, and L. Venkata Subramaniam.  Emtagger: A word embedding based novel method for hashtag recommendation on twitter.  In _2017 IEEE International Conference on Data Mining Workshops, ICDM Workshops 2017, New Orleans, LA, USA, November 18-21, 2017_ , pages 1025–1032, 2017.  doi:10.1109/ICDMW.2017.145.  URL https://doi.org/10.1109/ICDMW.2017.145.
  * Stewart et al. [2019] Michael Stewart, Wei Liu, and Rachel Cardell-Oliver.  Word-level Lexical Normalisation using Context-Dependent Embeddings.  _arXiv e-prints_ , art. arXiv:1911.06172, November 2019.
  * Alsini et al. [2019] Areej Alsini, Amitava Datta, Du Q. Huynh, and Jianxin Li.  Community aware personalized hashtag recommendation in social networks.  In Rafiqul Islam, Yun Sing Koh, Yanchang Zhao, Graco Warwick, David Stirling, Chang-Tsun Li, and Zahidul Islam, editors, _Data Mining_ , pages 216–227, Singapore, 2019. Springer Singapore.  ISBN 978-981-13-6661-1.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.  ISBN 9781510860964.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi:10.18653/v1/N19-1423.  URL https://www.aclweb.org/anthology/N19-1423.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.  URL http://arxiv.org/abs/1907.11692.  cite arxiv:1907.11692.
  * Lan et al. [2020] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  _ICLR_ , 2020.
  * Antoun et al. [2020] Wissam Antoun, Fady Baly, and Hazem Hajj.  AraBERT: Transformer-based model for Arabic language understanding.  In _Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection_ , pages 9–15, Marseille, France, May 2020. European Language Resource Association.  ISBN 979-10-95546-51-1.  URL https://www.aclweb.org/anthology/2020.osact-1.2.
  * Liang et al. [2020] Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang.  Bond: Bert-assisted open-domain named entity recognition with distant supervision.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’20, page 1054–1064, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450379984.  doi:10.1145/3394486.3403149.  URL https://doi.org/10.1145/3394486.3403149.
  * Kayesh et al. [2020] Humayun Kayesh, Md. Saiful Islam, Junhu Wang, Shikha Anirban, A.S.M. Kayes, and Paul Watters.  Answering binary causal questions: A transfer learning based approach.  In _2020 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–9, 2020.  doi:10.1109/IJCNN48605.2020.9207662.
  * Karami et al. [2018] Amir Karami, Aryya Gangopadhyay, Bin Zhou, and Hadi Kharrazi.  Fuzzy approach topic discovery in health and medical corpora.  _International Journal of Fuzzy Systems_ , 20:1334–1345, 2018.