# UnCommonSense:
Informative Negative Knowledge about Everyday Concepts

Hiba Arnaout<EMAIL_ADDRESS>Max Planck Institute for
InformaticsSaarbrückenGermany ,  Simon Razniewski<EMAIL_ADDRESS>Max
Planck Institute for InformaticsSaarbrückenGermany ,  Gerhard Weikum
<EMAIL_ADDRESS>Max Planck Institute for InformaticsSaarbrückenGermany
and  Jeff Z. Pan<EMAIL_ADDRESS>The University of EdinburghEdinburghUnited
Kingdom

(2022)

###### Abstract.

Commonsense knowledge about everyday concepts is an important asset for AI
applications, such as question answering and chatbots. Recently, we have seen
an increasing interest in the construction of structured commonsense knowledge
bases (CSKBs). An important part of human commonsense is about properties that
do not apply to concepts, yet existing CSKBs only store positive statements.
Moreover, since CSKBs operate under the open-world assumption, absent
statements are considered to have unknown truth rather than being invalid.
This paper presents the UnCommonSense framework for materializing informative
negative commonsense statements. Given a target concept, comparable concepts
are identified in the CSKB, for which a local closed-world assumption is
postulated. This way, positive statements about comparable concepts that are
absent for the target concept become seeds for negative statement candidates.
The large set of candidates is then scrutinized, pruned and ranked by
informativeness. Intrinsic and extrinsic evaluations show that our method
significantly outperforms the state-of-the-art. A large dataset of informative
negations is released as a resource for future research.

Knowledge Bases, Negation, Commonsense

††journalyear: 2022††copyright: rightsretained††conference: Proceedings of the
31st ACM International Conference on Information and Knowledge Management;
October 17–21, 2022; Atlanta, GA, USA††booktitle: Proceedings of the 31st ACM
Int’l Conference on Information and Knowledge Management (CIKM ’22), Oct.
17–21, 2022, Atlanta, GA, USA††doi: 10.1145/3511808.3557484††isbn:
978-1-4503-9236-5/22/10††ccs: Information systems Retrieval models and ranking

##  1\. Introduction

Motivation.  Commonsense knowledge (CSK) is crucial for robust AI applications
such as question answering and chatbots. The purpose is to enrich machine
knowledge with properties about everyday concepts (e.g., gorilla, pancake,
newspaper). Such statements are acquired, organized and stored in structured
knowledge bases (KBs) (Pan2016; DBLP:series/synthesis/2021Hogan;
DBLP:journals/ftdb/WeikumDRS21). Large commonsense KBs (CSKBs) include
ConceptNet (conceptnet), WebChild (webchild), ATOMIC (atomic), TransOMCS
(DBLP:conf/ijcai/ZhangKSR20), and Ascent (ascent). These projects are almost
exclusively focused on positive statements such as gorillas are mammals,
black, and live in forests, expressed in the form of subject-relation-object
triples, e.g., (gorilla, AtLocation, forest). This allows QA systems, for
instance, to answer “Where do gorillas live?”. On the other hand, CSKBs hardly
capture any negative statements such as “gorillas are not territorial” or
“gorillas are not carnivorous”. By the Open-world Assumption (OWA) underlying
most KBs, one cannot assume that an absent statement is invalid (FHPPW2006);
instead, its truth is simply unknown. While KB completion (WPKD2020;
DPWQ+2019; Malaviya_Bhagavatula_Bosselut_Choi_2020) is an active research
area, creating an ideal KB that fully represents real-world knowledge is
elusive, especially for the case of commonsense assertions
(DBLP:journals/ftdb/WeikumDRS21). Therefore, QA over KBs cannot answer “Are
gorillas territorial?”. However, such uncommon knowledge has value for robust
AI applications, asserting that gorillas are not territorial, unlike other
apes (and monkeys) like chimpanzees or gibbons.

State of the art and its limitations.  The focus in constructing CSKBs has
been on positive statements; only very few projects capture a small fraction
of negative statements. In ConceptNet (conceptnet), a crowdsourced KB, 6
negative relations are represented, namely NotIsA, NotCapableOf, NotDesires,
NotHasA, NotHasProperty, and NotMadeOf. Nonetheless, in its latest version,
the portion of negative statements is less than 2%. Moreover, many statements
are all but informative, such as (envelope, NotHasProperty, alive). In the
automatically constructed, Web-based CSKB Quasimodo (quasimodo), 350k negated
statements represent about 10% of all statements, but these are dominated by
uninformative knowledge, e.g., $\neg$(elephant, can, quit smoking). A recent
method that targets the problem of discovering relevant commonsense negations
is NegatER (safavi-etal-2021-negater; negaterworkshop). Given a CSKB and a
pre-trained language model (LM), e.g., BERT (BERT), in order to strengthen the
LM’s ability to classify true and false statements, the LM is first fine-tuned
using the CSKB statements. In a second step, plausible negation candidates are
generated using dense k-nearest-neighbors retrieval, by either replacing the
subject or the object with a neighboring phrase. In a final step, the set of
plausible candidates are ranked, using the fine-tuned LM, by descending order
of negativeness (i.e., higher scores are more likely to be negative). Even
though NegatER compiles lists of thematically-relevant negations, it suffers
from several limitations: (i) The taxonomic hierarchy between concept phrases
is not considered. For instance, from the positive statement (horse, IsA,
expensive pet), a semantically sensible corruption of the subject is hamster,
but horse riding or horserider are not. Even though they are closer in
embedding space, they describe concepts of completely different types
(activity, artifact) that cannot be pets. (ii) The method relies on the input
CSKB having well-defined relations (e.g., CapableOf). This causes issues when
triples are merely short phrases with no canonicalized relations (e.g., as in
the Quasimodo CSKB); (iii) The ranking based on the LM’s negativeness
prediction is not interpretable, and follows no clear trend.

Approach and contributions.  This paper presents the UnCommonSense method for
identifying informative negations about concepts in CSKBs. For a target
concept like gorilla, we first compute a set of comparable concepts (e.g.,
lion, zebra), by employing both structured taxonomies and latent similarity.
Among these concepts, we postulate a Local Closed-world Assumption (LCWA)
(galarraga2017predicting), and consider their positive statements that do not
hold for the target concept as candidate negations (e.g, has tail, is
territorial). To eliminate false positives, candidates are scrutinized against
related statements in the input KB using sentence embeddings, and against a
pre-trained LM acting as an external source of latent knowledge. In a final
step, we quantify the informativeness of negative statements by statistical
scores, and generate top-ranked negations with provenances showing why certain
negations are interesting. For instance, $\neg$(gorilla, has, tail), unlike
other land mammals, e.g., lion and zebra.

The salient contributions of this work are:

  1. (1)

We present a method for identifying informative negations about everyday
concepts in large-scale CSKBs grounded in taxonomic hierarchies between
concepts.

  2. (2)

We showcase the ability of our method to produce interpretable negations via
human-readable phrases.

  3. (3)

In intrinsic evaluations, our method achieves up to 18% improvement in
informativeness and 17% in recall, compared to the prior state-of-the-art.

  4. (4)

In three extrinsic evaluations, (i) trivia summaries, (ii) KB completion, and
(iii) multiple-choice QA, our method shows substantial improvements in
informativeness.

  5. (5)

We release the first large dataset of informative commonsense negations,
containing over 6 Million negative statements about 8,000
concepts.111https://www.mpi-inf.mpg.de/departments/databases-and-information-
systems/research/yago-naga/commonsense/uncommonsense

##  2\. Problem and Design Space

A commonsense KB consists of a finite set of statements in the form (s, r, t),
where s is a subject (or concept), r is a relation, and t is a tail phrase.
Following previous work (chalierjoint), we do not distinguish between r and t,
because for textual CSK expressions, these distinctions are often ad-hoc and a
crisp definition of relations is difficult. Hence, for the remainder of the
paper, we generalize the above form to (s, f), where s is the subject and f is
a short phrase combining r and t.

###### Definition 2.1.

A commonsense negative statement $\neg$(s, f), is a statement (s, f) that is
not true.

For example, “elephants are not carnivorous” is expressed as $\neg$(elephant,
is carnivore). One naive approach to produce such negations is to assume the
CWA (closed-world assumption) over the KB and consider all non-existing
statements as negatives. On top of not being materializable, this approach
faces the following challenges. C1: Avoid false negatives. In order to assert
a negation, it is not sufficient to check if a candidate negation is not
positive. KBs in general operate under the OWA (open-world assumption), which
means that absent information is merely unknown, and not necessarily false.
For example, in Ascent, the absence of statement (elephant, has eye) is
clearly due to missing information. C2: Generate judgeable negations. Whether
constructed using human crowdsourcing (conceptnet) or information extraction
techniques (ascentpp; quasimodo), KBs mainly reflect the “wisdom” of the crowd
about everyday concepts. This causes the augmentation of many subjective or
otherwise uninformative statements, such as (cat, is important) and (football,
is boring). A generated negation must be easily interpreted by a human
annotator as true or false. Therefore it is important to clean the candidate
space prior to materializing negations. C3: Generate informative negations.
Finally, the explicit materialization of all possible negations is not
necessary for most standard AI applications (e.g., user might confuse
tabbouleh as something that requires an oven but not a printer). In other
words, it is better to avoid nonsensical negative statements such as
$\neg$(printer, is baked in oven).

Research problem.  Given a target concept s in a CSKB, generate a ranked list
of truly negative and informative statements.

##  3\. The UnCommonSense Framework

We present UnCommonSense, a method for automatically identifying informative
negative knowledge about everyday concepts. UnCommonSense first retrieves
comparable concepts for a target concept s by exploiting embeddings and
taxonomic relations between these concepts. Over the positive knowledge about
these comparable concepts, a local closed-world assumption (LCWA)
(galarraga2017predicting) is made. These relevant positives are then
considered as potential informative negations for s. Consequently, these
candidates might contain many false negatives and nonfactual statements. This
is followed by an inspection step, where we use KB-based and LM-based checks
to measure the plausibility of candidates. Finally, to measure
informativeness, the remaining candidates are scored using relative frequency.
An overview is shown in Figure 1.

Figure 1. Architecture of UnCommonSense.

###  3.1. Identifying Comparable Concepts

To increase the thematic relevance of candidate negations, we define the parts
of the KB where the CWA is helpful to assume (galarraga2017predicting), i.e.,
the LCWA. For instance, if the target concept is an animal, negations should
mostly reflect animal-related statements such as “not carnivorous” or “not
nocturnal”, instead of “not beverage” or “cannot store data”. Therefore, we
need to collect comparable concepts (recoin). One way for collecting related
concepts is using pre-computed embeddings. For instance, elephant is related
to both tiger and lion, due to their proximity in the vector space
(KGembsurvey). The problem with relying solely on this similarity function is
that it does not take into consideration the taxonomic hierarchy of the
concepts. For example, trunk, circus, and jungle are also highly related to
elephant. Instead, one can consider using large collections of taxonomic
relations and collect comparable concepts only if they are listed as co-
hyponyms (e.g., lion and elephant are, trunk and elephant are not). Although
this option ensures that related concepts are taxonomic sibling, the group of
siblings is unordered. For instance, even though lion and spider are both
acceptable taxonomic siblings (under animal), one is clearly more related to
elephant than the other. Moreover, large-scale taxonomies are noisy. For
instance, using WebIsALOD (webisalod), elephant and robot are co-hyponyms
under the class toy. We overcome these limitations by combining both
techniques and compute comparable concepts that are both semantically and
taxonomically highly related. Given concept s:

  1. (1)

Using latent representations (wikipedia2vec), we compute the cosine similarity
score between embeddings of s and every other concept in the KB, and rank them
by descending order of similarity.

  2. (2)

Using hypernymy relations (wang-etal-2017-short), we retain siblings that are
co-hyponyms of s. In particular, for every concept, we collect the top-5
hypernyms (ranked by confidence score222Using WebIsALOD’s SPARQL endpoint:
https://webisadb.webdatacommons.org/). For instance, elephant has 843
hypernyms. Top ones include larger animal, land animal, and mammal, and bottom
ones include work of art, african, and symbol of power. We retain KB concepts
as comparable to our target concept if they pass the following taxonomic
checks: (i) There exists a common hypernym with the target concept (e.g., both
elephant and tiger share mammal), and (ii) There does not exist an IsA
relation with the target concept, e.g., african elephant, IsA, elephant, hence
african elephant is not a valid sibling.

The ideal number of comparable concepts to consider for every target concept
is a hyperparamater $\gamma$, which we tune in Section LABEL:sec:ablation. For
the remainder of the paper, we use the terms comparable concepts and siblings
interchangeably.

###### Example 3.1.

Given s = elephant from KB = Ascent, and $\gamma$ = 3, the concepts with the
highest cosine similarity are computed using Wikipedia2Vec (wikipedia2vec).
The ranked candidate concepts include tiger, lion, trunk, horse, $\dots$.
Here, trunk is an obvious intruder as it is not share a hypernym with s. This
is determined using WebIsALOD (webisalod; madoc47550), an Is-A database,
containing 400m hypernymy relations, mined, using over 50 Hearst-style
patterns, from a huge web crawl. We end up with the closest 3 siblings: tiger,
lion, and horse. This initial step is meant to address challenge C3, which we
further demonstrate in Section LABEL:sec:ablation (Table LABEL:tab:ablation).

###  3.2. Candidate Negation Inference

To produce a set of candidate negations, we query from the KB the set of
positives about s as well as positives about its siblings. We subtract both
sets to produce an initial set of candidates $N$:

(1) $N=B\ \backslash\ A$

where $B$ is the set of phrases describing sibling concepts (i.e., each phrase
holds for at least one sibling), and $A$ is the set of phrases that hold for
the target concept. So, $N$ contains phrases that are $\in$ $B$ but $\notin$
$A$.


# Event Detection in Twitter: A Content- and Time-Based Analysis*

1st Izzat Alsmadi  Dept. of computing and cyber security
Texas A&M, San Antonio
San Antonio, Texas
<EMAIL_ADDRESS>2nd Michael J. O’Brien  Office of the Provost
Texas A&M, San Antonio
San Antonio, Texas
<EMAIL_ADDRESS>
###### Abstract

The detection of events from online social networks is a recent, evolving
field that attracts researchers from across a spectrum of disciplines and
domains. Here we report a time-series analysis for predicting events. In
particular, we evaluated the frequency distribution of top n-grams of terms
over time, focusing on two indicators: high-frequency n-grams over both short
and long periods of time. Both indicators can refer to certain aspects of
events as they evolve. To evaluate the model’s accuracy in detecting events,
we built and used a Twitter dataset of the most-popular hashtags that
surrounded the well-documented protests that occurred at the University of
Missouri (Mizzou) in late 2015.

###### Index Terms:

Event detection, Feature extraction, N-gram time series, Online social
networks, Predictive models

##  I Introduction

Users create an enormous amount of content through online social networks
(OSNs), and tracking and extracting events from such content solely through
human labor is impossible. Time is often critical with respect to certain
types of events (e.g., security or safety incidents), so there is a need for
methods that can be used to track and create alerts for events that users talk
about or report through OSNs. Content in OSNs is driven by individual
activities or events/news relevant to many users, and the extraction of
knowledge regarding contexts and events rests on being able to aggregate
information beyond a single activity content or user-account activity. Events
themselves can be placed in different categories, such as type of event or
whether it was identified using supervised or unsupervised techniques [1]. In
the former, dictionaries or datasets of recent significant events can be used
to help classifiers make their choice of labels from a pool of alternatives
[2]. If event detection is supervised or based on a predefined list of events,
keyword-based filtering can be used ([3], [4] ). Similarly, Twitter streams
can be collected as either topic streams, which can be used as the tweets
label events, or random streams, which have no specific topics [5]. Event-
detection techniques in Twitter face similar challenges related to the
limitation of text size, spam, and other types of irrelevant contents, along
with the use of abbreviations, slang language, and OSN-specific terms [6]. Our
focus here is on social disruption, especially ones that are sudden. How can
normal events turn into disruptive events? Some national or international
events that turn into wide-spread disruption can occur on other occasions
without triggering similar large publicity. Why? In data-analytic research,
assumptions are made that disruptive events can be categorized through
temporal and textual features, ([7], [8]). In reality, however, social and
political factors can also contribute to such disruptive events. Those events
are not pre-planned, not expected, and mostly unwelcome [9]. They can be
triggered by natural disasters and by political or social events at regional,
national, and international levels.

##  II Related Work

Research in event detection in OSNs can be divided into several categories,
two of which are discussed here because of their relevance to our analysis.
One is a word or word-pair analysis: ([18], [19], [20], [21], [22], [23]),
where an event can be detected by following certain keywords that show sudden
bursts in frequency [18]. Trends on Twitter, for example, follow bursts of
features or keywords that frequently occur together [24]. Hossny and Mitchell
[23] evaluated the time series of the occurrence of word pairs and their
association with relevant events. We adopted a similar approach but extended
it beyond word pairs to an n-number of words. The second category of event-
detection methods focuses on topics-based detection ([25], [26], [27]). On
Twitter, topics can be categorized by a hashtag or a mention. Whereas a
hashtag is any keyword preceded by a hash sign, a mention is a tweet that
contains another user’s @username anywhere in its body. The analysis of
mentions and hashtags helps find users who have similar interests. This is
very important for many data analytics-based applications besides event
detection, such as studying types and natures of interactions and groups among
users in OSNs. Users can “favorite” and retweet the posts of other users as
well as engage in conversations using @mentions, replies, and hashtags. Events
are ranked based on (1) the average “burstiness” of topics [28], (2) the
average usage of emergent topic terms [29], or (3) latent Dirichlet allocation
[30]. Events can be detected based on analyzing user relationships [31].
Ideally, if we’re using Twitter, we need an event-detection model based on
tweet segments or related terms that occur more often (e.g., more than a
threshold level). For disruptive-events detection, the relevant segments
should contain more than one term. Further, they should exist in many tweets
from many accounts (e.g., include thresholds on the minimum number of tweets
and number of accounts). They should also exist on more than certain
consecutive time units (e.g., more than a threshold of days). They also need
to show up in the top m segments for more than n consecutive time units.
Search engines can be used to confirm where segments are popular (e.g., above
a certain threshold of search results). A “stickiness” function is defined for
optimal segmentation of tweets based on three factors: length normalization,
the segment’s presence in Wikipedia, and the segment’s “phraseness,” or the
probability of being a phrase based on global and local contexts [32].
Relevant research employs similar models for detecting “hot” topics ([33],
[34], [35], [36], [37]. Some studies indicate that hot-topics detection can be
used to identify rumors and other types of false information in OSNs. Bursty
event detection is used to detect hot topics and events that arise quickly,
especially in OSNs.

##  III The Event-Detection Model Based on Tweet Segmentation

Our event-detection model is based on the following methods and constraints:

  * •

Using Tweet segmentation to split a tweet into a sequence of consecutive
n-grams, each of which is called a segment. A segment can be a named entity
(e.g., a movie title, such as “Finding Nemo”) or a semantically meaningful
information unit [32].

  * •

Hashtags can be a good starting point, but a hashtag is only one word without
spaces. The relevant segments should contain more than one term. On the other
hand, we noticed that in a dataset of related tweets, hashtags will appear as
popular words or n-grams. As a result, we evaluated the use of hashtag
segmentation for event detection ([38], [39], [40]).

  * •

Segments should exist in many tweets from many accounts (thresholds on the
number of tweets and number of accounts).

  * •

Segments should also exist on more than a few consecutive days; they need to
show up in the top-10 segments for more than n consecutive days.

  * •

Segments should be popular or trending in search engines or on social networks
within days of interest.

Fig. 1 shows our proposed overall architecture for Twitter-based event
detection based on initial hashtags.

Figure 1: Twitter event-detection architecture

##  IV Experiments and Analysis

As the subject of this research is not a current event, and no public dataset
from Twitter can be used, we collected a set of related tweets based on
Twitter public API (https://developer.twitter.com/en/docs/twitter-api). In
order to ensure that collected tweets were relevant to the Mizzou protests, we
used a selection of popular hashtags related to the event as an input to the
tweet-collection process. We selected a period of approximately a year around
the protests and their immediate aftermath (November 1, 2015, to December 31,
2016).

###  IV-A Mizzou protests popular hashtags

We defined the following four hashtags as being the most relevant to the
event: #PrayForMizzou, #ConcernedStudent1950, #BlackLivesMatter, and #Mizzou.
Table I shows basic statistics on the tweets collected as a result of those
hashtags. The table also shows the average positive and negative polarity of
the sentiment in the tweets for each hashtag found using TextBlob sentiment
library (vol. 15.3).

TABLE I: Top hashtags surrounding the Mizzou 2015 protests

Hashtag | NO. of Tweets | Avg Pos. Sentiments | Avg Neg. entiments
---|---|---|---
#PrayForMizzou | 10,000 | 0.537517 | 0.462483
#MizzouHungerStrike | 6,514 | 0.632995 | 0.367005
#Mizzou | 71,630 | 0.550789 | 0.449211
#BlackLivesMatter | 71,421 | 0.550862 | 0.449138

Fig. 2 shows the hashtags’ timelines. The timelines for two
hashtags—#PrayForMizzou and #ConcernedStudent1950—are restricted to November
2015, with another—Mizzou—peaking during that month but lasting throughout the
timeline. #BlackLivesMatter fluctuates randomly throughout the timeline. The
timelines of the hashtags can show indications of how the focus or the subject
of the protest evolves. Fig. 3 shows a word cloud for the top single terms in
the dataset that contain tweets from all hashtags. The cloud shows major terms
from the protests as well as from sports-related activities.

Figure 2: Timeline for four hashtags related to the Mizzou protests

Figure 3: Mizzou-related tweets word cloud

### N-grams-based tweets segmentation

In order to extract events from a large corpus of text, we studied patterns of
statements and phrases. Our goal was to segment tweets into n-grams of words.
We limited our experiments to n-grams of 2-5 words. Table II shows the top-20
bi-gram terms. The majority of those share the term “Mizzou” in common, but
little in terms of relevant events can be extracted from those bi-grams. Table
III shows the top-20 tri-grams. Here we can start observing more possibilities
for extracting significant events. Tables IV and V shows grams for four and
five terms, respectively. We can see that with some necessary cleaning, those
grams will be better candidates to detect significant events in the tweets
dataset. Typical text analysis pre-processing includes removing stop words,
which are commonly used words such as articles, pronouns, and prepositions.
Each language typically includes a large list of stop words that can be used.
In our experiment, we did not employ stop-word removal in pre-processing
stages, as our goal was to search for tweet segments that could serve as
phrases or statements in the event-detection process.

TABLE II: Top bi-grams surrounding the Mizzou 2015 protests

Bi-gram | Count | Bi-gram | Count
---|---|---|---
pic twitter | 14031 | i m | 4737
twitter com | 14031 | it s | 4635
at mizzou | 9878 | on the | 4497
in the | 8050 | mizzou https | 4222
mizzou s | 6577 | for the | 3877
the mizzou | 6455 | to the | 3738
to mizzou | 6228 | mizzou mizzou | 3711
for mizzou | 6206 | mizzou football | 3566
of the | 5770 | don t | 3360
mizzou is | 5644 | at the | 3346

TABLE III: Top tri-grams surrounding the Mizzou 2015 protests.

Tri-gram | Count | Tri-gram | Count
---|---|---|---
pic twitter com | 14031 | let s go | 597
mizzou pic twitter | 2220 | to be a | 568
at mizzou arena | 1234 | miz pic twitter | 537
the mizzou game | 1146 | go to mizzou | 530
i don t | 1026 | i can t | 508
a mizzou fan | 931 | it s a | 505
in the sec | 753 | prayforbeirut prayformexico prayformizzou | 500
at the mizzou | 712 | prayforparis prayforjapan prayforbeirut | 497
to the mizzou | 666 | is going to | 485
mizzou https mizzou | 661 | can t wait | 483

TABLE IV: Top four-grams surrounding the Mizzou 2015 protests.

Four-gram | Count | Four-gram | Count
---|---|---|---
mizzou pic twitter com | 2220 | phenom rb arrested hours | 367
miz pic twitter com | 537 | rb arrested hours after | 367
prayforjapan prayforbeirut prayformexico prayformizzou | 442 | arrested hours after breaking | 355
prayforparis prayforjapan prayforbeirut prayformexico | 399 | breaking school records mug | 348
freshman phenom rb arrested | 388 | school records mug shot | 327
prayformizzou pic twitter com | 387 | mizzou tweets mizzou tweets | 324
hours after breaking school | 372 | can t wait to | 300
after breaking school records | 372 | let s go mizzou | 294
mizzou football freshman phenom | 370 | she took exams for | 294
football freshman phenom rb | 370 | took exams for mizzou | 285

TABLE V: Top five-grams surrounding the Mizzou 2015 protests.

Five-gram | Count | Five-gram | Count
---|---|---|---
prayforparis prayforjapan prayforbeirut prayformexico prayformizzou | 390 | she took exams for mizzou 285 |
hours after breaking school records | 372 | took exams for mizzou athletes | 285
mizzou football freshman phenom rb | 370 | missouri tigers vs az wildcats | 282
football freshman phenom rb arrested | 367 | school records mug shot http | 273
freshman phenom rb arrested hours | 367 | tutor says she took exams | 267
phenom rb arrested hours after | 367 | says she took exams for | 267
rb arrested hours after breaking | 355 | dec 10 10 missouri tigers | 264
arrested hours after breaking school 7351 | 10 10 missouri tigers vs | 264 |
after breaking school records mug | 348 | 10 missouri tigers vs az | 264
breaking school records mug shot | 327 | prayforjapan prayforbeirut prayformexico prayformizzou prayforlebanon | 243

Time series for hashtags and popular terms can help us visualize their
evolution in terms of frequency and dates/times. The large increase in some
terms within the short term is what we will use in events’ prediction. Fig. 4
shows a sample time series for the term “PrayForMizzou,” including but not
limited to the hashtag #PrayForMizzou. Fig. 4 shows a rise in the term of
almost 10,000 times within roughly a one-month period (November 2015). Fig. 5
shows correlated results between Twitter and Google Trends. We compared the
time series of #PrayForMizzou vs. #BlackLivesMatter. Fig. 5 shows that unlike
BlackLivesMatter, the term PrayForMizzou is associated with only a single
event (Mizzou protests). As hashtags are specially formed terms—a single word
formed from several words—they can be uniquely associated with some events.

Figure 4: #PrayForMizzou time series. The dots represent a few mentions after
December 2015.

Figure 5: #PrayForMizzou versus #BlackLivesMatter time series

We proposed three n-gram-based metrics that we wanted to evaluate in terms of
their ability to detect events:

  * •

Total count of top n-grams: Our observations as well as our literature review
indicated that significant events would be mentioned by many users. Our
assumption was that we could find some phrases that were repeated in high
volumes and that could be associated with certain events.

  * •

Duration: Our previous analysis of hashtags and Google Trends showed that
studying the frequencies or trends of certain phrases can be associated with
certain events. “Duration metric” refers to the difference in time between the
first and the last n-gram in the dataset.

  * •

Density: This is our key indicator to show the burst of the event by dividing
the total number of identical n-grams by the duration. Our detection model is
built around the analysis of a triangle of Twitter accounts, tweets n-grams’
frequencies, and tweets n-grams’ duration. Fig. 6 shows both frequencies and
duration for the top 138 three-grams. Based on Fig. 6 we identified two
categories of candidate n-grams:

  * •

A very high number of n-grams repeated over a short duration. Those would be
the data points that extend up the y-axis in Fig. 6.

  * •

A lower number of n-grams repeated over a longer duration. Those would be the
ones that occur around day 35 in Fig. 6 Those are second-level candidates for
detecting events, as they need to be evaluated further (e.g., using metrics in
addition to duration and frequency).

Figure 6: Duration versus count for three-gram hashtags

##  V Conclusion

The power and influence of Online Social Networks (OSNs) on our daily lives
are beyond question. People nowadays use those networks for social gatherings
and activities, movements, protests, and the like. Our effort was focused on
evaluating a model to detect and predict large-scale events based on OSNs:

  * •

We proposed and evaluated metrics that can be automatically collected based on
tracking certain inputs such as hashtags and Google Trends.

  * •

We evaluated our model based on the University of Missouri protests of 2015
and the aftermath.

  * •

We collected a large dataset of tweets around the events based on some of the
more popular hashtags used during that period.

  * •

We focused on top n-grams to generate candidate events. Our results showed
that n-grams of words are better than hashtags for predicting the occurrence
of events. There are several reasons for this. First, top n-grams of words
show popular word sequences, whereas hashtags represent single words or
phrases. Second, popular hashtags can be included in popular n-grams and
n-grams can show us combinations of popular hashtags related to each other.
For example, one of the popular three-grams is (prayformexico prayformizzou
prayforlebanon), which shows three popular, related hashtags that reflect
events in different parts of the world.

We proposed simple but effective metrics to calculate automatically what can
help predict the occur of events: n-gram frequency versus time. Based on the
relation between frequency and time, we identified two types of interesting
types: (1) high n-gram frequencies within a short time—a few hours or a few
days—and (2) high n-gram frequencies over a longer time—a few months, for
example. We think that both types can be good event predictors. And we need
good predictors because by the time we begin to sense that an event is upon
us, it might be too late to escape the aftermath. Take the case of the
University of Missouri. It took a little over a year for the fuse lit by the
August 2014 killing of Michael Brown in Ferguson, Missouri, to lead to the
first disruption in September 2015—a disruption that immediately led to
more—and more volatile—disruptions [43]. The toll exacted by disruptive events
can be significant. There is no doubt of the psychological and cognitive
fatigue caused by trolling, not to mention the possibility of financial
disaster. For example, in the two years following the Mizzou protests, the
university saw a 35% drop in freshman enrollment and an overall drop of 14%,
causing the campus to cut 12%—about $55 million—from the academic and
administrative operations budget [45]. This, in turn, led to significant
layoffs of faculty and staff. Even today, the campus is nowhere back to what
it was before the protests, neither financially, socially, nor politically.
The take-home message is clear: We need to continue to develop solid methods
for predicting disruptive events.

###  V-A Note

One of us, MOB, was dean of the College of Arts and Science at Mizzou during
the unrest and saw firsthand the protest and aftermath unfold.

## Acknowledgment

We thank Gloria O’Brien for her editorial assistance.

## References

  * [1] F. Atefeh, W. Khreich, A survey of techniques for event detection in Twitter, Computational Intelligence (2015) 132–164.
  * [2] Q. Le Sceller, E.B. Karbab, M. Debbabi, F. Iqbal, F. Sonar, Automatic detection of cyber security events over the Twitter stream, in: Proceedings of the 12th International Conference on Availability, Reliability and Security, 2017, pp. 1–11.
  * [3] T. Sakaki, M. Okazaki, Y. Matsuo, Earthquake shakes Twitter users: real-time event detection by social sensors, in: Proceedings of the 19th International Conference on the World Wide Web, 2010, pp. 851–860.
  * [4] Y. Kitagawa, M. Komachi, E. Aramaki, N. Okazaki, H. Ishikawa, Disease event detection based on deep modality analysis, in: Proceedings of the Association for Computational Linguistics–International Joint Conference on Natural Language Processing Student Research Workshop, 2015, pp. 28–34.
  * [5] S. Kumar, H. Liu, S. Mehta, L.V. Subramaniam, From tweets to events: exploring a scalable solution for Twitter streams, arXiv:1405.1392 (2014).
  * [6] A. Weiler, M. Grossniklaus, M.H. Scholl, Survey and experimental analysis of event detection techniques for Twitter, Computer Journal 60 (2017) 329–346.
  * [7] N. Alsaedi, P. Burnap, Feature extraction and analysis for identifying disruptive events from social media, in: Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, 2015, pp. 1495–1502.
  * [8] F. Miro-Llinares, J.J. Rodriguez-Sala, Cyber hate speech on Twitter: analyzing disruptive events from social media to build a violent communication and hate speech taxonomy, International Journal of Design & Nature and Ecodynamics 11 (2016) 406–415.
  * [9] E. Katz, T. Liebes. ‘No More Peace!’: how disaster, terror and war have upstaged media events, International Journal of Communication 1 (2007) 10.
  * [10] P.J. Lamberson, S.E. Page, Tipping points, Quarterly Journal of Political Science 7 (2012) 175–208.
  * [11] St. Louis Post-Dispatch, Missteps, not student revolt, led to Mizzou chancellor’s demise, Nov. 12, 2015.
  * [12] M.J. OBrien, R.A. Bentley, W.A. Brock, The Importance of Small Decisions, MIT Press, 2019.
  * [13] J. Stripling, How Missouri’s deans plotted to get rid of their chancellor, Chronicle of Higher Education, Nov. 22, 2015.
  * [14] New York Times, Long after protests, students shun the University of Missouri, July 9, 2016.
  * [15] J. Prier, Commanding the trend: social media as information warfare, Strategic Studies Quarterly 11(4) (2017) 50–85.
  * [16] Boatwright, B.C., Linvill, D.L., & Warren, P.L., Troll Factories: The Internet Research Agency and State-Sponsored Agenda Building, 2018, Resource Centre on Media Freedom in Europe. https://www.rcmediafreedom.eu/Publications/Academic-sources/Troll-Factories-The-Internet-Research-Agency-and-State-Sponsored-Agenda-Building.
  * [17] U.S. Department of Justice, United States of America v. Internet Research AgencyLLC A/K/A Mediasintez, et al., 2018, https://www.justice.gov/file/1035477/download.
  * [18] J. Kleinberg, Bursty and hierarchical structure in streams, Data Mining and Knowledge Discovery 7 (2003) 373–397.
  * [19] H. Sayyadi, M. Hurst, A. Maykov, Event detection and tracking in social streams, in: Third International AAAI Conference on Weblogs and Social Media, 2009, https://www.cs.umd.edu/ sayyadi/files/papers/9-sayyadi-EventDetection_KeyGraph_ICWSM09.pdf.
  * [20] J. Weng, B.S. Lee, Event detection in Twitter, in: Fifth International AAAI Conference on Weblogs and Social Media, 2011, https://www.aaai.org/Library/ICWSM/icwsm11contents.php.
  * [21] O. Ozdikis, P. Senkul, H. Oguztuzun, Semantic expansion of tweet contents for enhanced event detection in Twitter, in: 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, 2012.
  * [22] Y. Zhou, S. De, K. Moessner, Real-world city event extraction from Twitter data streams, Procedia Computer Science 98 (2016) 443–448.
  * [23] A.H. Hossny, L. Mitchell, Event detection in Twitter: a keyword volume approach, in: 2018 IEEE International Conference on Data Mining Workshops, 2018, pp. 1200–1208.
  * [24] M. Mathioudakis, N. Koudas, Twittermonitor: trend detection over the Twitter stream, in: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, 2010, pp. 1155–1158.
  * [25] A. Kontostathis, L.M. Galitsky, W.M. Pottenger, S. Roy, D.J. Phelps, A survey of emerging trend detection in textual data mining, in: M.W. Berry (Ed.), Survey of Text Mining: Clustering, Classification, and Retrieval, Springer, 2004, pp. 185–224.
  * [26] M. Naaman, H. Becker, L. Gravano, Hip and trendy: characterizing emerging trends on Twitter, Journal of the American Society for Information Science and Technology 62 (2011) 902–918.
  * [27] R. Long, H. Wang, Y. Chen, O. Jin, Y. Yu, Towards effective event detection, tracking, and summarization on microblog data, in: H. Wang, S. Li, S. Oyama, X. Hu, T. Qian (Eds.), Web-Age Information Management Lecture Notes in Computer Science, Springer, pp. 652–663.
  * [28] F. Alvanaki, M. Sebastian, K. Ramamritham, G. Weikum, EnBlogue: emergent topic detection in Web 2.0 streams, in: Proceedings of the ACM SIGMOD International Conference on Management of Data, 2011, pp. 1271–1274.
  * [29] M. Cataldi, L. Di Caro, C. Schifanella, Emerging topic detection on Twitter based on temporal and social terms evaluation, in: Proceedings of the Tenth International Workshop on Multimedia Data Mining, 2010, pp. 4.1–4.10.
  * [30] L. Shi, Y. Wu, L. Liu, X. Sun, L. Jiang, Event detection and identification of influential spreaders in social media data streams, Big Data Mining and Analytics 1 (2018) 34–46.
  * [31] Y. Du, Y. He, Y. Tian, Q. Chen, L. Lin, Microblog bursty topic detection based on user relationship, in: 6th IEEE Joint International Information Technology and Artificial Intelligence Conference (vol. 1), 2011, pp. 260–263.
  * [32] C. Li, A. Sun, J. Weng, Q. He, Tweet segmentation and its application to named entity recognition, IEEE Transactions on Knowledge and Data Engineering, 27 (2014) 558–570.
  * [33] D. Zheng, F. Li, Hot topic detection on BBS using aging theory, in: W. Liu, X. Luo, F.L. Wang, J. Lei (Eds.), International Conference on Web Information Systems and Mining, Springer, 2009, pp. 129–138.
  * [34] S. Ishikawa, Y. Arakawa, S. Tagashira, A. Fukuda, Hot topic detection in local areas using Twitter and Wikipedia, in: International Conference on Architecture of Computing Systems, 2012, pp. 1–5.
  * [35] Z. Yang, C. Wang, F. Zhang, Y. Zhang, H. Zhang, Emerging rumor identification for social media with hot topic detection, in: 12th Web Information System and Application Conference, 2015, pp. 53–58.
  * [36] W. Ai, K. Li, K. Li, An effective hot topic detection method for microblog on spark. Applied Soft Computing 70 (2018) 1010–1023.
  * [37] Q. Liqing, J. Wei, L. Haiyan, F. Xin, Microblog hot topics detection based on VSM and HMBTM model fusion, IEEE Access 7 (2019) 120273–120281.
  * [38] G. Berardi, A. Esuli, D. Marcheggiani, F. Sebastiani, ISTI@ TREC Microblog Track 2011: Exploring the Use of Hashtag Segmentation and Text Quality Ranking, https://trec.nist.gov/pubs/trec20/papers/NEMIS_ISTI_CNR.microblog.update.pdf.
  * [39] L.A.F. Park, G. Stone, The effect on accuracy of tweet sample size for hashtag segmentation dictionary construction, in: J. Bailey, L. Khan, T. Washio, G. Dobbie, J. Huang, R. Wang (eds.), Advances in Knowledge Discovery and Data Mining, Springer, 2016, pp. 382–394.
  * [40] A. Celebi, A. Özgür, Segmenting hashtags and analyzing their grammatical structure, Journal of the Association for Information Science and Technology, 69 (2018) 675–686.
  * [41] A. Guille, C. Favre, Event detection, tracking, and visualization in Twitter: a mention-anomaly-based approach, Social Network Analysis and Mining 5 (2015) 18.
  * [42] A. Guille, C. Favre, Mention-anomaly-based event detection and tracking in Twitter, in: 2014 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pp. 375–382.
  * [43] M. Pearson, A timeline of the University of Missouri protests. CNN, November 10, 2015, https://www.cnn.com/2015/11/09/us/missouri-protest-timeline/index.html.
  * [44] J. Aro, The cyberspace war: propaganda and trolling as warfare tools, European View, 15 (2016) 121–132.
  * [45] R. Keller, University of Missouri enrollment to decline more than 7 Percent; 400 jobs to be eliminated, Columbia Daily Tribune, May 15, 2017.