# MEE: A Novel Multilingual Event Extraction Dataset

Amir Pouran Ben Veyseh1, Javid Ebrahimi1,
Franck Dernoncourt2, and Thien Huu Nguyen1
1Department of Computer Science, University of Oregon, Eugene, OR, USA
2Adobe Research, Seattle, WA, USA
{apouranb<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Event Extraction (EE) is one of the fundamental tasks in Information
Extraction (IE) that aims to recognize event mentions and their arguments
(i.e., participants) from text. Due to its importance, extensive methods and
resources have been developed for Event Extraction. However, one limitation of
current research for EE involves the under-exploration for non-English
languages in which the lack of high-quality multilingual EE datasets for model
training and evaluation has been the main hindrance. To address this
limitation, we propose a novel Multilingual Event Extraction dataset (MEE)
that provides annotation for more than 50K event mentions in 8 typologically
different languages. MEE comprehensively annotates data for entity mentions,
event triggers and event arguments. We conduct extensive experiments on the
proposed dataset to reveal challenges and opportunities for multilingual EE.

##  1 Introduction

Event Extraction (EE) is one of the major tasks of Information Extraction (IE)
for text. In a complete EE pipeline, three major goals should be pursued: (1)
Entity Mention Detection (EMD): to recognize mentions of real world entities;
(2) Event Detection (ED): to identify event mentions/triggers and their types.
An event trigger is a word or phrase that most clearly refers to the
occurrence of an event; and (3) Event Argument Extraction (EAE): to find
participants/arguments of an event mentioned in text. A participant is an
entity mention that has an specific role in a given event mention. For
instance, in the sentence “The soldiers were hit by the forces.”, there are
two entity mentions “soldiers” and “forces” of types PERSON and ORGANIZATION
and an event trigger “hit” of type ATTACK. Also, the two event mentions
“soldiers” and “forces” play the argument roles of Victim and Attacker
(respectively) in the ATTACK event. An EE system could be employed in other
downstream applications such as Question Answering, Knowledge Base Population
and Text Summarization to assist extracting information about events in text.

Multiple methods have been proposed for Event Extraction. Early work has
employed feature-based models Ahn (2006); Ji and Grishman (2008); Liao and
Grishman (2010); Hong et al. (2011); Li et al. (2013); Yang and Mitchell
(2016) while later methods have explored deep learning to present state-of-
the-art performance for Event Extraction Nguyen and Grishman (2015); Chen et
al. (2015); Nguyen et al. (2016); Sha et al. (2018); Wang et al. (2019); Lai
et al. (2020); Veyseh et al. (2020); Lin et al. (2020); Nguyen et al. (2021a);
Liu et al. (2022). However, despite all advancements on event extraction in
recent years, a major limitation of current EE research is to overly focus on
a few popular languages, thus failing to adequately reveal challenges and
generalization of models in many other languages of the world. As such, a
critical barrier for studying EE over multiple languages is the lack of high
quality datasets that fully annotate data for many other languages for EE. For
instance, the most popular dataset for EE, i.e., ACE 2005 Walker et al.
(2006), only provide annotations for three languages English, Chinese and
Arabic while TAC KBP datasets Mitamura et al. (2016, 2017) only supports
English, Chinese and Spanish. The TempEval-2 dataset Verhagen et al. (2010)
involves 6 languages; however, it does not offer event argument annotation.
Even worse, recently created datasets, e.g., MAVEN Wang et al. (2020), RAMS
Ebner et al. (2020), and WikiEvents Li et al. (2021), are only annotated for
English. In all, such language and task limitations prevents research to
comprehensively develop and evaluate EE methods over different languages and
multilingual settings. Moreover, the limited size of these datasets, i.e. less
than 11K and 27K in ACE 2005 and TempEval-2 respectively, hinders training of
data-hungry deep learning models. Finally, we note that important multilingual
datasets for EE, e.g., ACE 2005 and TAC KBP, are not publicly available, which
further restricts research on this domain.

To address these limitations, in this work, we propose a large-scale
Multilingual Event Extraction (MEE) dataset that covers 8 typologically
different languages from multiple language families, including English,
Spanish, Portuguese, Polish, Turkish, Hindi, Korean, and Japanese. As such,
Portuguese, Polish, Turkish, Hindi, and Japanese are not explored in the
popular multilingual datasets for EE, i.e., ACE 2005 and TAC KBP. Importantly,
to enable public data sharing and diversity the data, we employ Wikipedia
articles for the 8 languages in diverse topics (i.e., Economy, Politics,
Technology, Crime, Nature and Military) for EE annotation.

Our dataset comprehensively annotates each document in a language for all the
three sub-tasks EMD, ED, and EAE. To be consistent with prior EE research, we
inherit the type anthologies for such tasks from the ACE 2005 dataset that
provides well-designed guidelines and examples for the types. In particular,
we include 7 entity types, 8 event types and 16 event sub-types, along with 23
argument roles in MEE to facilitate EE annotation over multiple languages.
Overall, our dataset involves more than 415K entity mentions, 50K event
triggers, and 38K arguments, which are much larger than previous multilingual
EE datasets to better support model training and evaluation with deep
learning.

Due to shared information schema over all the languages, our MEE dataset
enables cross-lingual transfer learning evaluation of MEE models where
training and test data comes from different languages. To this end, we conduct
comprehensive experiments for both monolingual and cross-lingual learning
settings to provide insights for language-specific challenges and cross-
lingual generalization of EE methods. By examining both pipeline and joint
inference models for EE, our experiments show that the proposed dataset
present unique challenges with less satisfactory performance of existing EE
models, especially for cross-lingual settings, thus calling for more research
efforts for multilingual EE in the future.

Category | English | Portuguese | Spanish | Polish | Turkish | Hindi | Japanese | Korean
---|---|---|---|---|---|---|---|---
Economy | 1,095 | 112 | 168 | 315 | 297 | 189 | 199 | 250
Politics | 3,202 | 308 | 772 | 1,270 | 1,233 | 349 | 232 | 248
Technology | 2,171 | 189 | 400 | 712 | 815 | 295 | 312 | 249
Crimes | 893 | 78 | 220 | 152 | 118 | 95 | 80 | 73
Nature | 1,195 | 398 | 705 | 455 | 398 | 245 | 299 | 185
Military | 4,444 | 415 | 1,003 | 1,575 | 1,619 | 326 | 378 | 495
Total | 13,000 | 1,500 | 3,268 | 4,479 | 4,480 | 1,499 | 1,500 | 1,500

Table 1: Numbers of annotated segments in each Wikipedia subcategory for our 8
languages.

##  2 Data Annotation

We follow the entity/event type definition and annotation guidelines from the
popular ACE 2005 dataset to benefit from its well-designed documentation and
be consistent with prior EE research. As such, entity mentions refer to
mentions of real-world entities in text that can be expressed via names,
nominals, and pronouns. Entity Mention Extraction (EMD) is more general than
Named Entity Recognition that only concerns names of entities. In addition, an
event is defined as an incident whose occurrence changes the state of real
world entities. An event mention is the part of input text that refers to an
event that consists of two components: (1) Event Trigger: the words that most
clearly refer to the occurrence of the event. It is noteworthy that we allow
an event trigger to span multiple words to accommodate trigger annotation for
multiple languages. For instance, in the Turkish phrase “tayin etmek”, both
words are necessary to indicate an event trigger of type “Appoint"; and (2)
Event Arguments: the entity mentions that are involved in the event with some
roles.

Based on the ACE 2005 dataset, our dataset annotates entity mentions for 7
entity types: PERSON (human entities), ORGANIZATION (corporations, agencies,
and other groups of people), GPE (geographical regions defined by political
and/or social groups), LOCATION (geographical entities such as landmasses or
bodies of water), FACILITY (buildings and other permanent man-made
structures), VEHICLE (physical devices primarily designed to move an object
from one location to another), and WEAPON (physical devices primarily used as
instruments for physically harming). For event types, to avoid confusion and
improve data quality, we prune the original ACE 2005 event types to only
include the types that are not ambiguous across multiple languages. For
instance, in Turkish, the event types Sentence and Convict are very similar
(both can be evoked by the phrase “Mahkum etmek”) so they are not retained in
our dataset. As such, we preserve 8 event types and 16 sub-types that are
distinct enough for annotation in our dataset. Finally, for event arguments,
we preserve all 23 argument roles in the ACE 2005 dataset. Table 4 shows the
list of event types along with their argument roles in our dataset.

###  2.1 Data Preparation

Our dataset MEE covers 8 different languages, i.e., English, Spanish,
Portuguese, Polish, Turkish, Hindi, Korean and Japanese. These languages are
selected based on their diversity in terms of typology and their novelty with
respect to existing multilingual EE datasets. For each language, we employ its
latest dump of Wikipedia articles as raw data for annotation. To focus on
event data, we select articles in the sub-categories under category Event in
Wikipedia. In particular, the following sub-categories are considered to
improve topic diversity: Economy, Politics, Technology, Crimes, Nature, and
Military. Note that we start with these categories in English Wikipedia.
Afterward, we follow interlinks between the categories in different languages
to locate the intended categories for Wikipedia for non-English languages in
MEE.

We process the collected articles with the WikiExtractor tool Attardi (2015)
to obtain clean textual data and meta-data for each article. The textual data
is then split into sentences and tokenized into words by the multilingual NLP
toolkit Trankit Nguyen et al. (2021b). Afterward, to annotate the data with
entity and event mentions, one approach is to directly ask annotators to read
each article entirely for annotation. However, as the articles in Wikipedia
might be lengthy, this approach can be overwhelming for annotators, thus
hindering their attention and lowering quality of annotated data. To address
this issue, we follow prior dataset creation efforts for EE, i.e., RAMS Ebner
et al. (2020), to divide the articles into segments of five consecutive
sentences. Each segment will then be annotated separately for EE tasks so
annotators can better capture the entire context to provide entity and event
annotation. Note that similar to RAMS, we annotate all event arguments in a
text segment for each event trigger, thus allowing event arguments to appear
in different sentences from the event trigger (i.e., document-level EAE).
Finally, to accomodate our budget, a sample of text segments is obtained for
each language for annotation. The numbers of selected text segments for each
category per language in our dataset are presented in Table 1.

Language | #Seg. | Avg. Length | #Entities | #Triggers | #Arguments | Challenging Entity Type | Challenging Trigger Type | Language Family
---|---|---|---|---|---|---|---|---
English | 13,000 | 123 | 190,592 | 17,642 | 13,548 | GPE | Personnel | Germanic
Spanish | 3,268 | 112 | 48,001 | 6,064 | 802 | GPE | Conflict | Italic
Portuguese | 1,500 | 102 | 25,463 | 1,953 | 12,329 | Location | Personnel | Italic
Polish | 4,479 | 108 | 62,971 | 10,875 | 3,395 | Facility | Transaction | Balto-Slavic
Turkish | 4,480 | 117 | 38,469 | 8,390 | 1,416 | GPE | Personnel | Turkic
Hindi | 1,499 | 98 | 18,797 | 1,810 | 2,117 | Facility | Conflict | Indo-Iranian
Japanese | 1,500 | 99 | 19,174 | 2,152 | 3,399 | Location | Personnel | Japonic
Korean | 1,500 | 103 | 12,508 | 1,125 | 1,742 | GPE | Personnel | Koreanic
Total (MEE) | 31,226 | - | 415,975 | 50,011 | 38,748 | - | - | -

Table 2: Statistics of the MEE dataset. #Seg. represents the numbers of
annotated text segments for each language. All annotated segments consist of 5
sentences and their lengths (Avg. Length) are computed in terms of numbers of
tokens. “Challenging Type” indicates the types where entity or event trigger
annotation involves largest disagreement between annotators in each language.

Language | #Annotator | EMD | ED | EAE
---|---|---|---|---
English | 10 | 0.792 | 0.834 | 0.820
Spanish | 10 | 0.788 | 0.812 | 0.823
Portuguese | 5 | 0.791 | 0.803 | 0.799
Polish | 8 | 0.780 | 0.799 | 0.813
Turkish | 10 | 0.785 | 0.813 | 0.822
Hindi | 6 | 0.790 | 0.803 | 0.812
Japanese | 5 | 0.793 | 0.789 | 0.780
Korean | 6 | 0.802 | 0.810 | 0.825

Table 3: Number of annotators and agreement scores for 8 languages in MEE for
Entity Mention Detection (EMD), Event Detection (ED) and Event Argument
Extraction (EAE).

###  2.2 Annotation Process

To annotate the sampled article segments, we employ the crowd-sourcing
platform upwork.com that allows us to hire freelancers across the globe with
different expertise. For each language in our dataset, we choose native
speakers as annotator candidates. In addition, we require them to be fluent in
English, have experience in related tasks (i.e., data annotation for
information extraction), and have approval rate higher than 95% (i.e.,
provided in their profiles). The candidates are first provided with annotation
guidelines and interfaces in English. Afterward, they are invited to an
annotation test for entity mentions, event triggers, and arguments. Those
candidates who correctly annotate all test cases are then officially hired to
work on our annotation jobs. Table 3 shows the numbers of annotators who are
hired to annotate data for each language in our dataset. Next, before the
actual annotation process, the English annotation guideline and examples are
translated to each target language by the hired annotators. Any language-
specific confusions and rules for annotation is discussed and included in the
translation to create a common understanding. Finally, our language experts
will review the annotation guideline in each language to avoid conflicts
across languages to be used for actual annotation.

Our annotation process is done in three separate steps to annotate data for
three EE tasks with entity mentions, event triggers, and event arguments in
this order. In particular, the annotation for a later task will be performed
over the text segments that have been annotated and finalized for previous
tasks (e.g., event arguments will be annotated over segments that are already
provided with entity mentions and event triggers). As such, for each task, 20%
of text segments for each language will be co-annotated by the annotators to
measure agreement score. The remaining 80% of text segments will be
distributed and annotated separately by the annotators for each language.
Based on the Krippendorff’s alpha Krippendorff (2011) with MASI distance
metric Passonneau (2006), we report the inter-annotator agreements (IAA) for
each task and language in Table 3, showing high agreement scores and quality
of our MEE dataset. Note that after independent annotation for each EE task,
the annotators also share their annotations and communicate with each other to
resolve any conflicts and finalize our data.

ID | Event | Arguments
---|---|---
1 | Life_Be-Born | Person, Time, Place
2 | Life_Marry | Person, Time, Place
3 | Life_Divorce | Person, Time, Place
4 | Life_Injure | Agent, Victim, Instrument, Time, Place
5 | Life_Die | Agent, Victim, Instrument, Time, Place
6 | Movement_Transport | Agent, Artifact, Vehicle, Price, Origin, Destination, Time
7 | Transaction_Transfer-Ownership | Buyer, Seller, Beneficiary, Price, Artifact, Time, Place
8 | Transaction_Transfer-Money | Giver, Recipient, Beneficiary, Money, Time, Place
9 | Business_Start-Organization | Agent, Organization, Time, Place
10 | Conflict_Attack | Attacker, Target, Instrument, Time, Place
11 | Conflict_Attack | Entity, Time, Place
12 | Contact_Meet | Entity, Time, Place
13 | Contact_Phone-Write | Entity, Time
14 | Personnel_Start-Position | Person, Entity, Position, Time, Place
15 | Personnel_End-Position | Person, Entity, Position, Time, Place
16 | Justice_Arrest-Jail | Person, Agent, Crime, Time, Place
Table 4: Event types and argument roles for each type in MEE. The types and
roles are inherited from the event extraction annotation guideline in the ACE
2005 dataset Walker et al. (2006).

###  2.3 Data Analysis

Table 2 shows the main statistics of MEE for each language. As such, comparing
to the popular multilingual ACE 2005 dataset Walker et al. (2006) for EE, our
MEE dataset provides more languages (i.e., 3 vs. 8) and much more event
mentions (i.e., 11K vs. 50K). For other multilingual datasets for EE, i.e.,
TAC KBP (with three languages and 6.5K event mentions) Mitamura et al. (2016,
2017) and TempEval-2 (with 6 languages and 27K event mentions) Verhagen et al.
(2010), they do not annotate entity mentions and event arguments. In contrast,
our MEE dataset fully annotates texts for three EE tasks (i.e., EMD, ED, and
EAE) and also with more languages and event mentions. This clearly
demonstrates the advantages of our dataset over existing multilingual datasets
for EE.

Figure 1: Distributions of event types in each language.

Figure 2: Distributions of entity types in each language.

Figure 3: Distributions of most argument roles for each language.

In addition, from the table, we find that the languages in our dataset
exhibits diverse densities for entity mentions, event triggers, and arguments
in texts. In particular, while the average number of entities in a text
segment in Portuguese is 16.9, this number is only 8.3 in Korean. For event
density, in Polish, there are 2.4 event mentions per article segment on
average while the average number in Korean is only 0.75. Similarly for event
arguments, the average number of arguments per event is 6.1 in Portuguese and
only 0.75 in English. Further, Table 2 highlights the divergences between
languages regarding challenging entity and event types. Specifically, we
employ the disagreement rates (i.e., number of disagreements divided by
frequency of mentions) between annotators for each entity and event types.
Those types that have highest disagreement rates are selected as challenging
entity or event types. Finally, Figures 2, 1, and 3 present the distributions
of entity types, event types, and argument roles (respectively) for each
language in our dataset, which further demonstrate the differences between
languages in MEE. In all, such differences over various dimensions will cause
significant challenges for EE models to adapt to new languages (e.g., for
cross-lingual transfer learning), thus presenting ample opportunities for
multilingual EE research with our dataset.

Language | Pipeline | OneIE | FourIE
---|---|---|---
| Entity | Event | Argument | Entity | Event | Argument | Entity | Event | Argument
English | 70.32 | 70.58 | 61.14 | 62.18 | 70.09 | 62.94 | 69.72 | 72.19 | 65.89
Spanish | 70.39 | 66.19 | 60.16 | 70.27 | 65.00 | 60.31 | 71.89 | 67.49 | 62.19
Portuguese | 75.13 | 71.33 | 69.15 | 73.19 | 70.13 | 71.27 | 74.98 | 72.99 | 70.17
Polish | 69.27 | 59.12 | 60.09 | 60.09 | 59.44 | 60.14 | 68.23 | 60.98 | 61.32
Turkish | 71.88 | 66.09 | 56.19 | 71.98 | 61.27 | 58.72 | 72.33 | 65.13 | 59.80
Hindi | 66.22 | 57.77 | 57.78 | 61.72 | 58.18 | 59.44 | 65.23 | 59.88 | 60.82
Japanese | 68.19 | 67.89 | 68.19 | 71.40 | 65.01 | 63.17 | 70.88 | 66.88 | 70.19
Korean | 57.17 | 61.26 | 67.87 | 55.87 | 61.10 | 65.41 | 58.18 | 60.09 | 69.23
Avg. | 68.57 | 65.03 | 62.57 | 65.84 | 63.78 | 62.68 | 68.93 | 65.70 | 64.95

Table 5: Performance (F1 scores) of models in the monolingual setting using
mBERT on MEE.

Language | Pipeline | OneIE | FourIE
---|---|---|---
| Entity | Event | Argument | Entity | Event | Argument | Entity | Event | Argument
English | 70.22 | 71.28 | 66.34 | 70.39 | 70.29 | 68.68 | 71.19 | 73.14 | 68.23
Spanish | 70.33 | 64.32 | 61.12 | 70.18 | 62.46 | 62.23 | 72.87 | 65.90 | 63.11
Portuguese | 70.39 | 71.88 | 71.75 | 72.16 | 69.43 | 70.33 | 73.98 | 70.43 | 72.23
Polish | 69.14 | 60.45 | 61.23 | 72.22 | 63.77 | 60.15 | 70.25 | 62.87 | 62.84
Turkish | 76.13 | 67.18 | 55.78 | 74.45 | 65.31 | 57.40 | 75.19 | 67.29 | 58.23
Hindi | 65.14 | 59.34 | 58.22 | 61.72 | 58.18 | 59.44 | 66.69 | 61.99 | 62.19
Japanese | 71.34 | 67.77 | 69.19 | 68.20 | 62.89 | 70.90 | 72.82 | 65.27 | 73.55
Korean | 59.13 | 62.34 | 69.70 | 59.99 | 60.55 | 66.89 | 60.24 | 61.18 | 70.09
Avg. | 68.98 | 65.57 | 64.17 | 68.84 | 64.36 | 64.57 | 70.40 | 66.01 | 66.31

Table 6: Performance (F1 scores) of models in the monolingual setting using
XLM-RoBERTa on MEE.

##  3 Experiments

This section evaluates the state-of-the-art models for Event Extraction to
reveal challenges in our new dataset MEE. To this end, the annotated article
segments for each language in MEE are randomly split into
training/development/test portions with the ratios of 80/10/10. Here, to
prevent any information leakage, we ensure that different segments of an
article (if any) are only assigned to one portion of the data split for each
language. We examine EE models in two different settings: (1) monolingual
learning where training and test data of models comes from the same language;
(2) cross-lingual transfer learning where models are trained on training data
of one language (i.e., the source language), but evaluated directly on test
data of the other languages (i.e., the target languages).

Models: We evaluate two typical approaches for EE models with pipeline and
joint inference in this work. First, for the pipeline approach, a model is
trained separately for each of the three tasks in EE, i.e., entity mention
detection (EMD), event detection (ED), and event argument extraction (EAE).
Here, the EMD and ED tasks are modeled as sequence labeling problems, aiming
to predict BIO tag sequences for each input sentence to capture spans and
types of entity and event mentions. As such, motivated by previous work Wang
et al. (2020), our EMD and ED models leverage a pre-trained transformer-based
language model to encode the input text. The representation for each token in
input text (obtained via average of hidden vectors of word-pieces in the last
transformer layer) is then sent into a feed-forward network to compute a tag
distribution for the token for training and decoding. For EAE, the task is
formulated as a text classification problem in which the input consists of an
input text and two word indices for the positions of an event trigger and an
entity mention of interest. The goal is to predict the argument role that the
entity mention plays for the event. To this end, we also use a pre-trained
language model to obtain representations for the tokens in input text. Next,
the representations for the event trigger and entity mention words are
concatenated and sent to a feed-forward network to predict argument role. Note
that the EAE model employs golden entity mentions and event triggers during
the training process while the outputs from the EMD and ED models are fed into
the EAE model in the test time.

Second, for the joint inference approach, EE models simultaneously predicts
entity mentions, event triggers, and arguments in end-to-end fashion to avoid
error propagation and leverage inter-dependencies between tasks. To this end,
we evaluate two state-of-the-art (SOTA) joint EE models, OneIE Lin et al.
(2020) and FourIE Nguyen et al. (2021a), in this work due to their language-
agnostic nature. Both OneIE and FourIE utilize pre-trained language models to
represent input texts and capture cross-task dependencies for joint inference.
Note that these models are original designed to include the relation
extraction task between entities. To adapt them to EE, we obtain their
implementations from the original papers and remove the relation extraction
components. Finally, for performance measure, we report the performance (F1
scores) of EE models over three tasks EMD (Entity), ED (Event), and EAE
(Argument) using the same correctness criteria as in prior work Lin et al.
(2020) (i.e., requiring correct prediction for both offsets and types of
entity mentions, event triggers, and argument roles).

Hyper-parameters: To facilitate evaluation with multiple languages, we
leverage the multilingual pre-trained language models (PLMs) mBERT Devlin et
al. (2019) and XLM-RoBERTa Conneau et al. (2020) (base versions) to encode
texts for EE models. For the pipeline approach, we fine-tune the hyper-
parameters for the EMD, ED, and EAE models over development data for English
and apply the selected values for all experiments for consistency. In
particular, our hyper-parameters for the pipeline model include: 2 hidden
layers with 250 hidden units in each layer for the feed-forward networks, 8
for mini-batch size, and 1$e$-2 for learning rate with the Adam optimizer. For
the joint IE models, we utilize the same hyper-parameters suggested in the
original papers, i.e., OneIE Lin et al. (2020) and FourIE Nguyen et al.
(2021a).

Language | XLM-RoBERTa | mBERT
---|---|---
| Entity | Event | Argument | Entity | Event | Argument
English | 69.72 | 72.19 | 65.89 | 71.19 | 73.14 | 68.23
Spanish | 61.96 | 59.70 | 52.23 | 60.72 | 60.06 | 50.77
Portuguese | 59.98 | 54.80 | 52.23 | 56.17 | 52.98 | 50.28
Polish | 52.89 | 51.78 | 52.44 | 53.44 | 50.29 | 53.56
Turkish | 60.13 | 53.32 | 52.19 | 59.19 | 52.76 | 53.10
Hindi | 56.32 | 59.76 | 57.17 | 55.39 | 58.44 | 55.65
Japanese | 41.13 | 44.95 | 40.13 | 42.43 | 43.76 | 41.18
Korean | 45.78 | 42.99 | 43.04 | 44.78 | 40.22 | 41.14

Table 7: Cross-lingual performance (F1 scores) of FourIE when it is trained on
English training data and evaluated on test data of other languages in MEE.

Results: The results for monolingual experiments over different languages in
MEE are presented in Tables 5 and 6 (i.e., with mBERT and XLM-RoBERTa encoders
respectively). There are several observations from the tables. First, the
models’ performance on individual languages and on average for all three tasks
EMD, ED, and EAE is still far from being perfect (i.e., all average
performance is less than 69%), thus indicating considerable challenges in our
multilingual EE dataset for future research. In addition, comparing the
current state-of-the-art joint IE model (i.e., FourIE) with the pipeline
method, we find that FourIE is better than the pipeline model on average,
especially for the EAE task with significant performance gap. As such, we
attribute this to the ability of joint models to mitigate error propagation to
EAE from EMD and ED to boost the performance. Due to its best average
performance, FourIE will be leveraged in our next experiments. Finally, we
find that XLM-RoBERTa generally has better performance than mBERT (i.e., on
average) for EE models. Future research can thus focus on XLM-RoBERTa to
develop better EE models for multilingual settings.

Cross-lingual Evaluation: To further understand the cross-lingual
generalization challenges in MEE, Table 7 reports the performance of FourIE in
the cross-lingual transfer learning settings where the model is trained on
English training data (source language) and tested on test data of the other
languages in MEE. As can be seen, compared to performance on English test set,
FourIE suffers from significant performance drops over different tasks and
multilingual encoders when it is evaluated on other languages. It thus
demonstrates inherent challenges of cross-lingual generalization for complete
EE models that can be further studied with MEE. In addition, the performance
loss due to cross-lingual testing varies across different target languages
(e.g., 10.88% loss for Spanish vs. 33.42% loss for Japanese in EAE task).
These variations can be attributed to different levels of divergence between
languages (e.g., sentence structures and morphology) that hinder cross-lingual
knowledge transfer for EE.

Language | Entity | Event | Argument
---|---|---|---
English Devlin et al. (2019) | 70.21 | 73.18 | 66.19
Spanish Cañete et al. (2020) | 67.29 | 65.14 | 60.13
Portuguese Souza et al. (2020) | 70.21 | 68.88 | 67.13
Polish Kłeczek (2021) | 65.78 | 61.23 | 59.14
Turkish MDZ (2021) | 67.34 | 64.19 | 58.72

Table 8: Test data performance (F1) of FourIE in monolingual learning using
available language-specific BERT models on MEE. The citations indicate the
sources of the language-specific models.

Language | Entity | Event | Argument
---|---|---|---
English Liu et al. (2019) | 70.32 | 72.28 | 69.19
Spanish MMG (2021) | 70.23 | 61.34 | 60.28
Polish CLARIN-PL (2021) | 68.12 | 60.89 | 60.34
Hindi Parmar (2021) | 64.91 | 59.09 | 60.38
Japanese Wongso (2021) | 69.72 | 60.45 | 71.45

Table 9: Test data performance (F1) of FourIE in monolingual learning using
available language-specific RoBERTa models on MEE. The citations indicate the
sources of the language-specific models.

Language-Specific Encoders: To study the effectiveness of pre-trained language
models as text encoders for EE models, we compare the performance of FourIE
when the multilingual encoders mBERT or XLM-RoBERTa are replaced with
comparable language-specific encoders (i.e., BERT-based models for mBERT and
RoBERTa-based models for XLM-RoBERTa). Using publicly available pre-trained
language models for our languages in MEE, Tables 8 and 9 show the monolingual
performance over test data of the languages for BERT-based and RoBERTa-based
models (respectively). Comparing corresponding performance in Tables 5, 6, 8
and 9, it is clear that language-specific language models all under-perform
their multilingual counterparts over different EE tasks and languages, thus
suggesting the benefits of multilingual data to train language model encoders
to boost EE performance over different languages.

Language | Trained on English | Trained on Polish
---|---|---
| Entity | Event | Argument | Entity | Event | Argument
Portuguese | 53.22 | 50.79 | 40.21 | 54.17 | 51.70 | 42.33
Spanish | 50.76 | 43.72 | 41.16 | 51.72 | 45.22 | 45.81
Turkish | 54.44 | 50.12 | 55.71 | 53.99 | 50.78 | 56.15
Hindi | 51.44 | 52.78 | 45.27 | 52.21 | 53.00 | 47.24
Japanese | 36.16 | 41.23 | 38.13 | 37.17 | 40.13 | 39.28
Korean | 43.72 | 40.08 | 37.29 | 42.08 | 39.78 | 37.10

Table 10: Cross-lingual performance (F1) of FourIE with XLM-RoBERTa encoder
when it is trained on English or Polish training data, and tested on test data
of the other languages in MEE. We use 3,500 random annotated segments from the
training sets of English and Polish to train the model.

Source Language Impact: Finally, to study the impact of the source language
for cross-lingual transfer learning for EE, we compare the performance of
FourIE when either English or another comparable language is used as the
source language to provide training data to train the model. In particular, we
choose Polish as a comparable language for English as it has the same sentence
structure (i.e., both languages have Subject-Verb-Object order) and entails
similar density and type distributions for entity/event mentions as English.
Table 10 shows the performance of the models when they are tested over test
data of the other 6 languages in MEE. Here, to make it comparable, we use the
same number of annotated segments (i.e., 3,500) sampled from training data of
English and Polish to train the FourIE model. Interestingly, we find that
Polish can lead to better performance for FourIE than English over a majority
of task and target language pairs (i.e., over 4 languages for EMD and ED, and
5 languages for EAE). A possible explanation for this issue comes from richer
event patterns that Polish might introduce to produce allow better cross-
lingual generalization for EE than those for English. As such, this superior
performance of Polish challenges the common practice of using English as the
source language in cross-lingual transfer learning studies for EE and NLP.
Future research can explore this direction to better understand the
differences between languages to best select a source language to optimize
performance over a target language for EE.

##  4 Related Works

Due to its importance, various datasets have been recently developed for EE in
different domains, including CySecED Man et al. (2020) (for cybersecurity
domain), LitBank (for literacy) Sims et al. (2019), MAVEN Wang et al. (2020),
RAMS Ebner et al. (2020), and WikiEvents Li et al. (2021) (for Wikipedia
texts). However, these datasets are only annotated for English texts. There
exist several multilingual datasets for EE, ACE Walker et al. (2006), TAC KBP
Mitamura et al. (2016, 2017), and TempEval-2 Verhagen et al. (2010); however,
such datasets only provide annotation for a handful of popular languages with
limited number of event mentions and might not fully support all EE tasks
(e.g., missing EAE in TAC KBP and TempEval-2).

Regarding model development, existing EE methods can be categorized into
feature-based Ahn (2006); Ji and Grishman (2008); Liao and Grishman (2010);
Hong et al. (2011); Li et al. (2013); Yang and Mitchell (2016) or deep
learning Chen et al. (2015); Nguyen et al. (2016); Sha et al. (2018); Wang et
al. (2019); Lin et al. (2020); Veyseh et al. (2021a, b); Liu et al. (2022);
Veyseh and Nguyen (2022); Nguyen et al. (2022) methods. While most prior EE
methods have been designed for one popular language, there have been growing
interests in multilingual and cross-lingual learning for EE in recent work,
featuring multilingual PLMs (i.e., mBERT and XLMR) as the key component for
representation learning Chen and Ji (2009); M’hamdi et al. (2019); Ahmad et
al. (2021); Nguyen et al. (2021c); Huang et al. (2022); Guzman-Nateras et al.
(2022). However, as such works only rely on existing multilingual EE datasets,
their evaluation is limited to a few popular languages and fails to evaluate
the generalization over many other languages.

##  5 Conclusion

We present a novel multilingual EE dataset, i.e., MEE, that covers 8
typologically different languages with more than 50K event mentions to support
training of large deep learning models. MEE provides complete annotation for
three EE sub-tasks, i.e., entity mention detection, event detection, and event
argument extraction. To study the challenges in MEE, we conduct extensive
analysis and experiments with different EE methods in the monolingual and
cross-lingual learning settings. Our results demonstrate various challenges
for EE in the multilingual settings that can be further pursued with MEE. In
the future, we will extend our dataset to include more languages and tasks for
IE.

## Limitations

In this work we present a novel large-scale multilingual dataset for Event
Extraction. As it is shown in the experiments, our dataset introduces many
challenges that can inform future research on multilingual Event Extraction.
However, there are still some limitations in the current work that can be
improved in future research. First, cross-lingual transfer learning for EE is
a challenging task that requires specifically designed models and methods.
However, in this work, we have mainly focused on existing state-of-the-art EE
models that are originally developed for the monolingual settings. As such,
future work can study cross-lingual transfer models that are specifically
designed to address the gaps between languages to better understand the
challenges in our multilingual EE dataset. Second, in addition to data
scarcity for multilingual EE, another challenge for this problem is the lack
of resources for text encoding and processing in multiple languages. In
particular, pre-trained language models and text processing tools might not be
available for some languages (e.g., low-resource languages) that hinder
dataset creation and model development efforts. As such, our work has not
explored datasets and methods for low-resource languages for EE. In addition,
as shown in our experiments, a majority of existing language-specific text
encoders under-perform their multilingual counterparts for EE models. However,
our work has not studied methods to improve such language-specific language
models for EE. Finally, although our experiments empirically challenge English
as the main source language for cross-lingual learning, we have not explored
why other languages might be better options for the source language in this
setting. Future research can perform more comprehensive analysis to shed light
on this direction.

## Acknowledgement

This research has been supported by the Army Research Office (ARO) grant
W911NF-21-1-0112 and the NSF grant CNS-1747798 to the IUCRC Center for Big
Learning. This research is also based upon work supported by the Office of the
Director of National Intelligence (ODNI), Intelligence Advanced Research
Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the
Better Extraction from Text Towards Enhanced Retrieval (BETTER) Program. The
views and conclusions contained herein are those of the authors and should not
be interpreted as necessarily representing the official policies, either
expressed or implied, of ARO, ODNI, IARPA, the Department of Defense, or the
U.S. Government. The U.S. Government is authorized to reproduce and distribute
reprints for governmental purposes notwithstanding any copyright annotation
therein. This document does not contain technology or technical data
controlled under either the U.S. International Traffic in Arms Regulations or
the U.S. Export Administration Regulations.

## References

  * Ahmad et al. (2021) Wasi Uddin Ahmad, Nanyun Peng, and Kai-Wei Chang. 2021.  Gate: Graph attention transformer encoder for cross-lingual relation and event extraction.  In _Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI)_.
  * Ahn (2006) David Ahn. 2006.  The stages of event extraction.  In _Proceedings of the Workshop on Annotating and Reasoning about Time and Events_.
  * Attardi (2015) Giusepppe Attardi. 2015.  Wikiextractor.  https://github.com/attardi/wikiextractor.
  * Cañete et al. (2020) José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. 2020.  Spanish pre-trained bert model and evaluation data.  In _PML4DC at International Conference on Learning Representations (ICLR) 2020_.
  * Chen et al. (2015) Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015.  Event extraction via dynamic multi-pooling convolutional neural networks.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Chen and Ji (2009) Zheng Chen and Heng Ji. 2009.  Can one language bootstrap the other: A case study on event extraction.  In _Proceedings of the NAACL-HLT 2009 Workshop on Semi-supervised Learning for Natural Language Processing_.
  * CLARIN-PL (2021) CLARIN-PL CLARIN-PL. 2021.  Polish roberta.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ebner et al. (2020) Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020\.  Multi-sentence argument linking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Guzman-Nateras et al. (2022) Luis Guzman-Nateras, Minh Van Nguyen, and Thien Nguyen. 2022.  Cross-lingual event detection via optimized adversarial training.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5588–5599, Seattle, United States. Association for Computational Linguistics.
  * Hong et al. (2011) Yu Hong, Jianfeng Zhang, Bin Ma, Jianmin Yao, Guodong Zhou, and Qiaoming Zhu. 2011\.  Using cross-entity inference to improve event extraction.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Huang et al. (2022) Kuan-Hao Huang, I-Hung Hsu, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. 2022\.  Multilingual generative language models for zero-shot cross-lingual event argument extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Ji and Grishman (2008) Heng Ji and Ralph Grishman. 2008.  Refining event extraction through cross-document inference.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Krippendorff (2011) Klaus Krippendorff. 2011.  Computing krippendorff’s alpha-reliability.
  * Kłeczek (2021) Darek Kłeczek. 2021.  Polish bert.
  * Lai et al. (2020) Viet Dac Lai, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020.  Event detection: Gate diversity and syntactic importance scores for graph convolution neural networks.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5405–5411, Online. Association for Computational Linguistics.
  * Li et al. (2013) Qi Li, Heng Ji, and Liang Huang. 2013.  Joint event extraction via structured prediction with global features.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 73–82.
  * Li et al. (2021) Sha Li, Heng Ji, and Jiawei Han. 2021.  Document-level event argument extraction by conditional generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , Online. Association for Computational Linguistics.
  * Liao and Grishman (2010) Shasha Liao and Ralph Grishman. 2010.  Filtered ranking for bootstrapping in event extraction.  In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_.
  * Lin et al. (2020) Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020.  A joint neural model for information extraction with global features.  In _Proceedings of The 58th Annual Meeting of the Association for Computational Linguistics_.
  * Liu et al. (2022) Xiao Liu, Heyan Huang, Ge Shi, and Bo Wang. 2022.  Dynamic prefix-tuning for generative template-based event extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5216–5228, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Man et al. (2020) Duc Trong Hieu Man, Duc Trong Le, Amir Pouran Ben Veyseh, Thuat Nguyen, and Thien Huu Nguyen. 2020.  Introducing a new dataset for event detection in cybersecurity texts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * MDZ (2021) Digital Library team MDZ. 2021.  Turkish bert.
  * M’hamdi et al. (2019) Meryem M’hamdi, Marjorie Freedman, and Jonathan May. 2019.  Contextualized cross-lingual event trigger extraction with minimal resources.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_.
  * Mitamura et al. (2016) Teruko Mitamura, Zhengzhong Liu, and Eduard H. Hovy. 2016.  Overview of TAC-KBP 2016 event nugget track.  In _Proceedings of the Text Analysis Conference (TAC)_.
  * Mitamura et al. (2017) Teruko Mitamura, Zhengzhong Liu, and Eduard H. Hovy. 2017.  Events detection, coreference and sequencing: What’s next? overview of the TAC KBP 2017 event track.  In _Proceedings of the Text Analysis Conference (TAC)_.
  * MMG (2021) MMG MMG. 2021.  Spanish roberta.
  * Nguyen et al. (2021a) Minh Van Nguyen, Viet Dac Lai, and Thien Huu Nguyen. 2021a.  Cross-task instance representation interactions and label dependencies for joint information extraction with graph convolutional networks.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_.
  * Nguyen et al. (2021b) Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. 2021b.  Trankit: A light-weight transformer-based toolkit for multilingual natural language processing.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_.
  * Nguyen et al. (2022) Minh Van Nguyen, Bonan Min, Franck Dernoncourt, and Thien Nguyen. 2022.  Joint extraction of entities, relations, and events via modeling inter-instance and inter-label dependencies.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4363–4374, Seattle, United States. Association for Computational Linguistics.
  * Nguyen et al. (2021c) Minh Van Nguyen, Tuan Ngo Nguyen, Bonan Min, and Thien Huu Nguyen. 2021c.  Crosslingual transfer learning for relation and event extraction via word category and class alignments.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5414–5426, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Nguyen et al. (2016) Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016.  Joint event extraction via recurrent neural networks.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 300–309.
  * Nguyen and Grishman (2015) Thien Huu Nguyen and Ralph Grishman. 2015.  Event detection and domain adaptation with convolutional neural networks.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_.
  * Parmar (2021) Suraj Parmar. 2021.  Hindi roberta.
  * Passonneau (2006) Rebecca Passonneau. 2006.  Measuring agreement on set-valued items (masi) for semantic and pragmatic annotation.  In _The International Conference on Language Resources and Evaluation (LREC)_.
  * Sha et al. (2018) Lei Sha, Feng Qian, Baobao Chang, and Zhifang Sui. 2018.  Jointly extracting event triggers and arguments by dependency-bridge rnn and tensor-based argument interaction.  In _Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI)_.
  * Sims et al. (2019) Matthew Sims, Jong Ho Park, and David Bamman. 2019.  Literary event detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3623–3634, Florence, Italy. Association for Computational Linguistics.
  * Souza et al. (2020) Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020.  BERTimbau: pretrained BERT models for Brazilian Portuguese.  In _9th Brazilian Conference on Intelligent Systems, BRACIS, Rio Grande do Sul, Brazil, October 20-23_.
  * Verhagen et al. (2010) Marc Verhagen, Roser Saurí, Tommaso Caselli, and James Pustejovsky. 2010.  SemEval-2010 task 13: TempEval-2.  In _Proceedings of the 5th International Workshop on Semantic Evaluation_.
  * Veyseh et al. (2021a) Amir Pouran Ben Veyseh, Viet Lai, Franck Dernoncourt, and Thien Huu Nguyen. 2021a.  Unleash GPT-2 power for event detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6271–6282, Online. Association for Computational Linguistics.
  * Veyseh et al. (2021b) Amir Pouran Ben Veyseh, Minh Van Nguyen, Nghia Ngo Trung, Bonan Min, and Thien Huu Nguyen. 2021b.  Modeling document-level context for event detection via important context selection.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Veyseh and Nguyen (2022) Amir Pouran Ben Veyseh and Thien Nguyen. 2022.  Word-label alignment for event detection: A new perspective via optimal transport.  In _Proceedings of the 11th Joint Conference on Lexical and Computational Semantics_ , pages 132–138, Seattle, Washington. Association for Computational Linguistics.
  * Veyseh et al. (2020) Amir Pouran Ben Veyseh, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020.  Graph transformer networks with syntactic and semantic structures for event argument extraction.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3651–3661, Online. Association for Computational Linguistics.
  * Walker et al. (2006) Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006.  Ace 2005 multilingual training corpus.  In _Technical report, Linguistic Data Consortium_.
  * Wang et al. (2019) Xiaozhi Wang, Xu Han, Zhiyuan Liu, Maosong Sun, and Peng Li. 2019.  Adversarial training for weakly supervised event detection.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 998–1008, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Wang et al. (2020) Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020.  Maven: A massive general domain event detection dataset.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Wongso (2021) Wilson Wongso. 2021.  Japanese roberta.
  * Yang and Mitchell (2016) Bishan Yang and Tom M. Mitchell. 2016.  Joint extraction of events and entities within a document context.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_.


# The MeLa BitChute Dataset

Milo Z. Trujillo,1 Maurício Gruppi,2 Cody Buntain,3 and Benjamin D. Horne4

###### Abstract

In this paper we present a near-complete dataset of over 3M videos from 61K
channels over 2.5 years (June 2019 to December 2021) from the social video
hosting platform BitChute, a commonly used alternative to YouTube.
Additionally, we include a variety of video-level metadata, including
comments, channel descriptions, and views for each video. The MeLa-BitChute
dataset can be found at:
https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/KRD1VS.

##  1 Introduction

Wilson and Starbird (2021) define “alt-tech” spaces as alternative, non-
mainstream platforms that exist largely in reaction to perceived risks of
censorship in mainstream spaces. These alt-tech platforms have received
significant attention from researchers, practitioners, and even policy makers,
due to their role in producing, spreading, and conserving anti-social content.
This anti-social content ranges from political disinformation to health-
related conspiracy theories to violent hate speech.

To this end, researchers have striven to study alt-tech platforms and to build
large datasets around those platforms; e.g., Gab (Zannettou et al. 2018),
Gettr (Paudel et al. 2021), Parler (Aliapoulios et al. 2021), Dissenter (Rye,
Blackburn, and Beverly 2020), and Telegram (Júnior et al. 2021). However, one
platform in this ecosystem that has, for the most part, lacked study is
BitChute, an alternative to YouTube. Studies on BitChute are rare because data
collected from the platform is rare, as BitChute does not have a publicly
available API like other social media platforms. This limitation makes data
collection a significant hurdle for researchers.

Despite this difficulty, the platform is deserving of study. Just as other
major alt-tech platforms, BitChute plays a critical role in harboring anti-
social content and communities (Trujillo et al. 2020). Most famously, BitChute
was a safe haven for the viral, COVID-19 conspiracy theory film, _Plandemic_ ,
which was quickly removed from Facebook, YouTube, and Twitter (Kearney,
Chiang, and Massey 2020; Buntain et al. 2021a). As shown in Rogers (2020),
these alternative spaces do not exist in isolation; rather, users of and
audiences in alt-tech spaces operate across several of these platforms
simultaneously. Hence, these spaces need to be studied holistically to
understand how extremists and far-right individuals leverage the many
affordances available across these alt-tech platforms. This need is apparent,
as Doesburg (2021) demonstrates that BitChute is already one of the most
popular alt-tech domains shared in Telegram, and Rogers (2020) shows that
BitChute is one of the main destinations for internet celebrities who have
been deplatformed on the mainstream platforms. As the broader information
ecosystem cannot be fully understood without YouTube, so too must one
understand BitChute’s role to understand the alt-tech ecosystem.

In this paper, we present the MeLa-BitChute dataset to help fill this gap. The
dataset contains data from 3,036,190 videos, 61,229 channels, and 11,434,571
comments between June 28th, 2019 and December 31st, 2021. This dataset
provides timestamped activities and estimates on views for the majority of
channels and videos on the platform, allowing researchers to align BitChute
videos with behavior on other platforms. Therefore, this dataset can
facilitate both studies of BitChute in isolation and studies of BitChute’s
role in the larger ecosystem.

In the remainder of this paper, we describe the data collection methodology
behind MeLa-BitChute, publicly-available data formats and documentation,
evaluations of the dataset’s completeness, and an extensive discussion on use
cases. While we do discuss some characteristics of BitChute in this paper, we
recommend reading Trujillo et al. (2020) for a more complete examination of
the platform’s history and early content.

Figure 1: The core data collection is done in two parallel parts: 1. Every 5
minutes we collect newly uploaded videos from the BitChute video stack. 2.
Every 24, hours we loop through all videos that are at least one week old, go
to the video page, and scrape views- and comments data. The third component,
not depicted here, is the collection of channel description, which is scraped
once for every new channel in the database. Figure 2: Database schema for
SQLite3 format. The video URLs are used as the primary key for the videos
table, while a composite key of the commenter user ID and the individual
comment ID are used as the primary key of the comments table, and a composite
key of the channel URL and the timestamp the channel description was scraped
are used as the primary key of the channels table.

##  2 Data Collection and Infrastructure

BitChute does not have a published API. Therefore, we build a custom
collection engine that uses web-scraping and parsing at regular intervals.
Broadly, using this infrastructure, we collect data on four entities within
BitChute: videos, video views, comments, and channels. Figure 1 shows the
high-level flow chart of this collection infrastructure.

###  2.1 Videos

The core part of the data collection is video metadata. To collect video data,
we utilize a web-page on BitChute that displays newly uploaded videos in a
stack (see Figure 1 for an example of this page). This page is scraped every
five minutes, recording the URLs of videos we have not yet seen. For each
newly-uploaded video, we visit the video URL, parsing the web interface to
extract the video title, description, uploader, channel, video category,
sensitivity rating, and exact upload date. We record these results in a
PostgreSQL database. An example of the video interface that data is scraped
from can be found in Figure 3.

###  2.2 Comments and Views

Since we collect videos within 5 minutes of being uploaded to the platform, we
must wait to collect audience engagement data. To this end, we have a second,
concurrent process, which queries the database for a list of videos uploaded
at least one week ago that have not yet been re-examined. This script then
visits each video URL again, this time recording the number of views and
scraping all of the comments that the video has received. If the video is no
longer available, we record whether it has been retracted by the uploader or
removed by BitChute moderation.

When scraping the comments, we collect the comment author, comment text,
creation date, and scrape date.

###  2.3 Channels

Lastly, to collect description data on the channels, we have a third
concurrent process that queries the database for a list of channels that have
yet to have their description information collected. This script then visits
each channel ‘About’ page to scrape the HTML for an author-provided
description, if available.

###  2.4 Collecting Dynamically Loaded Data

While most of the metadata we are interested in is available through HTML
parsing, the view counts and comments are loaded dynamically. We initially
overcame this by automating a web browser with Selenium, so that BitChute’s
JavaScript could run without modification. Using this method, we could extract
the view count and comments from the browser’s DOM. However, automating a
browser scaled poorly, so we instead determined what HTTP requests the site
JavaScript was making, and automated those requests ourselves to retrieve
views and comments.

#### Collecting Dynamically Loaded Views

First, to retrieve views, we visit the video URL and record a CSRF
token111Cross-Site Request Forgery Token, a type of cookie typically used to
prevent attackers from tricking a browser into making malicious HTTP requests.
Then when visiting /{video-URL}/counts/, we present the CSRF token and collect
the loaded views data.

#### Collecting Dynamically Loaded Comments

Second, to retrieve comments, we must rely on BitChute’s comment
infrastructure, which changed throughout our collection timeline. Originally,
from 2017 to around September 2020, BitChute contracted the third-party blog
comment hosting service, Disqus, to provide commenting infrastructure for
their website. After Disqus terminated the contract, BitChute implemented
their own commenting software, “CommentFreely.” CommentFreely is open source,
and can be examined at https://github.com/BitChute/commentfreely.

When Disqus was being used, we derived the Disqus URL from the BitChute URL,
then visited the Disqus page, which included comments as both HTML and JSON
for easy parsing. Once BitChute switched to their own CommentFreely
infrastructure, we retrieved comments as follows: 1. Parse the CommentFreely
JavaScript embedded in the video page to find a unique “video token.” 2. Make
a POST request to https://commentfreely.bitchute.com/api/get˙comments/
containing the video token. 3. Parse the response JSON.

Figure 3: Example of metadata collected for each video. Note, the channel of
the video is “BANNED.VIDEO” while the profile is “Infowars.” The Infowars
profile has several channels, including an Infowars channel and “The Alex
Jones Show.”

##  3 Publicly Available Data Formats

In order to accommodate the largest audience possible, we provide two widely-
used data formats.

###  3.1 SQLite3 Database

The first format is an SQLite3 database with three tables: videos, comments,
and channels. The schema for this database can be found in Figure 2. While our
collection engine stores data in a PostgreSQL database, we convert it to an
SQLite3 database to allow researchers to use the data without any database
server setup.

The primary table in the database is the videos table, which includes the
video URL, title, date the video was posted, timestamp when video was scraped,
a description of the video if provided by the uploader, the profile of the
user who posted the video, the channel that posted the video (see Appendix
Table 1 for description of profile vs. channel), the user picked topical
category for the video, the user picked sensitivity for the video, the number
of times the video is viewed, and a timestamp of when the views were scraped.

The second table is the comments table, which contains the video URL that the
comment is under, the user ID of the commenter, the comment ID for the
individual comment, the parent comment ID if it is a nested comment, the
comment HTML, and the comment text.

The third table is the channels table, which contains the URL to the channel,
a the HTML of the channel description if the channel owner provides one, the
text of the channel description if the channel owner provides one, and a
timestamp of when the channel data was scraped.

A detailed description of each data column can be found in Appendix A.

###  3.2 CSV

The second format in which we provide the dataset is a set of Comma-Separated
Value (CSV) files. We provide three CSV files, one for each table in the
database: videos, comments, and channels. The columns in each CSV file are the
same as the columns in each corresponding SQLite3 database table.

###  3.3 FAIR Principles

We are careful to ensure that the MeLa-BitChute dataset follows FAIR
principles222https://www.force11.org/group/fairgroup/fairprinciples.

  * •

Findable - The dataset is persistently stored on Harvard Dataverse, is
documented, and described with rich metadata.

  * •

Accessible - The dataset is freely and publicly accessible through Harvard
Dataverse’s GUI, is stored in two widely-used formats, and comes with an
example Python script for data extraction and use.

  * •

Interoperable - The dataset can be parsed automatically using standard
languages (Python, SQL, R), and can be parsed by human annotators using the
CSV formats provided.

  * •

Re-usable - The data can be reused for many types of studies, given the
breadth of the collection. The data can be paired and augmented with other
social media datasets for rich studies of alt-tech, deplatforming,
disinformation spread and more. See Section 5 for an in-depth discussion of
these use cases. Given that the URLs are stored and rich metadata is well-
documented, provenance is maintained.

###  3.4 Ethical Considerations of Stored Data

The key ethical consideration with the collection and sharing of this dataset
is the privacy of the platform’s producers and consumers. While we considered
anonymizing the content producers (channels/profiles), we choose not to as
their identity is important to understand both activity within the platform
and studies across other platforms. Censoring the channel and video URLs would
reduce the dataset’s usefulness in the research community and would destroy
the provenance of the data. Furthermore, since the content producers on
BitChute post videos for public consumption, they should not have the
expectation of anonymity.

On the other hand, we do choose to anonymize the comment posters. We have
assigned each commenter a unique ID, by creating a salted hash of their
account information. This allows researchers to identify all comments made by
a particular author, without revealing the username of the author. We do this
because commenters have a greater expectation of privacy than content
publishers, and the right to be forgotten. Furthermore, the importance of
tracking influential content creators across multiple platforms does not apply
to commenters.

(a) Videos published per month

(b) Comments posted per month

(c) Views per month

Figure 4: In (a) we show the number of videos published each month. In (b) we
show the number of comments posted in each month. Note, the grey background
indicates dates where we were not able to collect comments. Please see
discussion in Section 4. In (c) we show the number of video views per month.
Note both the growth in videos and views collected per month (a and c) and the
growth in Google Trends interest in BitChute in Figure 5. Figure 5: Google
Trends interest across the dataset’s timespan, where the mean is shifted to 0
for comparison.

(a) Top 20 user-defined categories by number of videos

(b) Top 20 channels by number of videos

(c) Top 20 channels by number of views

(d) Top 20 channels by number of comments

Figure 6: In (a) we show the top 20 user-picked video categories by number of
videos. Note, just as described in (Trujillo et al. 2020), ‘Other’ was
originally the default category for all uploaded videos. However, in September
2020 BitChute changed the ‘Other’ category to ‘None’. We show the combined
number of videos across those two categories. In (b) we show the top 20
channels by number of videos, in (c) by number of views, and in (d) by number
of comments.

##  4 Evaluation of Data Completeness

Given the complexity of collecting this data, it is important to clearly
document where we are confident in the data completeness and where we are not.
To this end, below we discuss several known data outages and caveats.

#### Known collect server outages

During the collection of this data, we documented several collection outages
due to either BitChute itself or due to issues at our server location, which
changed several time during the timeline. Below is a list of documented
shutdowns of our data collection:

  * •

October 11th, 2019 - Our collection server’s IP address was blocked from
accessing BitChute.

  * •

October 23rd, 2019 and November 18th, 2019 - Our collection server lost
connectivity during the Pacific Gas & Electric preemptive power
shutdowns.333www.latimes.com/california/story/2019-11-18/another-power-outage-
pge-may-shut-down-grid-northern-california Our sever was located in California
at the time.

  * •

November 22nd, 2019 - BitChute itself was down. The cause is unknown.

  * •

November 3rd, 2020 - BitChute itself was down due to one of their service
providers cutting their account off.

  * •

May 22nd, 2021 to May 24th, 2021 - A power outage in the block of the
University of Tennessee Knoxville’s campus where our server was located. The
cause of the outage was a mouse entering switch gear at the iconic American
football stadium, Neyland Stadium.

  * •

June 12th, 2021 to June 14th, 2021 - Another power outage in the block of the
University of Tennessee Knoxville’s campus where our server was located.

In most of these cases, we were able to recover videos published during the
outages, but we cannot guarantee we recovered them all. Given how short these
outages were, we are confident that the number of videos collected over-time
is near-complete.

#### Known missing comment data

As discussed in Section 2.4, BitChute’s comment infrastructure changed
throughout our collection timeline. Since our collection method is highly
dependent on the structure of BitChute’s platform, these changes directly
impacted our ability to collect comments. Originally, BitChute used third-
party Disqus, then created their own called CommentFreely after Disqus
terminated its contract with BitChute. During this transition, multiple
structural changes by BitChute made it impossible to collect comments.
Furthermore, we were unable to recover comment data because the comments made
during the Disqus era were removed by BitChute and no longer exist. In Figure
4(b), dates where comment data is missing due to this are shown in grey. Based
on the average number of comments per video, we estimate that we missed at
maximum 8.8M comments. This estimate is likely an over-estimate given the
platform growth during this time-span. In total, 22.9% (697K) of the videos
collected were published during the time-span where comments could not be
collected. Therefore, we are confident in conclusions being drawn from the
temporal patterns of the videos throughout the collection timeframe but not in
conclusions drawn from the temporal patterns of comments during 2020.

#### Caveats on views and comments data

Again, due to BitChute’s multiple commenting systems, and each system having a
different comment ID format. Comment IDs should be assumed to be text, not
numeric, and may be NULL. Author IDs cannot link authors from before and after
the Disqus to CommentFreely change.

Unfortunately, due to the software changes on BitChute, some of the views and
comments are gathered later than one week after video publication. Since our
comment data includes the creation date of each comment, researchers can
choose to filter out comments more than a week newer than their corresponding
videos to obtain a consistent “one week later” view of the dataset.

##  5 Dataset Use Cases

As one of the main video-sharing platforms in the growing alt-tech ecosystem,
researchers are increasingly interested in BitChute and its users as lenses
for studying these alt-tech spaces. This interest is evident in Google Scholar
data, where the number of articles mentioning BitChute has doubled annually
over the past two years. Despite this interest, limited data resources exist
to support these studies. Given the limited availability of such data, the
MeLa-BitChute dataset can facilitate such studies in a multitude of ways,
which we outline below.

##### Content Moderation and Deplatforming

As social media spaces have become central sources of information despite the
prevalence of misinformation and efforts to manipulate audiences (Tucker et
al. 2018), the moderation tools and interventions used to ensure the safety of
online audiences have likewise become critical aspects of the information
ecosystem. “Deplatforming”, or removing/suppressing content or individuals
from an online platform, has emerged as a popular moderation intervention
across both the mainstream social media platforms (e.g., Twitter, Facebook,
and Reddit) (Masters of Media 2020; Van Dijck, de Winkel, and Schäfer 2021;
Rogers 2020) and infrastructure platforms (e.g., Amazon Web Services and
Google/Apple app stores) (Buckley and Schafer 2021). Assessing impact of these
interventions can no longer be done within the context of a single platform,
however, as the proliferation of alt-tech platforms has provided new spaces
for creators to circumvent moderation. Of particular concern is whether
deplatforming pushes individuals from mainstream spaces to these more extreme
spaces, thereby increasing exposure to toxic and extreme content in spaces
like BitChute (Buntain et al. 2021c).

Researchers have already begun investigations into the cross-platform effects
of deplatforming, especially between YouTube and BitChute: Buntain et al.
(2021a) examines YouTube’s de-recommendation strategy, finding that removing
recommendations to misinforming content appears to suppress its sharing on
Twitter and Reddit, but little is known about whether this change simply moves
that de-recommended content to alt-tech spaces like BitChute. Similarly, in
the aftermath of the “Great Deplatforming” around the January 6 attack on the
US Capitol (Bond 2022), Buntain et al. (2021c) shows interest in alt-tech
platforms like Gab and BitChute has increased. Rauchfleisch and Kaiser (2021),
on the other hand, show that, only about 20% of far-right channels on YouTube
that were deplatformed between 2018-2019 had a BitChute presence at the end of
2019. The ecosystem has evolved since these studies, however, and the MeLa-
BitChute dataset enables these and new studies of whether and how BitChute
creators and audiences respond to these deplatforming efforts. Such questions
include whether creators push new content to their BitChute channels or
whether BitChute audiences grow and videos are seen or shared more in the
aftermath of deplatforming – as suggested with the Plandemic video (Bellemare,
Nicholson, and Ho 2020; Buntain et al. 2021a).

As a simple example of the platform’s evolution since these studies, one can
compare popular channels in Figures 6(c) and 6(d) (across 2019-2021) to those
in 2019, as documented in Trujillo et al. (2020). We see some overlap in the
channels with high engagement from late-2019 (infowars, rongibson, x22report,
styxhexenhammer666, nextnewsnetwork), but new channels have emerged as highly
engaged in this dataset, such as drcharlieward, chembuster, and sixthsense.
Many of these new top channels are focused on health and vaccination
conspiracy theories, where as the top channels in late-2019 were mostly
focused on far-right politics.

##### Conspiracy and Political Misinformation

Like other alt-tech spaces, BitChute has a high proportion of political
content, much of which focuses on conspiracy theories and otherwise
politically extreme (Trujillo et al. 2020; David 2020). These alt-tech spaces
are especially popular among far-right audiences, who often use these
platforms to share and amplify right-wing conspiracies and misinformation
(Freelon, Marwick, and Kreiss 2020). For example, despite moderation action by
the mainstream platforms to suppress election-related misinformation in the
aftermath of the January 6 attack, such conspiracies have flourished in
BitChute (Heilweil 2021). The QAnon constellation of conspiracy theories is
similarly popular on BitChute (Trujillo et al. 2020), as early research is
already examining QAnon’s use of the platform for sharing its increasingly
popular messages (Forberg 2021; Hoseini et al. 2021). The MeLa-BitChute
dataset is valuable in this space, as much of the content contained therein
would be removed or suppressed on other platforms, and the alt-tech spaces
available are primarily text-oriented. Consequently, the content in BitChute
and captured by the MeLa-BitChute dataset represents a unique multi-modal
resource that can provide insight into current and emerging topics of
conspiracy and political misinformation.

##### Supply, Demand, and Health Misinformation

Following the release of the misinformation-laden _Plandemic_ film on social
media, Facebook, YouTube, and Twitter responded quickly to limit its spread
(Kearney, Chiang, and Massey 2020). The film contained many unfounded
conspiracy theories about the origin of the COVID-19 pandemic, leading these
mainstream platforms to ban or suppress it as it violated policies on
misinformation related to public health. Despite these interventions, the film
remained widely available on alt-tech spaces and on BitChute in particular
(Bellemare, Nicholson, and Ho 2020), and analysis in Buntain et al. (2021a)
suggests BitChute received more traffic in response, as interest in the film
drove viewers to platforms that were willing to host it. Viewed through the
supply-and-demand framework in Munger and Phillips (2019), as the mainstream
platforms limit COVID-19 misinformation in their spaces, the supply of this
content moves to these alt-tech spaces. If demand remains constant,
interventions by the mainstream platforms may then push audiences toward these
alternative spaces, where both misinformation and extreme rhetoric are more
common. Two questions then emerge: First, is health misinformation becoming
more popular in BitChute and related spaces as Facebook, YouTube, and others
increasingly suppress that content locally, and second, how are content
producers on BitChute, Gab, etc. responding to this influx of demand – e.g.,
are they producing more such content? The MeLa-BitChute dataset provides
insight into these questions, first by allowing researchers to evaluate trends
in engagement and viewership, and second through longitudinal data of content
creators and the videos they post to their channels over time.

##### Alternative Monetization

Related to the supply and demand questions above, creators have multiple ways
to monetize their supply of content. YouTube’s Partner Program, for example,
pays creators a portion of advertising revenue based on views (Kopf 2020).
When YouTube deplatforms or otherwise suppresses content, however, creators
are “demonetized” and lose out on this revenue stream. Despite these
interventions, creators have found ways to circumvent deplatforming by posting
their more violative content on spaces like BitChute and sharing trailers to
this content on mainstream platforms (Trujillo et al. 2020). To this end, an
ecology of alternative monetization schemes now exists, allowing creators to
monetize their content through other means, such as donation, cryptocurrency,
affiliate marketing, or merchandise (Chu et al. 2022). BitChute also supports
a variety of these monetization options, both through on-platform advertising
and integration with donation-based platforms (e.g., Patreon, PayPal, and
others). Warreth (2021) details how the far-right and extremist groups use
these alternatives, especially cryptocurrency as funding sources. Via the the
MeLa-BitChute dataset, researchers can study how BitChute’s alternative
content is monetized through these alternative means.

##### Hate and Online Extremism

A rich body of work has examined hateful and radicalizing content in the
mainstream platforms. Evidence shows such content on Facebook has contributed
to violence and radicalization; e.g., Jihadist groups have used the platform
to radicalize potential recruits (Thompson 2011), anti-refugee sentiment
predicts criminal acts targeting refugees in otherwise similar communities
(Müller and Schwarz 2021), and anti-Muslim sentiment has been used to
stimulate fear and violence against Muslim communities (Fink 2018). Alt-tech
spaces like Gab are known to have high proportions of hate speech (Zannettou
et al. 2018), and BitChute is no exception, with much of its content
containing hateful and extreme rhetoric, often antisemitic or racist in nature
(Trujillo et al. 2020; David 2020; Papadamou 2021). Hate speech need not be
confined to textual modalities either, as the Anti-Defamation League has shown
through its database of hateful symbols (Anti-Defamation League 2022), and the
data contained in the MeLa-BitChute dataset may yield data for studying
hateful imagery, as the platform’s core affordance is video sharing. Prior
work has also shown such propensity towards hate is both indicative and
predictive of violent acts (Abdalla, Ally, and Jabri-Markwell 2021). Coupling
these works with the MeLa-BitChute dataset can provide a lens through which
this hateful content can be studied and assessed for potential harm or as an
indicator for new violent attacks. Likewise, the multi-year timeframe covered
by the MeLa-BitChute dataset may allow researchers insight into radicalization
processes among BitChute content creators and their audiences. Research
enabled by this dataset those studying terrorism, who have expressed a desire
for more data-driven analyses of hateful and extreme online content, actors,
and audiences (Pelzer 2018).

##  6 Related Datasets

Many available social media datasets are topically focused – e.g., covering
disasters (Olteanu et al. 2014; Buntain et al. 2021b), COVID-19 (Qazi, Imran,
and Ofli 2020), and hate/harassment (Davidson et al. 2017) – with new datasets
regularly released from spaces like SemEval444https://semeval.github.io/ or
the annual Text Retrieval Conference (TREC). While these datasets are valuable
resources for understanding specific phenomena, they provide limited general
insight into trends across the full platforms. To study new questions not
covered by these topic-specific datasets, researchers often need to build new
datasets, which introduces confounders when these datasets need to be
collected retrospectively. Such problems include limits on search timeframes
(i.e., one can only go back so far), memory-hole problems (Marshall 2020)
(i.e., relevant information may have been deleted from the target platform,
especially a risk for anti-social behavior), or API changes. The MeLa-BitChute
dataset outlined herein solves these issues by providing a reusable, general
collection that characterizes the full BitChute platform over a multi-year
period, facilitating research questions in a consistent context.

These issues can be addressed with sufficiently large samples of the target
platform, and it is this type of sample that the MeLa-BitChute dataset
provides. Similar platform-wide datasets are available for other platforms,
most notably the Pushshift.io Reddit dataset (Baumgartner et al. 2020). Unlike
Reddit, BitChute is part of the alt-tech space and fits into a constellation
of recent work studying these alternative spaces, including releases from
Parler (Aliapoulios et al. 2021), Gab (Fair and Wesslen 2019), and Mastodon
(Zignani et al. 2019). Paralleling YouTube’s centrality in the mainstream
information ecosystem, BitChute and its video-hosting is likewise a core
element of the alt-tech space and is often a highly shared domain in other
fringe platforms and Telegram channels. Taken together, these collections
provide a crucial cross-platform view into the ecosystem.

Separate from the above published datasets, “hacktivists” have released
several large-scale datasets from alt-tech spaces, including Gab and Parler
(Sharma 2021). These datasets, such as those hosted on DDoSecrets.com and
Wikileaks, provide more insight into these platforms but at significant
ethical and intellectual risk. In particular, these leaked datasets often
contain private data, such as direct messages, that users would not intend for
public distribution. The MeLa-BitChute dataset instead exclusively contains
public-facing data, which may miss out on important activity like collusion,
brigading, radicalization, or other anti-social behaviors. These two sources
come with different insights and ethical considerations, and we leave
questions about which source is most appropriate to future researchers.

##  7 Conclusion

In this paper, we presented a dataset covering 3M+ videos, 61K+ channels, and
11.4M+ comments from the alt-tech, social, video-hosting platform BitChute. We
provided an in-depth description of our custom built data collection
infrastructure, documentation on the stored data, and a discussion of
potential use cases for the dataset. We argued due to the difficulty of data
collection, the academic literature is lacking diverse, large-scale studies of
BitChute and its role in the alt-tech ecosystem. By filling this gap,
researchers can gain a holistic-view of the alt-tech environment and the
potential public harms fueled by BitChute. The MeLa-BitChute dataset and
sample code can be found at:
https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/KRD1VS.

## References

  * Abdalla, Ally, and Jabri-Markwell (2021) Abdalla, M.; Ally, M.; and Jabri-Markwell, R. 2021.  Dehumanisation of ‘Outgroups’ on Facebook and Twitter: towards a framework for assessing online hate organisations and actors.  _SN Social Sciences_ 1(9): 1–36.  doi:10.1007/s43545-021-00240-4.
  * Aliapoulios et al. (2021) Aliapoulios, M.; Bevensee, E.; Blackburn, J.; Bradlyn, B.; Cristofaro, E. D.; Stringhini, G.; and Zannettou, S. 2021.  A Large Open Dataset from the Parler Social Network.  _Proceedings of the International AAAI Conference on Web and Social Media_ 15(1): 943–951.
  * Anti-Defamation League (2022) Anti-Defamation League. 2022.  Hate on Display: Hate Symbols Database.  Technical report, AntisDefamation League, New York.  URL https://www.adl.org/hate-symbols.
  * Baumgartner et al. (2020) Baumgartner, J.; Zannettou, S.; Keegan, B.; Squire, M.; and Blackburn, J. 2020.  The Pushshift Reddit Dataset.  _Proceedings of the International AAAI Conference on Web and Social Media_ 14(1): 830–839.  URL https://ojs.aaai.org/index.php/ICWSM/article/view/7347.
  * Bellemare, Nicholson, and Ho (2020) Bellemare, A.; Nicholson, K.; and Ho, J. 2020.  How a debunked COVID-19 video kept spreading after Facebook and YouTube took it down.  URL https://www.cbc.ca/news/science/alt-tech-platforms-resurface-plandemic-1.5577013.
  * Bond (2022) Bond, S. 2022.  Kicked off Facebook and Twitter, far-right groups lose online clout.  URL https://www.npr.org/2022/01/06/1070763913/kicked-off-facebook-and-twitter-far-right-groups-lose-online-clout.
  * Buckley and Schafer (2021) Buckley, N.; and Schafer, J. S. 2021.  “Censorship-free” platforms: Evaluating content moderation policies and practices of alternative social media.  doi:10.31235/osf.io/yf9qz.  URL https://osf.io/preprints/socarxiv/yf9qz/.
  * Buntain et al. (2021a) Buntain, C.; Bonneau, R.; Nagler, J.; and Tucker, J. A. 2021a.  Youtube recommendations and effects on sharing across online social platforms.  _PACM on Human-Computer Interaction_ 0(0).  ISSN 23318422.  doi:10.1145/3449085.  URL https://doi.org/10.1145/3449085.
  * Buntain et al. (2021b) Buntain, C.; Buntain, C.; Mccreadie, R.; and Soboroff, I. 2021b.  Incident Streams 2020 : TRECIS in the Time of COVID-19.  _18th International Conference on Information Systems for Crisis Response and Management_ (April).
  * Buntain et al. (2021c) Buntain, C.; Innes, M.; Mitts, T.; and Shapiro, J. N. 2021c.  Effects of the Post-January 6 Deplatforming on Social Media Discourse.
  * Chu et al. (2022) Chu, A.; Arunasalam, A.; Ozmen, M. O.; and Celik, Z. B. 2022.  Behind the Tube: Exploitative Monetization of Content on YouTube.  In _31st USENIX Security Symposium (USENIX Security 22)_. Boston, MA: USENIX Association.  URL https://www.usenix.org/conference/usenixsecurity22/presentation/chu.
  * David (2020) David, G. 2020.  BitChute: Platforming Hate and Terror in the Uk.  Technical report, HOPE not hate Charitable Trust.
  * Davidson et al. (2017) Davidson, T.; Warmsley, D.; Macy, M.; and Weber, I. 2017.  Automated Hate Speech Detection and the Problem of Offensive Language.  URL https://aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15665/14843.
  * Doesburg (2021) Doesburg, J. V. 2021.  _An alternative rabbit hole? An analysis of the construction of echo chambers within coronavirus activism groups on Telegram_.  Masters, Utrecht University.
  * Fair and Wesslen (2019) Fair, G.; and Wesslen, R. 2019.  Shouting into the void: A database of the alternative social media platform Gab.  _Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019_ (Icwsm): 608–610.
  * Fink (2018) Fink, C. 2018.  Dangerous Speech, Anti-Muslim Violence, and Facebook In Myanmar.  _Journal of International Affairs_ 71(1.5): 43–52.  ISSN 0022197X.  URL https://www.jstor.org/stable/26508117.
  * Forberg (2021) Forberg, P. L. 2021.  From the Fringe to the Fore: An Algorithmic Ethnography of the Far-Right Conspiracy Theory Group QAnon.  _Journal of Contemporary Ethnography_ 1–27.  ISSN 15525414.  doi:10.1177/08912416211040560.
  * Freelon, Marwick, and Kreiss (2020) Freelon, D.; Marwick, A.; and Kreiss, D. 2020.  False equivalencies: Online activism from left to right.  _Science_ 369(6508): 1197–1201.  ISSN 10959203.  doi:10.1126/SCIENCE.ABB2428.
  * Heilweil (2021) Heilweil, R. 2021.  Platforms are cracking down hard on political misinformation, but it’s still easy to find.  URL https://www.vox.com/recode/22240408/facebook-youtube-twitter-qanon-misinformation-inauguration.
  * Hoseini et al. (2021) Hoseini, M.; Melo, P.; Benevenuto, F.; Feldmann, A.; and Zannettou, S. 2021.  On the Globalization of the QAnon Conspiracy Theory Through Telegram.  In _arXiv preprint_.  URL http://arxiv.org/abs/2105.13020.
  * Júnior et al. (2021) Júnior, M.; Melo, P.; Da Silva, A. P. C.; Benevenuto, F.; and Almeida, J. 2021.  Towards Understanding the Use of Telegram by Political Groups in Brazil.  _ACM International Conference Proceeding Series_ 237–244.  doi:10.1145/3470482.3479640.
  * Kearney, Chiang, and Massey (2020) Kearney, M. D.; Chiang, S. C.; and Massey, P. M. 2020.  The Twitter origins and evolution of the COVID-19 “plandemic” conspiracy theory.  _Harvard Kennedy School Misinformation Review_ 1(October): 1–18.  doi:10.37016/mr-2020-42.
  * Kopf (2020) Kopf, S. 2020.  “Rewarding Good Creators”: Corporate Social Media Discourse on Monetization Schemes for Content Creators.  _Social Media and Society_ 6(4).  ISSN 20563051.  doi:10.1177/2056305120969877.
  * Marshall (2020) Marshall, J. M. 2020.  The Modern Memory Hole: Cyberethics Unchained.  _Athenaeum Review_ 1(3).  URL https://athenaeumreview.org/essay/the-modern-memory-hole-cyberethics-unchained/.
  * Masters of Media (2020) Masters of Media. 2020.  Deplatforming and the rise of the alt-tech video hosting platform BitChute in 2020.  Technical report, University of Amsterdam.  URL http://mastersofmedia.hum.uva.nl/blog/2020/09/27/deplatforming-and-bitchute/.
  * Müller and Schwarz (2021) Müller, K.; and Schwarz, C. 2021.  Fanning the Flames of Hate: Social Media and Hate Crime.  _Journal of the European Economic Association_ 19(4): 2131–2167.  ISSN 1542-4766.  doi:10.1093/jeea/jvaa045.
  * Munger and Phillips (2019) Munger, K.; and Phillips, J. 2019.  A Supply and Demand Framework for YouTube Politics.  In _OSF Preprint_. OSF.  URL osf.io/4wk63.
  * Olteanu et al. (2014) Olteanu, A.; Castillo, C.; Diaz, F.; and Vieweg, S. 2014.  CrisisLex: A Lexicon for Collecting and Filtering Microblogged Communications in Crises.  _Proc. of the 8th International Conference on Weblogs and Social Media_ 376\.  doi:10.1.1.452.7691.  URL http://www.aaai.org/ocs/index.php/ICWSM/ICWSM14/paper/download/8091/8138.
  * Papadamou (2021) Papadamou, K. 2021.  _Characterizing Abhorrent, Misinformative, and Mistargeted Content on YouTube_.  Doctoral thesis, Cyprus University of Technology.  URL http://arxiv.org/abs/2105.09819.
  * Paudel et al. (2021) Paudel, P.; Blackburn, J.; De Cristofaro, E.; Zannettou, S.; and Stringhini, G. 2021.  An Early Look at the Gettr Social Network URL http://arxiv.org/abs/2108.05876.
  * Pelzer (2018) Pelzer, R. 2018.  Policing of Terrorism Using Data from Social Media.  _European Journal for Security Research_ 3(2): 163–179.  ISSN 2365-0931.  doi:10.1007/s41125-018-0029-9.  URL https://doi.org/10.1007/s41125-018-0029-9.
  * Qazi, Imran, and Ofli (2020) Qazi, U.; Imran, M.; and Ofli, F. 2020.  GeoCoV19: A Dataset of Hundreds of Millions of Multilingual COVID-19 Tweets with Location Information.  _SIGSPATIAL Special_ 12(1): 6–15.  doi:10.1145/3404820.3404823.  URL https://doi.org/10.1145/3404820.3404823.
  * Rauchfleisch and Kaiser (2021) Rauchfleisch, A.; and Kaiser, J. 2021.  Deplatforming the Far-right: An Analysis of YouTube and BitChute.  _SSRN Electronic Journal_ (108).  doi:10.2139/ssrn.3867818.
  * Rogers (2020) Rogers, R. 2020.  Deplatforming: Following extreme Internet celebrities to Telegram and alternative social media.  _European Journal of Communication_ 35(3): 213–229.  ISSN 14603705.  doi:10.1177/0267323120922066.
  * Rye, Blackburn, and Beverly (2020) Rye, E.; Blackburn, J.; and Beverly, R. 2020.  Reading In-Between the Lines: An Analysis of Dissenter.  _Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC_ 133–146.  doi:10.1145/3419394.3423615.
  * Sharma (2021) Sharma, A. 2021.  Anonymous Leaked a Bunch of Data From a Right-Wing Web Host.  URL https://arstechnica.com/information-technology/2021/09/anonymous-leaks-gigabytes-of-data-from-epik-web-host-of-gab-and-parler/.
  * Thompson (2011) Thompson, R. 2011.  Radicalization and the Use of Social Media.  _Journal of Strategic Security_ 4(4): 167–190.  ISSN 19440464, 19440472.  URL http://www.jstor.org/stable/26463917.
  * Trujillo et al. (2020) Trujillo, M.; Gruppi, M.; Buntain, C.; and Horne, B. D. 2020.  What is BitChute? Characterizing the ”Free Speech” Alternative to YouTube.  In _Proceedings of the 31st ACM Conference on Hypertext and Social Media_ , HT ’20, 139–140. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450370981.  doi:10.1145/3372923.3404833.  URL https://doi.org/10.1145/3372923.3404833.
  * Tucker et al. (2018) Tucker, J.; Guess, A.; Barber, P.; Vaccari, C.; Nyhan, B.; Seigel, A.; Sanovich, S.; and Stukal, D. 2018.  Social Media, Political Polarization, and Political Disinformation: A Review of the Scientific Literature.  Technical Report March, Hewlett Foundation.
  * Van Dijck, de Winkel, and Schäfer (2021) Van Dijck, J.; de Winkel, T.; and Schäfer, M. T. 2021.  Deplatformization and the governance of the platform ecosystem.  _New Media and Society_ (January).  ISSN 14617315.  doi:10.1177/14614448211045662.
  * Warreth (2021) Warreth, S. 2021.  _Comparing Far Right and Jihadi Use of Crowdfunding, Cryptocurrencies, and Blockchain Technology: Accessibility, Geography, Ideology_.  Masters thesis, University of Glasgow.  doi:10.13140/RG.2.2.27543.09123.
  * Wilson and Starbird (2021) Wilson, T.; and Starbird, K. 2021.  Cross-platform Information Operations: Mobilizing Narratives and Building Resilience through both ’Big’ and ’Alt’ Tech.  _Proceedings of the ACM on Human-Computer Interaction_ 5(CSCW2).  ISSN 25730142.  doi:10.1145/3476086.
  * Zannettou et al. (2018) Zannettou, S.; Bradlyn, B.; De Cristofaro, E.; Kwak, H.; Sirivianos, M.; Stringini, G.; and Blackburn, J. 2018.  What is gab: A bastion of free speech or an alt-right echo chamber.  In _Companion Proceedings of the The Web Conference 2018_ , volume 12, 1007–1014.  ISBN 9781450356404.  ISSN 10367128.  doi:10.1254/fpj.130.403.
  * Zignani et al. (2019) Zignani, M.; Quadri, C.; Galdeman, A.; Gaito, S.; and Rossi, G. P. 2019.  Mastodon content warnings: Inappropriate contents in a microblogging platform.  _Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019_ (Icwsm): 639–645.

##  Appendix A Data Column Descriptions

In this appendix, we provide detailed descriptions of each data column in the
MeLa-BitChute dataset. Below are tables for each table in the database
(videos, comments, channels).

Column Name |  Description
---|---
url |  URL to video
title |  Title of the video
postdate |  Date video was uploaded to BitChute. Unparsed from the website, like “First published at 17:31 UTC on August 28th, 2020.”
scrapedate |  Date video was added to our database, Unix Epoch time, accuracy in seconds
description |  Uploader-chosen description of their video
channel |  URL to the channel
profile |  URL to the uploader’s profile. Note, a profile can have multiple channels, but a channel belongs to one profile.
category |  Uploader picked category of the video. Note, the default category is ‘None’ or ‘Other’ depending on if the video comes before or after September 2020.
sensitivity |  Uploader chosen sensititivity score, chosed from: “Normal”, “NSFW”, and “NSFL”
views |  Integer number of video views at data collection time. If the views data is $-1$, the video was removed by the uploader. If views is $-3$, the video was removed by BitChute.
view_scrapedate |  Unix Epoch time when views and comments were added to the database. Guaranteed to be at least one week after scrapedate.
Table 1: videos data description Column Name |  Description
---|---
url |  URL to video that the comment falls under
userid |  A SHA256 hash that uniquely identifies each commenter
posthtml |  The full HTML of the comment
posttext |  The body text of the comment (a pre-processed version of posthtml)
comment_id |  A text ID identifying a comment on a video
parent_id |  If non-NULL, refers to the comment_id of the parent comment
Table 2: comments data description Column Name |  Description
---|---
url |  URL to the channel
scrapedate |  UTC timestamp of the date on which the description data was collected
description |  Full HTML of channel description on the ‘About’ page of the channel. These range from very long descriptions to short or blank descriptions. If no description was found, the value of the column will be ‘Null’. However, note that occasionally we found channels with descriptions that were multiple blank characters, making the stored value not ‘Null’
description_stripped |  Text of description stripped from the HTML
Table 3: channels data description