# The Surprising Performance of Simple Baselines for Misinformation Detection

Kellin Pelrine<EMAIL_ADDRESS>SOCS, McGill UniversityMila -
Quebec AI Institute ,  Jacob Danovitch<EMAIL_ADDRESS>SOCS,
McGill UniversityMila - Quebec AI Institute and  Reihaneh Rabbany
<EMAIL_ADDRESS>SOCS, McGill UniversityMila - Quebec AI Institute

(2021; 2021)

###### Abstract.

As social media becomes increasingly prominent in our day to day lives, it is
increasingly important to detect informative content and prevent the spread of
disinformation and unverified rumours. While many sophisticated and successful
models have been proposed in the literature, they are often compared with
older NLP baselines such as SVMs, CNNs, and LSTMs. In this paper, we examine
the performance of a broad set of modern transformer-based language models and
show that with basic fine-tuning, these models are competitive with and can
even significantly outperform recently proposed state-of-the-art methods. We
present our framework as a baseline for creating and evaluating new methods
for misinformation detection. We further study a comprehensive set of
benchmark datasets, and discuss potential data leakage and the need for
careful design of the experiments and understanding of datasets to account for
confounding variables. As an extreme case example, we show that classifying
only based on the first three digits of tweet ids, which contain information
on the date, gives state-of-the-art performance on a commonly used benchmark
dataset for fake news detection –Twitter16. We provide a simple tool to detect
this problem and suggest steps to mitigate it in future datasets.

misinformation, social media, natural language processing, datasets, COVID-19

††journalyear: 2021††copyright: iw3c2w3††conference: Proceedings of the Web
Conference 2021; April 19–23, 2021; Ljubljana, Slovenia††booktitle:
Proceedings of the Web Conference 2021 (WWW ’21), April 19–23, 2021,
Ljubljana, Slovenia††doi: 10.1145/3442381.3450111††isbn:
978-1-4503-8312-7/21/04††copyright: iw3c2w3††journalyear: 2021††doi:
10.1145/3442381.3450111††conference: WWW ’21, April 19–23, 2021, Ljubljana,
Slovenia; Apr 2021; ††booktitle: The Web Conference ’21††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies††ccs: Computing
methodologies Artificial intelligence††ccs: Computing methodologies Natural
language processing

##  1\. Introduction

Social media is filled with both treasure troves and landmines of information.
There are far too many interactions to mine them by hand. But failing to
understand them can have dire consequences. Misinformation can challenge fair
elections (Meel and Vishwakarma, 2020) and cost billions of dollars (Rapoza,
2017). Misinformation spread in the COVID-19 ”infodemic” (Cinelli et al.,
2020; Pulido et al., 2020) can cost lives. At the same time, successful mining
of useful information can enable contact tracing and other measures to save
lives (Li et al., 2020; Qin et al., 2020), as well as other applications like
explainable fact checking (Cui et al., 2020). Motivated by the profound
impact, there has been substantial research on detecting fake news,
misinformation, and related topics (such as bot and troll detection) in the
past few years (Bondielli and Marcelloni, 2019; Shu et al., 2017a; Mosinzova
et al., 2019; Cao et al., 2018; Nguyen et al., 2020b). A majority of these
works are based on text/content classification (Shu et al., 2017b; Zhou and
Zafarani, 2018; Zhou et al., 2019; Shu et al., 2019a). As real-world events
such as the spread of COVID-19 misinformation show (Cinelli et al., 2020;
Pulido et al., 2020), we are still far from addressing misinformation and more
work is needed.

In this paper, we examine the use of modern pre-trained language models (LMs)
for classifying content. Although language models are ubiquitous baselines in
this domain, the ones used are often older models such as SVMs, CNNs, LSTMs,
etc. (Denaux and Gomez-Perez, 2020; Cheng et al., 2020a; Shu et al., 2019b;
Han et al., 2020), instead of transformer-based models that have been dominant
in NLP recently, such as the exemplar BERT model (Devlin et al., 2018; Xia et
al., 2020). Here, we report the performance of these recently proposed
language models on common benchmark datasets for misinformation detection.
More specifically, we consider small, medium and large models and show that
most can achieve comparable or even better performance than state-of-the-art
(SOTA) methods. Even the smallest 4 million parameter BERT-Tiny (Turc et al.,
2019) can approach or beat state-of-the-art in some cases. These SOTA methods
are usually much more complex and in many cases incorporate more information
beyond the content, e.g. the reply thread. Although some of these more
sophisticated methods are also designed to incorporate language models, it is
not straightforward how to change their language module and they may or may
not synergize well with the latest language models.

We further study commonly used benchmark datasets and report a significant
issue with the Twitter15 and Twitter16 datasets (Liu et al., 2015; Ma et al.,
2016, 2017), which can produce unrealistically high performance just by
inferring the tweet date. This issue is also present to a lesser degree in
FakeNewsNet (Shu et al., 2018). We propose a simple test to identify it in
other datasets and suggest how to mitigate it in data collection.

We also provide the exact splits to use our results as benchmarks, as well as
other suggestions on proper evaluation techniques, and discuss potential
future directions of work in this domain.

To summarize, the main contributions of this paper are threefold:

  * •

We show that recent pretrained LMs are competitive with and sometimes even
substantially better than SOTA models in common benchmarks for misinformation
detection.

  * •

We discuss experiment design considerations, effect of different data splits,
and further demonstrate potential data leakage issues on commonly used
benchmark datasets.

  * •

We package our framework as a simple yet strong baseline for relevant tasks,
and provide the tweet IDs to exactly reconstruct our splits, facilitating use
of our reported results for benchmarks. Our framework is available on GitHub.

##  2\. Background and Related Work

There are two key literatures related to this work: (mis-)information
classification on the application domain side, and natural language processing
models on the methodological side. We discuss first the former and then the
latter.

###  2.1. Detecting Misinformation

We can broadly divide misinformation detection algorithms into methods that
analyze content and (social) context. Here, we focus on the methods that are
content-based. For more complete review of the related works, please refer to
the recent surveys (Bondielli and Marcelloni, 2019; Shu et al., 2017a;
Mosinzova et al., 2019; Cao et al., 2018; Zhou et al., 2019; Shu et al.,
2019a). The content has been analyzed previously on multiple levels. At the
word level, methods have used features including word counts, derived measures
like TF-IDF, small combinations of words like bigrams, and word embeddings
(Huang et al., 2020a; Rubin et al., 2016; Castillo et al., 2011; Benamira et
al., 2019). At the sentence level, methods rely on features like syntax and
complexity (Huang et al., 2020a; Rubin et al., 2016; Castillo et al., 2011;
Benamira et al., 2019). Finally, there are features derived from the text as a
whole, including general representations and more specialized ones such as
topic or sentiment analysis (Cheng et al., 2020a; Cui et al., 2019; Islam et
al., 2019; Ito et al., 2015).

More specifically, we consider the following state-of-the-art algorithms as a
representative set of the current methods:

  * •

Cheng et al. (2020a) use a multitask (rumor detection, tracking, veracity
classification, and stance classification) LSTM-based VAE (Kingma and Welling,
2013), with the text as input.

  * •

Han et al. (2020) use a GNN (Scarselli et al., 2008) to encode the propagation
network and features of the users in it (not including text).

  * •

Huang et al. (2020c) build a heterogeneous tweet-word-user graph, linking the
representations with an attention mechanism.

  * •

Lu and Li (2020) use a co-attention mechanism to combine text and propagation
information.

  * •

Shu et al. (2019b) use attention mechanisms to combine word- and sentence-
level features from articles, encoded with a GRU, with a comment (tweet)
encoder.

  * •

Wang et al. (2020) link text and object detections in images with entities and
a knowledge graph constructed from them.

  * •

Wu et al. (2020b) uses a combination of a decision tree model and co-attention
to evaluate credibility of tweets using text, propagation, and user
information.

  * •

Wu et al. (2020a) build a propagation graph from tweets and replies. They
embed each using Doc2Vec (Le and Mikolov, 2014), and use a GRU (Gated
Recurrent Unit) (Cho et al., 2014) and attention mechanism to pass information
and pool the graph.

The most related work to ours is (Huang et al., 2020b). They highlight ”cross-
source” failure of existing misinformation detection methods, in terms of
training on one dataset and testing on another, and also propose a new model
designed to solve it. Our results support and expand on theirs by identifying
”cross-topic” and ”cross-domain” effects even within the same datasets,
depending on how they are split, as well as differences between datasets. We
take a different direction for modeling: rather than focusing on cross-source
tasks and developing a specialized model, we look at a spectrum of tasks and
ask how general language models perform in this domain.

###  2.2. Detecting Informative Content

Instead of detecting which tweets are spreading misinformation, there are also
many contexts in which we want detect tweets that provide useful information.
Recently, there has been a big focus in the research community on COVID-19
(Brainard, 2020). Mining information about disaster events is not new (Imran
et al., 2016; Takahashi et al., 2015; Kankanamge et al., 2020; Chen et al.,
2020; Kumar et al., 2011; To et al., 2017; Singh et al., 2019; Aggarwal,
2019). But COVID-19 is unprecedented in its global scale and impact, and
requires new methods and understanding.

Although the focus of our experiments is on misinformation, we include a
dataset for detecting informative COVID-19 tweets - discussed in more detail
in section 3.2 - to highlight the connections between these two areas and how
language models can model both as content classification.

###  2.3. Pre-Trained Language Models

The invention of word embeddings was a fundamental breakthrough (Mikolov et
al., 2013a; Mikolov et al., 2013b) leading towards today’s language models. By
predicting words, one can go beyond one-hot and similar encodings and convert
words into vectors that encode their meaning and are more useful for
downstream tasks.

The first embedding models such as Word2Vec (Mikolov et al., 2013a; Mikolov et
al., 2013b), GloVe (Pennington et al., 2014), and FastText (Joulin et al.,
2016) generate fixed vectors for each input word or token. This has the
advantage of simplicity, but can struggle to capture words whose meanings vary
depending on the context, and in turn struggle with downstream tasks that
require this level of language understanding. Therefore, better methods to
incorporate context became a key research topic. A followup direction was to
use convolutional neural networks, ubiquitous in computer vision, on text
(Kim, 2014). These could effectively capture local context, but still missed
incorporating longer range context. Similarly, RNNs, which could capture local
sequences, had problems with vanishing gradients and could not capture longer
range information (Minaee et al., 2020).

LSTMs (Hochreiter and Schmidhuber, 1997), which like RNNs operate on sequences
but allow longer range dependencies, helped solve this issue. There are state
of the art models for misinformation detection, including the ones discussed
in the domain section above, which use various versions or improvements on
LSTMs, such as the GRU by Cho et al. (2014). These also led to contextualized
embeddings such as ELMo (Peters et al., 2018).

Another major breakthrough came with attention (Bahdanau et al., 2014; Yang et
al., 2016) and then, based on attention, transformers (Vaswani et al., 2017).
These facilitate far more parallelization than LSTMs (Minaee et al., 2020),
which means models based on them can be trained on much larger and more
general corpora. A key early model using these is BERT (Devlin et al., 2018)
(discussed further in the following section), and there has subsequently been
numerous different models proposed based on transformers (Xia et al., 2020).
These vary from small tweaks to existing models such as pretraining on domain-
specific data (Müller et al., 2020; Nguyen et al., 2020a), to significantly
different models such as GPT (and then GPT-2 and GPT-3) (Radford et al., 2018,
2019; Brown et al., 2020). Virtually all of these models can provide
contextualized word/sentence/document embeddings for downstream tasks. These
models are easily accessible through packages such as (Wolf et al., 2019), and
with hundreds of versions to choose from, as Xia et al. (2020) suggests the
hardest question may be which one to use.

Although there are many excellent options, a large number of recent papers in
the misinformation domain still benchmark their methods against older models
(Denaux and Gomez-Perez, 2020; Cheng et al., 2020a; Shu et al., 2019b; Han et
al., 2020; Cui et al., 2020; Wu et al., 2019; Ajao et al., 2019). While there
are still contexts in which these models do well and there is nothing wrong
with making these comparisons, our results show that this can sometimes result
in weak benchmarks when the more modern language models are omitted. Among
other results, we give suggestions on how to incorporate these new models into
evaluation in this domain. Please note that although training these newest
models from scratch can require a prohibitive amount of computational
resources, pretrained weights are readily available and are commonly used
instead.

##  3\. Methodology

Our baselines simply consist of a set of language models and benchmark
datasets. Here we first explain the language models we consider in our
baseline framework in Section 3.1, then we describe the benchmark datasets in
Section 3.2. For the Funnel Transformer and ALBERT, we employ a mean pooling
strategy over the final layer of output. For ELMo we use an LSTM pooler. For
all others, we use the [CLS] token embedding from the final layer of output.
For all models, we allow all parameters of the language model to be fine-
tuned, use a single fully-connected layer on the pooled embeddings for
classification, train using cross-entropy loss, and use AdamW (Loshchilov and
Hutter, 2017) with a slanted triangular learning rate scheduler for
optimization.

###  3.1. Language Models

We evaluate a variety of language models. Where available, we use the
implementations and pre-trained weights provided by the Huggingface
Transformers library (Wolf et al., 2019), and train using PyTorch (Paszke et
al., 2017) and AllenNLP (Gardner et al., 2017). We include the following
language models (with the exact version111As available at
https://huggingface.co/models in parentheses):

BERT (bert-base-uncased) (Devlin et al., 2018).
Bidirectional Encoder Representations from Transformers, or BERT for short, is
a large pre-trained bidirectional transformer. BERT was pre-trained using two
objectives. The first, masked language modelling, required BERT to predict a
masked token from the input. The second, next sentence prediction, required
predicting whether two sentences appeared consecutively in the training
corpus. To complete the latter task, BERT prepends the input text with a
special [CLS] token, and inserts a special [SEP] token between the first and
second sentence as well as at the end of the second sentence. The [CLS] token
is commonly used as a document-level representation for classification tasks.

BERT-Tiny (bert_uncased_L-2_H-128_A-2) (Turc et al., 2019).
BERT-Tiny is the smallest of several pre-trained language models that follow
the same pre-training procedure as BERT, as well as a knowledge distillation
fine-tuning procedure. These smaller models achieve strong performance on
downstream tasks with significantly fewer parameters.

RoBERTa (roberta-large) (Liu et al., 2019).
RoBERTa follows a similar architecture to BERT but removes the next-sentence
prediction pre-training objective while making the masking procedure dynamic
by regenerating the mask for each example every time and leverage larger batch
sizes and increased training iterations to improve performance. They find that
BERT underfits its training data.

ALBERT (albert-large-v2) (Lan et al., 2020).
ALBERT adds an additional pre-training objective while incorporating two
methods that reduce the number of parameters in the model, factorizing the
embedding layer and tying weights across hidden layers.

BERTweet (bertweet-base) (Nguyen et al., 2020a).
BERTweet follows an identical training procedure to RoBERTa, and is pre-
trained on 850 million tweets.

COVID-Twitter-BERT (covid-twitter-bert-v2) (Müller et al., 2020).
COVID-Twitter-BERT (CT-BERT) follows an identical training procedure to BERT,
and the latest version is pre-trained on 1.2 billion training examples
generated from 97 million tweets.

DeCLUTR (declutr-base) (Giorgi et al., 2020).
DeCLUTR is a transformer-based language model that proposes a contrastive,
self-supervised method for learning general purpose sentence embeddings. The
model is trained with a masked language modelling objective as well as with
contrastive loss using both easy and hard negatives.

Funnel Transformer (xlarge-base) (Dai et al., 2020).
The Funnel Transformer improves the efficiency of bidirectional transformer
models by applying a pooling operation after each layer, akin to convolutional
neural networks, to reduce the length of the input.

ELMo (Original222Available at https://allennlp.org/elmo) (Peters et al.,
2018).
ELMo is one of the first large-scale pre-trained language models. It is a
character-based model and is the only model in this list that does not
optimize for masked language modelling. Instead, it uses bidirectional LSTMs
(Hochreiter and Schmidhuber, 1997) to perform autoregressive language
modelling in both the forward and backward directions.

###  3.2. Benchmark Datasets

We consider the comprehensive set of benchmark datasets available in the
literature for misinformation (information) detection. In the following we
briefly explain each dataset.

####  3.2.1. PHEME

The PHEME dataset (Zubiaga et al., 2016; Kochkina et al., 2018) contains 6425
tweets about 9 newsworthy events. The events are unrelated. The tweets were
collected as the stories developed, with a journalist annotating them as
rumour vs. non-rumour. They grouped them by story, and then marked whether the
stories were true or false once it was confirmed, or as unverified if they
could not be certain during the collection period. There are also other labels
such as stance, which we do not consider here.

We split this dataset five different ways to compare with different state-of-
the-art results that use the corresponding settings.

PHEME9 T/F which matches Wu and Rao (2020).
First, we take only the tweets marked as true or false and do a 70-10-20
train-dev-test split.

PHEME5 R/NR which matches Wang et al. (2020).
Next we take the 5 largest events (comprising 90% of the dataset) and use the
rumour and non-rumour labels. We again split 70-10-20. This setup matches both
(Wang et al., 2020) (70-30 split with no dev set) and (Cheng et al., 2020b)
(which notes 10% withheld for validation).

PHEME5 3-way which matches Cheng et al. (2020b).
Here, we take the 5 largest events and tweets labeled true, false, and
unverified, splitting 70-10-20.

PHEME9 4-way which matches Wu et al. (2020a).
For this, we subsample 800, 400, 600, 500 tweets with at least 3 replies, that
are non-rumor, false, true, and unverified, respectively. They are split
80-10-10.

PHEME5 Lc which matches Cheng et al. (2020b).
Finally, we also examine an event-based split. Starting with the 5 largest
events, we train on four and test on the largest (tweets related to Charlie
Hebdo terror attack), with the 3-way true, false, and unverified labels.

####  3.2.2. FakeNewsNet

The FakeNewsNet dataset (Shu et al., 2018, 2017c, 2017b) contains articles
fact-checked by PolitiFact333https://www.politifact.com/ or
GossipCop444https://www.gossipcop.com/, and related tweets. The labels are
”real” and ”fake.” We retrieve 467 thousand tweets in the PolitiFact dataset,
and 1.25 million in GossipCop. We split this dataset by first splitting the
articles, then assigning tweets to each split based on the article they are
associated with. We divide it 75-10-15, corresponding to the splits in (Shu et
al., 2019a; Han et al., 2020; Shu et al., 2020). We remove two events from
Politifact that are labeled both real and fake, politifact14920 and
politifact14940. Work on this dataset such as (Shu et al., 2019a; Han et al.,
2020) has focused on classifying the articles, and using the tweets and
related user information as supplemental information to improve performance.
Because our pipeline was set up for tweet data rather than articles, we apply
a simple way of classifying the articles through the tweets, by classifying
the tweets and converting the predictions to an article prediction based on
majority vote of the corresponding tweets’ labels. Note that the works we
compare with use the tweets, so this is not using an additional type of data
that they have excluded. Following (Shu et al., 2019a; Han et al., 2020), we
evaluate only on articles with at least 3 corresponding tweets.

####  3.2.3. Twitter15 and Twitter16

These datasets (Liu et al., 2015; Ma et al., 2016, 2017) contain tweets
related to stories from fact-checking websites and random tweets. There are
1490 and 818 tweets respectively, labeled true, false, unverified, or non-
rumor. We reserve 10% for the dev set, then split the remainder 75:25,
matching (Huang et al., 2020c).

####  3.2.4. Twitter15 T/F and Twitter16 T/F

These are Twitter15 and Twitter16 with the true and false examples only. This
gives 742 and 412 examples respectively. We split 70-10-20, giving the same
training set size as (Lu and Li, 2020), which split 70-30 with no dev set.

####  3.2.5. WNUT-2020

This dataset was created for a shared task in WNUT-2020 on classifying tweets
as “informative” or “uninformative”, in providing information about
”recovered, suspected, confirmed and death cases as well as location or travel
history of the cases” (Nguyen et al., 2020b). Many of the models submitted in
the competition were based on pre-trained transformer-based language models
(Wadhawan, 2020; Tran et al., 2020; Nguyen, 2020; Huynh et al., 2020; Nguyen,
2020; Huynh et al., 2020; Babu and Eswari, 2020; Perrio and Madabushi, 2020;
Maveli, 2020; Chauhan, 2020), including the first place model which was an
ensemble of CT-BERT and RoBERTa (Kumar and Singh, 2020), and the tied first
place model which was a carefully tuned version of CT-BERT (Møller et al.,
2020). A 70-10-20 split was provided by the task organizers.

####  3.2.6. CoAID

The CoAID dataset (Cui and Lee, 2020) is a misinformation dataset related to
COVID-19. This dataset is new and consists of tweets, articles, and claims.
The authors provide baselines on the articles. To the best of our knowledge,
as of this writing, no existing work has performed classification on the
tweets.

We conducted experiments here aimed to establish a baseline for future works.
However, we found that most splits result in virtually all models obtaining
near-perfect (above 95) F1 score. We examined in particular the “news” tweets.
Many tweets about the same news item have similar or even duplicated text, so
we tried splitting similar to FakeNewsNet, first splitting the news articles
and then assigning tweets to each set based on their corresponding article. We
also tried subsampling a small train set to make the problem harder. This did
result in one set of results with lower F1, but we found it was unstable and
did not replicate consistently when we redid the splitting and training
process. Thus, we do not report results on this dataset here. However, it
contains a significant amount of useful data on an important problem, so we
encourage future work to determine challenging splits and help others take
full advantage of this resource.

###  3.3. Implementation Details

Each model is trained on an RTX8000 GPU using mixed precision with a batch
size of 32. and learning rate of $1\mathrm{e}$-5. We do not perform any
hyperparameter tuning. We train for 50 epochs on all datasets except
Politifact and Gossipcop. There we train for 2 epochs, because they are much
larger than the rest.

We run each model to completion exactly 5 times, and report mean and standard
deviation, with two exceptions. First, datasets with a fixed split (PHEME5 Lc
and WNUT-2020), which are run once. Second, a bug caused a small number of
GossipCop results with Funnel to be lost, and due to time constraints we were
unable to fully repeat the experiments. Thus we do not report results for that
model and dataset.

Complete implementation details, hyperparameter configurations, and tweet IDs
for each split of each dataset can be found on
GitHub.555https://github.com/ComplexData-MILA/misinfo-baselines

###  3.4. Experimental Results

####  3.4.1. Performance Evaluation

We roughly categorize the algorithms by size. The first set are ”large” models
with around 400 million parameters. The second set are ”medium” ones with
around 100 million. The third set are ”small” ones under 20 million. Order in
the tables is alphabetical within category. The exact parameter counts are
shown in table 1.

Results are shown in tables 2 and 3. The first row presents the algorithm that
achieves, to the best of our knowledge, state-of-the-art (SOTA) performance on
the given dataset and split.

Table 1. Number of Parameters CT-BERT | Funnel | RoBERTa | BERT | BERTweet | DeCLUTR | ELMo | ALBERT | BERT-Tiny
---|---|---|---|---|---|---|---|---
340M | 468M | 355M | 110M | 135M | 125M | 94M | 17M | 4M
Table 2. PHEME Results (Macro F1 score) | PHEME9 T/F | PHEME5 R/NR | PHEME5 3-way | PHEME9 4-way | PHEME5 Lc  | Average Rank
---|---|---|---|---|---|---
SOTA | 82.5 (Wu et al., 2020b) | 87.6 (Cheng et al., 2020b; Wang et al., 2020) | 66.7 (Cheng et al., 2020b) | 75.3 (Wu et al., 2020a) |  51.3 (Cheng et al., 2020b) | 5.6
CT-BERT | 92.0 $\pm$ 0.9 | 89.0 $\pm$ 0.8 | 84.6 $\pm$ 1.5 | 79.0 $\pm$ 2.6 | 27.9  | 3.4
Funnel | 86.7 $\pm$ 3.2 | 87.3 $\pm$ 0.6 | 79.4 $\pm$ 3.7 | 71.4 $\pm$ 3.3 | 28.7  | 6.4
RoBERTa | 93.2 $\pm$ 0.9 | 89.4 $\pm$ 0.3 | 87.7 $\pm$ 1.9 | 82.5 $\pm$ 3.3 | 29.0  | 2.0
BERT | 89.9 $\pm$ 1.1 | 87.2 $\pm$ 0.4 | 81.2 $\pm$ 1.4 | 76.8 $\pm$ 2.7 | 24.2  | 5.8
BERTweet | 89.8 $\pm$ 0.6 | 87.3 $\pm$ 0.6 | 81.8 $\pm$ 0.9 | 76.6 $\pm$ 4.1 | 29.0  | 5.0
DeCLUTR | 90.2 $\pm$ 0.8 | 88.3 $\pm$ 0.4 | 83.7 $\pm$ 2.1 | 77.8 $\pm$ 3.5 | 30.2  | 3.2
ELMo | 81.7 $\pm$ 2.4 | 84.2 $\pm$ 0.8 | 65.8 $\pm$ 1.8 | 64.3 $\pm$ 4.0 | 30.3  | 9.4
ALBERT | 85.3 $\pm$ 2.9 | 84.2 $\pm$ 2.7 | 71.1 $\pm$ 2.2 | 65.7 $\pm$ 3.1 | 29.4  | 7.2
BERT-Tiny | 81.6 $\pm$ 2.0 | 84.7 $\pm$ 0.8 | 67.3 $\pm$ 2.0 | 61.0 $\pm$ 2.5 | 36.5  | 7.6
Table 3. Other Dataset Results (Macro F1 score) | PolitiFact | GossipCop | Twitter15 | Twitter16 | Twitter15 T/F | Twitter16 T/F |  WNUT-2020  | Average Rank
---|---|---|---|---|---|---|---|---
SOTA |  92.8 (Shu et al., 2019a) |  85.0 (Han et al., 2020) |  91.0 (Huang et al., 2020c) |  92.4 (Huang et al., 2020c) |  82.5 (Lu and Li, 2020) |  75.9 (Lu and Li, 2020) |  91.0 (Kumar et al., 2011; Møller et al., 2020) | 4.4
CT-BERT |  86.0 $\pm$ 3.2 |  90.6 $\pm$ 0.2 |  83.5 $\pm$ 2.8 |  83.9 $\pm$ 0.9 |  93.8 $\pm$ 1.6 |  94.0 $\pm$ 3.5 |  90.6  | 2.8
Funnel |  86.4 $\pm$ 3.2 |  –666Due to a bug, runs for this model and dataset were lost. See section 3.3 for details. |  66.9 $\pm$ 3.0 |  69.6 $\pm$ 2.9 |  83.2 $\pm$ 3.8 |  90.8 $\pm$ 2.2 |  88.5  | –
RoBERTa |  86.7 $\pm$ 1.2 | 92.8 $\pm$ 0.5 |  81.8 $\pm$ 1.5 |  84.8 $\pm$ 1.9 | 94.4 $\pm$ 0.8 | 95.7 $\pm$ 2.8 |  90.5  | 2.3
BERT |  81.8 $\pm$ 3.0 |  89.8 $\pm$ 0.4 |  77.5 $\pm$ 3.3 |  78.2 $\pm$ 4.1 |  89.7 $\pm$ 1.6 |  91.6 $\pm$ 4.5 |  88.5  | 5.3
BERTweet |  88.5 $\pm$ 1.2 |  92.6 $\pm$ 0.6 |  76.7 $\pm$ 2.9 |  77.7 $\pm$ 2.7 |  86.7 $\pm$ 1.8 |  92.0 $\pm$ 3.7 |  88.8  | 4.4
DeCLUTR |  36.6 $\pm$ 1.4777Although in most cases our untuned hyperparameters work well, it appears they are not appropriate for DeCLUTR on FakeNewsNet. |  43.3 $\pm$ 0.4b |  80.4 $\pm$ 2.6 |  80.5 $\pm$ 1.7 |  91.7 $\pm$ 1.5 |  94.5 $\pm$ 2.5 |  89.1  | 5.1
ELMo |  83.1 $\pm$ 1.6 |  92.0 $\pm$ 0.5 |  53.7 $\pm$ 2.7 |  55.5 $\pm$ 4.9 |  74.4 $\pm$ 3.5 |  83.3 $\pm$ 5.0 |  82.4  | 8.0
ALBERT |  80.1 $\pm$ 2.9 |  88.2 $\pm$ 0.9 |  63.4 $\pm$ 4.0 |  68.0 $\pm$ 3.5 |  83.3 $\pm$ 1.8 |  88.9 $\pm$ 4.3 |  86.8  | 7.7
BERT-tiny |  85.3 $\pm$ 2.8 |  86.5 $\pm$ 0.6 |  54.6 $\pm$ 3.4 |  48.8 $\pm$ 3.9 |  77.8 $\pm$ 4.8 |  77.8 $\pm$ 4.9 |  79.9  | 8.9

We discuss these results in section 4.

####  3.4.2. Potential Data Leakage

During our analysis, we observed a trend in the Twitter15 and Twitter16
dataset. It appears that time is a highly discriminative factor in separating
the classes. This creates the possibility of data leakage by means of several
potential confounding variables. To illustrate, we demonstrate that
competitive performance can be achieved by using only the first 2 or 3 digits
of the tweet ID. Due to the way unique IDs are generated for tweets, these
digits reveal information about the time the tweet was
posted.888https://github.com/client9/snowflake2time We train a random forest
on these digits, setting a high depth (25, such that increasing it further
does not change validation performance) and otherwise using the default
scikit-learn (Pedregosa et al., 2011) parameters.

The class labels are balanced, so a random baseline will have 25% F1. This is
roughly what one would intuitively expect from a model using only IDs as
features, because the time, particularly at this approximate scale, should
have little correlation if any with whether a tweet is true or false, and
would not be useful on its own for classification in applied settings.
Unfortunately, as shown in table 6, this is not the case. The IDs perform
moderately well on Twitter15, and approach state-of-the-art on Twitter16. The
non-rumor class on Twitter15 is also trivial to detect from the date.

Table 4. Evaluating tweet ID classification (Macro F1 score) Twitter15 | False | True | Unverified | Non-rumor | Macro Avg.
---|---|---|---|---|---
SOTA (Huang et al., 2020c) | 92.9 | 90.5 | 85.4 | 95.3 | 91.0
2-digit RF | 62.4 | 65.6 | 61.1 | 99.4 | 72.1
3-digit RF | 73.0 | 69.5 | 79.7 | 98.2 | 80.1
Twitter16 | False | True | Unverified | Non-rumor | Macro Avg.
SOTA (Huang et al., 2020c) | 91.3 | 94.7 | 89.9 | 93.5 | 92.4
2-digit RF | 83.5 | 87.6 | 82.1 | 90.7 | 86.0
3-digit RF | 90.7 | 95.3 | 84.4 | 92.9 | 90.8

Of course, classification using tweet IDs alone is uninteresting in isolation.
However, this paints a broader picture of potential confounding variables
within these datasets (and others collected in the same manner). Tweet IDs are
not the only things that evolve over time; the content posted to Twitter
itself changes drastically over time, as language evolves and topics enter and
exit public discourse. For example, consider two particularly time-sensitive
words: ”Clinton” and ”Trump”, the surnames of the 2016 US presidential
candidates. The presence of these words, like the tweet IDs, is revealing
about the time at which a tweet was posted (and therefore the potential label
distribution), as public discourse largely focused on their campaigns
throughout 2016. In table 6, we see that the presence of those words (uncased)
alone allow us to rule out ”true,” and in the case of Twitter15 also ”false.”
This is obviously very unrealistic, and can cause a classifier to rely too
much on particular keywords and fail to generalize to unseen data in applied
settings.

Table 5. Label counts of tweets containing ”Clinton” and ”Trump” Twitter15 | Clinton | Trump | Twitter16 | Clinton | Trump
---|---|---|---|---|---
True | 0 | 0 | True | 0 | 0
False | 0 | 0 | False | 17 | 18
Unverified | 22 | 30 | Unverified | 17 | 39
Non-rumor | 6 | 14 | Non-rumor | 8 | 6

After discovering this problem with Twitter15 and Twitter16, we next examined
the other datasets. Results are shown in table 4. Here Top Model refers to the
best model reported in this paper, either the SOTA model (indicated by a
citation) or one of our language models. PHEME9 All is comprised of all PHEME
tweets with the full 4-way labels. We did not compare results on this split,
so we do not list a top model entry in that column. Random refers to a
stratified random classifier, i.e. outputting a class randomly with
probability according to the training label distribution.

Table 6. Evaluating tweet ID classification (F1 score) | PolitiFact | GossipCop | PHEME9 All | CoAID | WNUT-2020
---|---|---|---|---|---
Top Model | 92.8 (Shu et al., 2019a) | 91.0 | - | 82.2 | 91.6
Random | 50.1 | 49.9 | 26.8 | 35.1 | 51.4
2-digit RF | 77.3 | 65.8 | 25.6 | 49.5 | 34.6
3-digit RF | 76.2 | 66.2 | 43.5 | 46.2 | 51.4

The issue is not as pronounced on these datasets as on Twitter15 and
Twitter16, but is still present, especially in PolitiFact and GossipCop. CoAID
and PHEME are not too bad; classifying the IDs gives better than random
performance, but the performance is still bad. WNUT-2020 is split particularly
well in time, as classifying the IDs is no better than random.

To examine the data from another perspective, we can look at words by label in
Twitter16. In figure 1, we show an example visualization of the label ”false”
versus the rest. We can see that ”steve” is extremely correlated with a label
of ”false.” This is due to a false tweet about Steve Jobs being adopted that
is exactly duplicated 12 times, and nearly duplicated an additional 5. In the
real world, it is unlikely that ”Steve” guarantees a false label, but in this
data it does. We can also see that there seem to be some false tweets about
mass shootings - again, unlikely to be so strongly discriminative in the real
world.

Figure 1. False vs. Other Classes

Scatter plot of words and labels

##  4\. Discussion

These datasets fall on a spectrum of required generalization. On one end,
there are ”in-topic” datasets such as WNUT-2020 and the four PHEME splits not
including PHEME5 Lc. These correspond to a real-world problem where one wants
to detect misinformation or other content on a known topic. For example, a
current task is detecting 5G COVID-19 conspiracy tweets (Benchmark, 2020). On
the other end, there are ”cross-domain” datasets, such as PHEME5 Lc, where
although the task is shared, the content of the test examples is otherwise
unrelated to the train examples. A real-world application of this is detecting
misinformation about a new event. In the middle, there are datasets whose
examples share related domains, but have somewhat different topics, such as
PolitiFact.

The language models examined here excel at ”in-topic” classification, often
beating state-of-the-art methods by large margins in the results here. On the
other hand, they perform poorly at cross-domain tasks. For example, they
produce very mediocre performance on PHEME Lc, likely due to overfitting to
the topics of the training set and failing to generalize to a different topic.
State-of-the-art models in this setting, such as (Cheng et al., 2020b), use
other information beyond content to avoid this failure. Meanwhile, in between
the two extreme types of datasets, we see more variable results. For example,
LMs beat state-of-the-art on GossipCop but not PolitiFact.

Real-world problems can fall anywhere along the entire spectrum, and even the
best results on these datasets - whether ours or others’ - still fail on a
significant fraction of detections. So better methods are needed on all
counts. The results here suggest that understanding where one’s task falls on
this spectrum is very helpful. When it is further towards the ”in-topic” side,
recent language models may be very effective, at minimum providing strong
benchmarks, and potentially a foundation on which to build more sophisticated
models.

Comparing new domain-specific models with older language models may complicate
interpretability of the results. The obvious, definitive solution is to always
compare with the latest language models. However, due to time and computation
constraints, this may not always be feasible - for example, even the largest
models we report here are not the largest models available. There are three
alternatives. One is to match splits and rely on other papers that run the
latest LMs. This is may not be viable for new data, but we encourage work on
standard datasets to provide solid benchmarks, and matching splits with those
benchmarks whenever possible to facilitate comparison. As noted in the
introduction, we provide IDs for our exact splits so that our results can be
used in this way.999https://github.com/ComplexData-MILA/misinfo-baselines
Another alternative is to build models which can incorporate the latest
language model, and provide evidence that it improves results compared to
language models alone (and does so better in some way than existing work).
These may be particularly valuable, as they are to some degree future-proofed,
or at least future-adaptive, since they can be updated with newer language
models. Finally, researchers can explore different directions that are
orthogonal to content, and give evidence that they provide information that
cannot be extracted from the content by language models. There is already a
great deal of research that can fall along these lines, such as (Huang et al.,
2020b; Chen and Freire, 2020; Rosenfeld et al., 2020). But it is not always
clear to what degree other features are orthogonal to content, and explicit
analysis in future work could be useful.

Our results on classifying tweet IDs show that depending on the collection
strategy, the tweet date may be surprisingly informative for the labels. This
means the distribution of data may not be a good match for real-world tasks. A
simple random forest on the first few digits of the tweet ID can be used to
detect if this issue is present in the data. We suggest this test be applied
to future datasets to ensure a reasonable level of temporal randomization, or
to find and report that it is not random and thus to facilitate designing
algorithms and interpreting results in light of that. Similar techniques can
be applied to data from other social networks.

It is also possible to time-randomize data after the primary collection. In
this domain, the fake examples are typically the ones that are hard to find,
as in most contexts the majority of tweets are real (Grinberg et al., 2019).
We can time-randomize without retrieving new fake tweets if we can retrieve
additional real tweets (and tweets from the other classes, when working with
more than real/fake) from approximately the same time as the fake ones. Then
we can replace the original real tweets, yielding both fake and real tweets
that are hard to distinguish using time. This may be facilitated by the new
Twitter Academic
API,101010https://developer.twitter.com/en/solutions/academic-research which
gives more historical access than the normal API. Since Twitter16 is commonly
used (Huang et al., 2020c; Islam et al., 2019; Veyseh et al., 2019), has
severe time-determinacy, and does not have too many tweets to deal with,
future work to improve this dataset in particular could be worthwhile, though
the keyword issues with fake tweets might be hard to fix.

##  5\. Conclusion

Overall, our findings here highlight the need to combine both development of
better algorithms and data science. To solve critical real-world problems like
misinformation, we need better understandings of how different models and
datasets compare and interact. Otherwise we risk creating sophisticated models
that are beaten by brute force approaches - applying the biggest LM one can
run - or models that work well on standard datasets but poorly in practice,
e.g. by learning to predict the date or keywords like ”Steve.” The SOTA models
we compare with in this work have more to offer than pure performance (for
example, explainability), and there are other tasks like early detection that
we have not examined here. But we suggest future work can benefit from
adjusting and further considering how we frame these problems. This will help
not only build higher metrics but also real-world solutions.

Besides considerations for improving current lines of research, this paper
suggests two other promising directions for future work:

  * •

Benchmarking standard misinformation datasets and publishing exact splits. We
see, for example on PHEME, that there are many ways researchers have split the
data, and benchmarks are lacking. This leads to difficulty for the research
community in comparing results. Better benchmarks and the ability to compare
exact splits would help mitigate this issue.

  * •

New, thoroughly benchmarked and validated datasets. Although Twitter15 and
Twitter16 have flaws highlighted in this paper, they also have excellent
propagation information, which has likely motivated many of the approaches
that leverage that to use those datasets. Improved datasets should lead to
improved real-world results.

Finally, we hope to integrate these models with work on other features and
data types, and produce thoroughly evaluated models that can combine the best
of both recent language models and misinformation detection domain algorithms.

## References

  * (1)
  * Aggarwal (2019) Piush Aggarwal. 2019\.  Classification approaches to identify informative tweets. In _Proceedings of the Student Research Workshop Associated with RANLP 2019_. 7–15.
  * Ajao et al. (2019) Oluwaseun Ajao, Deepayan Bhowmik, and Shahrzad Zargari. 2019\.  Sentiment aware fake news detection on online social networks. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2507–2511.
  * Babu and Eswari (2020) Yandrapati Prakash Babu and R. Eswari. 2020.  CIA_NITT at WNUT-2020 Task 2: Classification of COVID-19 Tweets Using Pre-trained Language Models.  _ArXiv_ abs/2009.05782 (2020).
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014\.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ (2014).
  * Benamira et al. (2019) Adrien Benamira, Benjamin Devillers, Etienne Lesot, Ayush K Ray, Manal Saadi, and Fragkiskos D Malliaros. 2019.  Semi-supervised learning and graph neural networks for fake news detection. In _2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_. IEEE, 568–569.
  * Benchmark (2020) MediaEval Multimedia Benchmark. 2020\.  FakeNews: Corona virus and 5G conspiracy.  https://multimediaeval.github.io/editions/2020/tasks/fakenews/.  Accessed: 2020-10-20.
  * Bondielli and Marcelloni (2019) Alessandro Bondielli and Francesco Marcelloni. 2019.  A survey on fake news and rumour detection techniques.  _Information Sciences_ 497 (2019), 38–55.
  * Brainard (2020) Jeffrey Brainard. 2020\.  Scientists are drowning in COVID-19 papers. Can new tools keep them afloat.  _Science_ (2020).
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ (2020).
  * Cao et al. (2018) Juan Cao, Junbo Guo, Xirong Li, Zhiwei Jin, Han Guo, and Jintao Li. 2018\.  Automatic Rumor Detection on Microblogs: A Survey.  _CoRR_ abs/1807.03505 (2018).  arXiv:1807.03505 http://arxiv.org/abs/1807.03505
  * Castillo et al. (2011) Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011\.  Information credibility on twitter. In _Proceedings of the 20th international conference on World wide web_. 675–684.
  * Chauhan (2020) Kumud Chauhan. 2020\.  NEU at WNUT-2020 Task 2: Data Augmentation To Tell BERT That Death Is Not Necessarily Informative.  _ArXiv_ abs/2009.08590 (2020).
  * Chen et al. (2020) Sijing Chen, Jin Mao, Gang Li, Chao Ma, and Yujie Cao. 2020.  Uncovering sentiment and retweet patterns of disaster-related tweets from a spatiotemporal perspective–A case study of Hurricane Harvey.  _Telematics and Informatics_ 47 (2020), 101326.
  * Chen and Freire (2020) Zhouhan Chen and Juliana Freire. 2020.  Proactive Discovery of Fake News Domains from Real-Time Social Media Feeds. In _Companion of The 2020 Web Conference 2020, Taipei, Taiwan, April 20-24, 2020_ , Amal El Fallah Seghrouchni, Gita Sukthankar, Tie-Yan Liu, and Maarten van Steen (Eds.). ACM / IW3C2, 584–592.  https://doi.org/10.1145/3366424.3385772
  * Cheng et al. (2020a) Mingxi Cheng, Shahin Nazarian, and Paul Bogdan. 2020a.  VRoC: Variational Autoencoder-aided Multi-task Rumor Classifier Based on Text. In _WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020_ , Yennun Huang, Irwin King, Tie-Yan Liu, and Maarten van Steen (Eds.). ACM / IW3C2, 2892–2898.  https://doi.org/10.1145/3366423.3380054
  * Cheng et al. (2020b) Mingxi Cheng, Shahin Nazarian, and Paul Bogdan. 2020b.  VRoC: Variational Autoencoder-aided Multi-task Rumor Classifier Based on Text. In _Proceedings of The Web Conference 2020_. 2892–2898.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ (2014).
  * Cinelli et al. (2020) Matteo Cinelli, Walter Quattrociocchi, Alessandro Galeazzi, Carlo Michele Valensise, Emanuele Brugnoli, Ana Lucia Schmidt, Paola Zola, Fabiana Zollo, and Antonio Scala. 2020\.  The covid-19 social media infodemic.  _arXiv preprint arXiv:2003.05004_ (2020).
  * Cui and Lee (2020) Limeng Cui and Dongwon Lee. 2020.  CoAID: COVID-19 Healthcare Misinformation Dataset.  arXiv:2006.00885 [cs.SI]
  * Cui et al. (2020) Limeng Cui, Haeseung Seo, Maryam Tabar, Fenglong Ma, Suhang Wang, and Dongwon Lee. 2020\.  DETERRENT: Knowledge Guided Graph Attention Network for Detecting Healthcare Misinformation. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 492–502.
  * Cui et al. (2019) Limeng Cui, Suhang Wang, and Dongwon Lee. 2019.  SAME: sentiment-aware multi-modal embedding for detecting fake news. In _ASONAM ’19: International Conference on Advances in Social Networks Analysis and Mining, Vancouver, British Columbia, Canada, 27-30 August, 2019_ , Francesca Spezzano, Wei Chen, and Xiaokui Xiao (Eds.). ACM, 41–48.  https://doi.org/10.1145/3341161.3342894
  * Dai et al. (2020) Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V. Le. 2020\.  Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing.  arXiv:2006.03236 [cs.LG]
  * Denaux and Gomez-Perez (2020) Ronald Denaux and Jose Manuel Gomez-Perez. 2020.  Linked Credibility Reviews for Explainable Misinformation Detection.  _arXiv preprint arXiv:2008.12742_ (2020).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _CoRR_ abs/1810.04805 (2018).  arXiv:1810.04805 http://arxiv.org/abs/1810.04805
  * Gardner et al. (2017) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017.  AllenNLP: A Deep Semantic Natural Language Processing Platform.  arXiv:arXiv:1803.07640
  * Giorgi et al. (2020) John M Giorgi, Osvald Nitski, Gary D. Bader, and Bo Wang. 2020\.  DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations.  _ArXiv_ abs/2006.03659 (2020).
  * Grinberg et al. (2019) Nir Grinberg, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson, and David Lazer. 2019\.  Fake news on Twitter during the 2016 U.S. presidential election.  _Science_ 363, 6425 (2019), 374–378.  https://doi.org/10.1126/science.aau2706 arXiv:https://science.sciencemag.org/content/363/6425/374.full.pdf
  * Han et al. (2020) Yi Han, Shanika Karunasekera, and Christopher Leckie. 2020\.  Graph Neural Networks with Continual Learning for Fake News Detection from Social Media.  _arXiv preprint arXiv:2007.03316_ (2020).
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Huang et al. (2020c) Qi Huang, Junshuai Yu, Jia Wu, and Bin Wang. 2020c.  Heterogeneous Graph Attention Networks for Early Detection of Rumors on Twitter.  _arXiv preprint arXiv:2006.05866_ (2020).
  * Huang et al. (2020a) Yen-Hao Huang, Ting-Wei Liu, Ssu-Rui Lee, Fernando Henrique Calderon Alvarado, and Yi-Shin Chen. 2020a.  Conquering Cross-source Failure for News Credibility: Learning Generalizable Representations beyond Content Embedding. In _WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020_ , Yennun Huang, Irwin King, Tie-Yan Liu, and Maarten van Steen (Eds.). ACM / IW3C2, 774–784.  https://doi.org/10.1145/3366423.3380158
  * Huang et al. (2020b) Yen-Hao Huang, Ting-Wei Liu, Ssu-Rui Lee, Fernando Henrique Calderon Alvarado, and Yi-Shin Chen. 2020b.  Conquering Cross-source Failure for News Credibility: Learning Generalizable Representations beyond Content Embedding. In _Proceedings of The Web Conference 2020_. 774–784.
  * Huynh et al. (2020) Tin Van Huynh, L. Nguyen, and Son T. Luu. 2020.  BANANA at WNUT-2020 Task 2: Identifying COVID-19 Information on Twitter by Combining Deep Learning and Transfer Learning Models.  _ArXiv_ abs/2009.02671 (2020).
  * Imran et al. (2016) Muhammad Imran, Prasenjit Mitra, and Carlos Castillo. 2016\.  Twitter as a Lifeline: Human-annotated Twitter Corpora for NLP of Crisis-related Messages. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_ (Portoroz, Slovenia, 23-28). European Language Resources Association (ELRA), Paris, France.
  * Islam et al. (2019) Mohammad Raihanul Islam, Sathappan Muthiah, and Naren Ramakrishnan. 2019.  RumorSleuth: joint detection of rumor veracity and user stance. In _ASONAM ’19: International Conference on Advances in Social Networks Analysis and Mining, Vancouver, British Columbia, Canada, 27-30 August, 2019_ , Francesca Spezzano, Wei Chen, and Xiaokui Xiao (Eds.). ACM, 131–136.  https://doi.org/10.1145/3341161.3342916
  * Ito et al. (2015) Jun Ito, Jing Song, Hiroyuki Toda, Yoshimasa Koike, and Satoshi Oyama. 2015.  Assessment of tweet credibility with LDA features. In _Proceedings of the 24th International Conference on World Wide Web_. 953–958.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016.  Bag of tricks for efficient text classification.  _arXiv preprint arXiv:1607.01759_ (2016).
  * Kankanamge et al. (2020) Nayomi Kankanamge, Tan Yigitcanlar, Ashantha Goonetilleke, and Md Kamruzzaman. 2020.  Determining disaster severity through social media analysis: Testing the methodology with South East Queensland Flood tweets.  _International journal of disaster risk reduction_ 42 (2020), 101360\.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  _arXiv preprint arXiv:1408.5882_ (2014).
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ (2013).
  * Kochkina et al. (2018) Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018\.  All-in-one: Multi-task learning for rumour verification.  _arXiv preprint arXiv:1806.03713_ (2018).
  * Kumar and Singh (2020) Priyanshu Kumar and Aadarsh Singh. 2020.  NutCracker at WNUT-2020 Task 2: Robustly Identifying Informative COVID-19 Tweets using Ensembling and Adversarial Training.  _arXiv preprint arXiv:2010.04335_ (2020).
  * Kumar et al. (2011) Shamanth Kumar, Geoffrey Barbier, Mohammad Ali Abbasi, and Huan Liu. 2011.  Tweettracker: An analysis tool for humanitarian and disaster relief.  _ICwSM_ 11 (2011), 78–82.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020\.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  _ArXiv_ abs/1909.11942 (2020).
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents. In _International conference on machine learning_. 1188–1196.
  * Li et al. (2020) L. Li, Q. Zhang, X. Wang, J. Zhang, T. Wang, T. Gao, W. Duan, K. K. Tsoi, and F. Wang. 2020\.  Characterizing the Propagation of Situational Information in Social Media During COVID-19 Epidemic: A Case Study on Weibo.  _IEEE Transactions on Computational Social Systems_ 7, 2 (2020), 556–562.
  * Liu et al. (2015) Xiaomo Liu, Armineh Nourbakhsh, Quanzhi Li, Rui Fang, and Sameena Shah. 2015.  Real-time rumor debunking on twitter. In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_. 1867–1870.
  * Liu et al. (2019) Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _ArXiv_ abs/1907.11692 (2019).
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled Weight Decay Regularization.  arXiv:1711.05101 [cs.LG]
  * Lu and Li (2020) Yi-Ju Lu and Cheng-Te Li. 2020.  GCAN: Graph-aware co-attention networks for explainable fake news detection on social media.  _arXiv preprint arXiv:2004.11648_ (2020).
  * Ma et al. (2016) Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016.  Detecting rumors from microblogs with recurrent neural networks.  (2016).
  * Ma et al. (2017) Jing Ma, Wei Gao, and Kam-Fai Wong. 2017.  Detect rumors in microblog posts using propagation structure via kernel learning. Association for Computational Linguistics.
  * Maveli (2020) Nickil Maveli. 2020\.  EdinburghNLP at WNUT-2020 Task 2: Leveraging Transformers with Generalized Augmentation for Identifying Informativeness in COVID-19 Tweets.  _ArXiv_ abs/2009.06375 (2020).
  * Meel and Vishwakarma (2020) Priyanka Meel and Dinesh Kumar Vishwakarma. 2020.  Fake news, rumor, information pollution in social media and web: A contemporary survey of state-of-the-arts, challenges and opportunities.  _Expert Systems with Applications_ 153 (2020), 112986.  https://doi.org/10.1016/j.eswa.2019.112986
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ (2013).
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b.  Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_. 3111–3119.
  * Minaee et al. (2020) Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2020.  Deep learning based text classification: A comprehensive review.  _arXiv preprint arXiv:2004.03705_ (2020).
  * Møller et al. (2020) Anders Giovanni Møller, Rob Van Der Goot, and Barbara Plank. 2020\.  NLP North at WNUT-2020 Task 2: Pre-training versus Ensembling for Detection of Informative COVID-19 English Tweets. In _Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)_. 331–336.
  * Mosinzova et al. (2019) Valeryia Mosinzova, Benjamin Fabian, Tatiana Ermakova, and Annika Baumann. 2019.  Fake News, Conspiracies and Myth Debunking in Social Media-A Literature Survey Across Disciplines.  _Conspiracies and Myth Debunking in Social Media-A Literature Survey Across Disciplines (February 3, 2019)_ (2019).
  * Müller et al. (2020) Martin Müller, Marcel Salathé, and Per E Kummervold. 2020\.  COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter.  _arXiv preprint_ arXiv:2005.07503 (2020).
  * Nguyen (2020) A. Nguyen. 2020\.  TATL at W-NUT 2020 Task 2: A Transformer-based Baseline System for Identification of Informative COVID-19 English Tweets.  _ArXiv_ abs/2008.12854 (2020).
  * Nguyen et al. (2020a) Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020a.  BERTweet: A pre-trained language model for English Tweets. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_.
  * Nguyen et al. (2020b) Dat Quoc Nguyen, Thanh Vu, Afshin Rahimi, Mai Hoang Dao, Linh The Nguyen, and Long Doan. 2020b.  WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. In _Proceedings of the 6th Workshop on Noisy User-generated Text_.
  * Paszke et al. (2017) Adam Paszke, S. Gross, Soumith Chintala, G. Chanan, E. Yang, Zachary Devito, Zeming Lin, Alban Desmaison, L. Antiga, and A. Lerer. 2017\.  Automatic differentiation in PyTorch.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12 (2011), 2825–2830.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  GloVe: Global Vectors for Word Representation. In _Empirical Methods in Natural Language Processing (EMNLP)_. 1532–1543.  http://www.aclweb.org/anthology/D14-1162
  * Perrio and Madabushi (2020) Calum Perrio and Harish Tayyar Madabushi. 2020.  CXP949 at WNUT-2020 Task 2: Extracting Informative COVID-19 Tweets – RoBERTa Ensembles and The Continued Relevance of Handcrafted Features.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations. In _Proc. of NAACL_.
  * Pulido et al. (2020) Cristina M Pulido, Beatriz Villarejo-Carballido, Gisela Redondo-Sama, and Aitor Gómez. 2020.  COVID-19 infodemic: More retweets for science-based information on coronavirus than for false information.  _International Sociology_ (2020), 0268580920914755.
  * Qin et al. (2020) Lei Qin, Qiang Sun, Yidan Wang, Ke-Fei Wu, Mingchih Chen, Ben-Chang Shia, and Szu-Yuan Wu. 2020.  Prediction of Number of Cases of 2019 Novel Coronavirus (COVID-19) Using Social Media Search Index.  _International Journal of Environmental Research and Public Health_ 17, 7 (Mar 2020), 2365.  https://doi.org/10.3390/ijerph17072365
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language Models are Unsupervised Multitask Learners.  (2019).
  * Rapoza (2017) Kenneth Rapoza. 2017\.  Can ‘fake news’ impact the stock market?  _by Forbes_ (2017).
  * Rosenfeld et al. (2020) Nir Rosenfeld, Aron Szanto, and David C. Parkes. 2020\.  A Kernel of Truth: Determining Rumor Veracity on Twitter by Diffusion Pattern Alone. In _WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020_ , Yennun Huang, Irwin King, Tie-Yan Liu, and Maarten van Steen (Eds.). ACM / IW3C2, 1018–1028.  https://doi.org/10.1145/3366423.3380180
  * Rubin et al. (2016) Victoria L Rubin, Niall Conroy, Yimin Chen, and Sarah Cornwell. 2016.  Fake news or truth? using satirical cues to detect potentially misleading news. In _Proceedings of the second workshop on computational approaches to deception detection_. 7–17.
  * Scarselli et al. (2008) Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008\.  The graph neural network model.  _IEEE Transactions on Neural Networks_ 20, 1 (2008), 61–80.
  * Shu et al. (2019a) Kai Shu, Limeng Cui, Suhang Wang, Dongwon Lee, and Huan Liu. 2019a.  defend: Explainable fake news detection. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 395–405.
  * Shu et al. (2019b) Kai Shu, Limeng Cui, Suhang Wang, Dongwon Lee, and Huan Liu. 2019b.  dEFEND: Explainable Fake News Detection. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_, Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (Eds.). ACM, 395–405.  https://doi.org/10.1145/3292500.3330935
  * Shu et al. (2018) Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2018.  FakeNewsNet: A Data Repository with News Content, Social Context and Dynamic Information for Studying Fake News on Social Media.  _arXiv preprint arXiv:1809.01286_ (2018).
  * Shu et al. (2017a) Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017a.  Fake News Detection on Social Media: A Data Mining Perspective.  _SIGKDD Explorations_ 19, 1 (2017), 22–36.  https://doi.org/10.1145/3137597.3137600
  * Shu et al. (2017b) Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017b.  Fake news detection on social media: A data mining perspective.  _ACM SIGKDD Explorations Newsletter_ 19, 1 (2017), 22–36.
  * Shu et al. (2017c) Kai Shu, Suhang Wang, and Huan Liu. 2017c.  Exploiting Tri-Relationship for Fake News Detection.  _arXiv preprint arXiv:1712.07709_ (2017).
  * Shu et al. (2020) Kai Shu, Guoqing Zheng, Yichuan Li, Subhabrata Mukherjee, Ahmed Hassan Awadallah, Scott Ruston, and Huan Liu. 2020.  Leveraging Multi-Source Weak Social Supervision for Early Detection of Fake News.  _arXiv preprint arXiv:2004.01732_ (2020).
  * Singh et al. (2019) Jyoti Prakash Singh, Yogesh K Dwivedi, Nripendra P Rana, Abhinav Kumar, and Kawaljeet Kaur Kapoor. 2019.  Event classification and location prediction from tweets during disasters.  _Annals of Operations Research_ 283, 1 (2019), 737–757.
  * Takahashi et al. (2015) Bruno Takahashi, Edson C Tandoc Jr, and Christine Carmichael. 2015\.  Communicating on Twitter during a disaster: An analysis of tweets during Typhoon Haiyan in the Philippines.  _Computers in human behavior_ 50 (2015), 392–398.
  * To et al. (2017) Hien To, Sumeet Agrawal, Seon Ho Kim, and Cyrus Shahabi. 2017\.  On identifying disaster-related tweets: Matching-based or learning-based?. In _2017 IEEE Third International Conference on Multimedia Big Data (BigMM)_. IEEE, 330–337.
  * Tran et al. (2020) Khiem Vinh Tran, Hao Phu Phan, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen. 2020.  UIT-HSE at WNUT-2020 Task 2: Exploiting CT-BERT for Identifying COVID-19 Information on the Twitter Social Network.  _arXiv preprint arXiv:2009.02935_ (2020).
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Well-Read Students Learn Better: On the Importance of Pre-training Compact Models.  _arXiv preprint arXiv:1908.08962v2_ (2019).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Veyseh et al. (2019) Amir Pouran Ben Veyseh, My T Thai, Thien Huu Nguyen, and Dejing Dou. 2019.  Rumor detection in social networks via deep contextual modeling. In _Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining_. 113–120.
  * Wadhawan (2020) Anshul Wadhawan. 2020\.  Phonemer at WNUT-2020 Task 2: Sequence Classification Using COVID Twitter BERT and Bagging Ensemble Technique based on Plurality Voting.  _arXiv preprint arXiv:2010.00294_ (2020).
  * Wang et al. (2020) Youze Wang, Shengsheng Qian, Jun Hu, Quan Fang, and Changsheng Xu. 2020.  Fake News Detection via Knowledge-driven Multimodal Graph Convolutional Networks. In _Proceedings of the 2020 International Conference on Multimedia Retrieval_. 540–547.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  _ArXiv_ abs/1910.03771 (2019).
  * Wu and Rao (2020) Lianwei Wu and Yuan Rao. 2020.  Adaptive Interaction Fusion Networks for Fake News Detection.  _arXiv preprint arXiv:2004.10009_ (2020).
  * Wu et al. (2019) Lianwei Wu, Yuan Rao, Haolin Jin, Ambreen Nazir, and Ling Sun. 2019.  Different absorption from the same sharing: Sifted multi-task learning for fake news detection.  _arXiv preprint arXiv:1909.01720_ (2019).
  * Wu et al. (2020b) Lianwei Wu, Yuan Rao, Yongqiang Zhao, Hao Liang, and Ambreen Nazir. 2020b.  DTCA: Decision tree-based co-attention networks for explainable claim verification.  _arXiv preprint arXiv:2004.13455_ (2020).
  * Wu et al. (2020a) Zhiyuan Wu, Dechang Pi, Junfu Chen, Meng Xie, and Jianjun Cao. 2020a.  Rumor Detection Based On Propagation Graph Neural Network With Attention Mechanism.  _Expert Systems with Applications_ (2020), 113595.
  * Xia et al. (2020) Patrick Xia, Shijie Wu, and Benjamin Van Durme. 2020.  Which* BERT? A Survey Organizing Contextualized Encoders.  _arXiv preprint arXiv:2010.00854_ (2020).
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification. In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies_. 1480–1489.
  * Zhou and Zafarani (2018) Xinyi Zhou and Reza Zafarani. 2018.  Fake news: A survey of research, detection methods, and opportunities.  _arXiv preprint arXiv:1812.00315_ (2018).
  * Zhou et al. (2019) Xinyi Zhou, Reza Zafarani, Kai Shu, and Huan Liu. 2019\.  Fake news: Fundamental theories, detection strategies and challenges. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_. 836–837.
  * Zubiaga et al. (2016) Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. 2016\.  Analysing how people orient to and spread rumours in social media by looking at conversational threads.  _PloS one_ 11, 3 (2016), e0150989.


Results on $D^{+}$ $\to\ell^{+}\nu$ and $D^{+}_{s}$ $\to\ell^{+}\nu$ decays at
Charm factories

L.L. Jiang
Institute of High Energy Physics
No.918 P.O.X,Beijing,China

Proceedings of CKM 2012, the 7th International Workshop on the CKM Unitarity
Triangle, University of Cincinnati, USA, 28 September - 2 October 2012

##  1 Introduction

In the Standard Model (SM) of particle physics, $D^{+}_{(s)}$111throughout
this manuscript, charge conjugation is implied mesons can decay into
$\ell^{+}\nu$ (where $\ell^{+}$ is $e$, $\mu$ or $\tau$) through a virtual
$W^{+}$ boson. The virtual $W^{+}$ boson is produced in the annihilation of
the $c$ and $\overline{d}$($\overline{s}$) quarks. The decay rate of this
process is determined by the wave-function overlap of the two quarks at the
origin, and is parameterized by the $D^{+}_{(S)}$ decay constant,
$f_{D^{+}_{(S)}}$. Figure 1 shows the decay diagram for the Cabibbo-suppressed
purely leptonic decay of the $D^{+}_{(S)}$ meson. In this decay process, all
strong interaction effects between the two quarks in initial state are
absorbed into $f_{D^{+}_{(S)}}$.

$D^{+}_{(S)}$$\ell^{+}$$\nu$$c$$\overline{d}(\overline{s})$$W^{+}$

Figure 1: The decay diagram for $D^{+}_{(S)}\rightarrow\ell^{+}\nu$.

The decay width of $D^{+}_{(S)}\rightarrow\ell^{+}\nu$ is given by the formula
[1]

$\Gamma(D^{+}_{(S)}\rightarrow\ell^{+}\nu)=\frac{G^{2}_{F}f^{2}_{D^{+}_{(S)}}}{8\pi}\mid
V_{cd(s)}\mid^{2}m^{2}_{\ell}m_{D^{+}_{(S)}}\left(1-\frac{m^{2}_{\ell}}{m^{2}_{D^{+}_{(S)}}}\right)^{2},$
(1)

where $G_{F}$ is the Fermi coupling constant, $V_{cd(s)}$ is the
$c\rightarrow\overline{d}(\overline{s})$ Cabibbo-Kobayashi-Maskawa (CKM)
matrix element [2], $m_{\ell}$ is the mass of the lepton, and
$m_{D^{+}_{(S)}}$ is the mass of the $D^{+}_{(S)}$ meson. The ratios of decay
rates for $D^{+}\rightarrow\tau^{+}\nu_{\tau}$,
$D^{+}\rightarrow\mu^{+}\nu_{\mu}$ and $D^{+}\rightarrow e^{+}\nu_{e}$ are
expected by the well-known lepton-masses, which are

$\displaystyle R$ $\displaystyle=$
$\displaystyle\Gamma(D^{+}\rightarrow\tau^{+}\nu_{\tau}):\Gamma(D^{+}\rightarrow\mu^{+}\nu_{\mu}):\Gamma(D^{+}\rightarrow
e^{+}\nu_{e})$ (2) $\displaystyle=$ $\displaystyle 2.67:1:2.35\times 10^{-5}.$

while the ratios of decay rates for $D^{+}_{s}\rightarrow\tau^{+}\nu_{\tau}$,
$D^{+}_{s}\rightarrow\mu^{+}\nu_{\mu}$ and $D^{+}_{s}\rightarrow e^{+}\nu_{e}$
are expected to be 9.8:1:2.4$\times 10^{-5}$. Any significant deviation of
experimental results from theory expectation may indicate that there are New
Physics (NP).

The pseudoscalar decay constants $f_{D^{+}_{(s)}}$ are very important
constants in heavy flavor physics. Within the context of the SM, measurement
of the purely leptonic decay of the $D^{+}_{(s)}$ meson provides a means of
determining $f_{D^{+}_{(s)}}$. As the decay constant is related to the
annihilation probability of the heavy and the light quarks inside the meson,
they play an important role both in characterizing the properties of
confinement and as absolute normalization of numerous heavy-flavor
transitions, including semileptonic decays and non-leptonic decays of the
mesons as well as mixing of neutral and anti-neutral meson pairs. For example,
$f_{B}$ relates the measurements of the $B\overline{B}$ mixing ratio to CKM
matrix elements. At present it is impossible to precisely determine
$f_{B^{+}}$ experimentally from the purely leptonic $B^{+}$ decay and is never
possible to measure $f_{B^{0}_{s}}$ since $B^{0}_{s}$ does not have charge
current leptonic decay, so theoretical calculations of the $f_{B^{+}}$
and$f_{B^{0}_{s}}$ have to be used. The ratios of $f_{D^{+}}/f_{B^{+}}$ and
$f_{D_{s}^{+}}/f_{B^{0}_{s}}$ from LQCD calculations are determined with
higher precision than the calculations of each of the decay constants. The
LQCD calculations of $f_{B}$ can be used with some level of confidence given
by the stringent tests from leptonic D decays on $f_{D^{+}}$ and
$f_{D_{s}^{+}}$.

Conversely, taking the theoretical calculations of the decay constants as
input, one can also determine the CKM matrix elements $|V_{\rm cd}|$ and
$|V_{\rm cs}|$ by analyzing the $D^{+}$ and $D^{+}_{s}$ purely leptonic
decays. Thanks to the unquenched LQCD calculations of $f_{D^{+}}$, which have
reached a high precision of 2%, one can get more precisely experimental result
of $|V_{\rm cd}|$ than the one historically measured based on the
$D\to\pi\ell^{+}\nu$ semileptonic decays.

##  2 Methods

The $D$ and $\overline{D}$ meson are produced in pairs near open-charm meson
pair production energy thresholds in $e^{+}e^{-}$ experiments, which provides
the cleanest experimental environment for studies of the leptonic decays of
$D^{+}_{(s)}$ meson. Taking advantage of the $D\overline{D}$ production, if a
$D$ meson decay is fully reconstructed (this is called singly tagged D meson),
a $\overline{D}$ must exist in the system recoiling against the tagged $D$
meson. Based on the accumulated singly tagged $D$ meson sample, one can
measure the absolute branching fraction of the leptonic decays of
$D^{+}_{(s)}$. The neutrino in the final states of leptonic decays of
$D^{+}_{(s)}$ can be reconstructed with the missing momentum and the missing
energy of the events. If there is a neutrino in the recoil side of the singly
tagged $D$ meson, the distribution of the missing mass squared, which is the
missing energy square minus the missing momentum square, should characterize
with a peak at zero.

##  3 Leptonic decays of $D^{+}$ meson

Historically MARK-III, BES-I, BES-II and CLEO-c experiments made measurements
of $f_{D^{+}_{(s)}}$. Today’s running BES-III experiment focus on measurement
of $f_{D^{+}}$.

###  3.1 Results at previous experiments

####  3.1.1 Search for $D^{+}\rightarrow l^{+}\nu$ decay at Mark-III
experiment

In 1988, MARK III collaboration first searched for the decay of
$D^{+}\rightarrow\ell^{+}\nu$ and got no signal event. They set an upper limit
on the decay constant at $90\%$ C.L. to be $f_{D^{+}}<290$ MeV. [3].

####  3.1.2 First observed one candidate for $D^{+}\rightarrow\ell^{+}\nu$ at
the BES-I experiments

In 1998, the BES collaboration observed one candidate event for
$D^{+}\rightarrow\ell^{+}\nu$ in the recoil side of the 10082 singly tagged
$D$ mesons, by analyzing 22.3 pb-1 of data taken at 4.03 GeV. From this
experiment, they measured the branching fraction for
$D^{+}\rightarrow\mu^{+}\nu$ to be $(0.08^{+0.16+0.05}_{-0.05-0.02})\%$,
corresponding to a value of decay constant of
$f_{D^{+}}=(300^{+180+80}_{-150-40})$ MeV [4].

####  3.1.3 First absolute measurements of $B(D^{+}\rightarrow l^{+}\nu)$ and
$f_{D^{+}}$ at the BES-II experiments

At the “04 Electroweak Interactions $\&$ Unified Theories” international
meeting, the BES-II collaboration reported their analysis results of the
leptonic decay of $D^{+}$ for the first time. With 33 pb-1 of data taken in
$e^{+}e^{-}$ annihilation with their upgraded BES-II detector at the BEPC
collider, they accumulated $5321\pm 149\pm 160$ $D^{-}$ single tags and
observed 3 candidates for $D^{+}\rightarrow\mu^{+}\nu$ decays in the recoil
side of the single tags. With the 3 candidates, which contain 0.3 background
events estimated from MC simulation and the accumulated $5321\pm 149\pm 160$
$D^{-}$ single tags, they measured the branching fraction for
$D^{+}\rightarrow\mu^{+}\nu$ decays to be
$B(D^{+}\rightarrow\mu^{+}\nu)=(0.122^{+0.111}_{-0.053}\pm 0.010)\%$,
corresponding to a value of the decay constant
$f_{D^{+}}=(371^{+129}_{-119}\pm 25)$ MeV [5]. Their analysis results were
finally published in 2005. These are absolute measurements of the decay
branching fraction and decay constant, which do not depend on the yield of
$D^{+}$ meson production and do not depend on some branching fractions for
$D^{+}$ meson decay into other modes.

####  3.1.4 Precise measurements of $B(D^{+}\rightarrow l^{+}\nu)$ and
$f_{D^{+}}$ at CLEO-c experiments

CLEO-c published 3 paper to report their studies of the leptonic decay of
$D^{+}$ mesons in 2004 [6], 2005 [7] and 2008 [8], respectively. Because the
previous results are superseded by the most recent one, we here only discuss
the results published in 2008. In that paper, the CLEO collaboration analyzed
818 pb-1 of data taken at 3.773 GeV. They accumulated $460055\pm 787$ $D^{-}$
tags with 6 hadronic decay modes of the $D^{-}$ meson and observed $149.7\pm
12.0$ signal events for $D^{+}\rightarrow\mu^{+}\nu$. In the abstract and the
summary in that paper, they presented the measured the decay branching
fraction of $B(D^{+}\rightarrow\mu^{+}\nu)=(3.82\pm 0.32\pm 0.09)\times
10^{-4}$ and the corresponding decay constant of $f_{D^{+}}=(205.8\pm 8.5\pm
2.5)$ MeV [8].

###  3.2 New results at BES-III experiment

The BES-III collaboration has collected 2.9 fb-1 of data at 3.773 GeV with
BES-III detector [9] at the BEPC-II [10] during the time period from 2010 to
2011. In this section, we report measurements of the branching fraction for
$D^{+}\rightarrow\mu^{+}\nu_{\mu}$ decay and the pseudoscalar decay constant
$f_{D^{+}}$ obtained by analyzing this data sample.

MB [GeV/$c^{2}$]

Number of Events

(a)(b)(c)(d)(e)(f)(g)(h)(i)

Figure 2: Distributions of the beam energy constraint masses of the $mKn\pi$
combinations for the 9 single tag modes from the data; where (a), (b), (c),
(d), (e), (f), (g), (h), (i) are for the modes of $D^{-}\rightarrow
K^{+}\pi^{-}\pi^{-}$, $D^{-}\rightarrow K^{0}_{s}\pi^{-}$, $D^{-}\rightarrow
K^{0}_{s}K^{-}$, $D^{-}\rightarrow K^{+}K^{-}\pi^{-}$, $D^{-}\rightarrow
K^{+}\pi^{-}\pi^{-}\pi^{0}$, $D^{-}\rightarrow\pi^{+}\pi^{-}\pi^{-}$,
$D^{-}\rightarrow K^{0}_{s}\pi^{-}\pi^{0}$, $D^{-}\rightarrow
K^{+}\pi^{-}\pi^{-}\pi^{-}\pi^{+}$, and $D^{-}\rightarrow
K^{0}_{s}\pi^{-}\pi^{-}\pi^{+}$, respectively.

Nine non-leptonic $D^{-}$ decay modes $K^{+}\pi^{-}\pi^{-}$,
$K^{0}_{s}\pi^{-}$, $K^{0}_{s}K^{-}$, $K^{+}K^{-}\pi^{-}$,
$K^{+}\pi^{-}\pi^{-}\pi^{0}$, $\pi^{+}\pi^{-}\pi^{-}$,
$K^{0}_{s}\pi^{-}\pi^{0}$, $K^{+}\pi^{-}\pi^{-}\pi^{-}\pi^{+}$, and
$K^{0}_{s}\pi^{-}\pi^{-}\pi^{+}$ are used in accumulating the singly tagged
$D^{-}$ mesons. The singly tagged $D^{-}$ mesons are fully reconstructed by
requiring that the energy of the reconstructed candidate $D^{-}$ must be
consistent with the beam energy, and then examine the beam energy constraint
mass of the tagged $mKn\pi$ system,

$M_{\rm B}=\sqrt{E_{\rm beam}^{2}-|\vec{p}_{mKn\pi}|^{2}},$ (3)

where $E_{\rm beam}$ is the beam energy, and $|\vec{p}_{mKn\pi}|$ is the
magnitude of the momentum of the daughter particle $mKn\pi$ system. The
$M_{\rm B}$ distributions and the maximum likelihood fit results for the
$M_{\rm B}$ distributions of the nine $D^{-}$ tag modes are shown in Fig. 2.
The fits give a total of $1586056\pm 2327$ $D^{-}$ tags. After subtracting
20103 double counting $D^{-}$ tags, BES-III reconstructed $1565953\pm 2327$
$D^{-}$ tags which are used for further analysis of measuring the branching
fraction for $D^{+}\rightarrow\mu^{+}\nu_{\mu}$ decays.

Candidate events for the decay $D^{+}\rightarrow\mu^{+}\nu_{\mu}$ are selected
from the surviving charged tracks in the system recoiling against the singly
tagged $D^{-}$ mesons. For the $D^{+}\rightarrow\mu^{+}\nu_{\mu}$ final
states, there should be only one candidate charged track, which is identified
as muon. At BES-III, the $\mu^{+}$ can be well identified with the Muon
detector (MUC). Taking the difference of the passage lengths in MUC for
difference types of the charged tracks, the candidate muon is required to have
large passage length. Except for this, no extra good photon with energy
greater than 300 MeV is allowed to be present.

$M^{2}_{\rm miss}$[GeV2/$c^{4}$]

Momentum p [GeV/$c$]

$M^{2}_{\rm miss}$ [GeV2/$c^{4}$]

p [GeV/$c$]

$M^{2}_{\rm miss}$[GeV2/$c^{4}$]

Number of events

(a)(b)(c)

Figure 3: Distributions of $M^{2}_{\rm miss}$, where (a) and (b) are scatter
plots of the identified muon momentum $p$ VS $M^{2}_{\rm miss}$, and (c) is
the distribution of $M^{2}_{\rm miss}$. The insert shows the signal region for
$D^{+}\rightarrow\mu^{+}\nu_{\mu}$ on a log scale, where dots with error bars
are for the data, histograms are for the simulated backgrounds from
$D^{+}\rightarrow K^{0}_{L}\pi^{+}$ (red), $D^{+}\rightarrow\pi^{0}\pi^{+}$
(green), $D^{+}\rightarrow\tau^{+}\nu_{\tau}$ (blue) and other decays of $D$
mesons (yellow) as well as from $e^{+}e^{-}\rightarrow$non-$D\overline{D}$
decays (pink).

Figures 3(a) and (b) show the scatter-plots of the momentum of the identified
muon satisfying the requirement for selecting
$D^{+}\rightarrow\mu^{+}\nu_{\mu}$ decay versus $M^{2}_{miss}$, where the blue
box in Fig. 3(a) shows the signal region for
$D^{+}\rightarrow\mu^{+}\nu_{\mu}$ decays. Within the signal region, there are
$425$ candidate events for $D^{+}\rightarrow\mu^{+}\nu_{\mu}$ decay. The two
concentrated clusters out side of the signal region are from $D^{+}$ non-
leptonic decays and some other background events. The events whose peak is
around 0.25 GeV2/$c^{4}$ in $M^{2}_{miss}$ are mainly from $D^{+}\rightarrow
K_{L}^{0}\pi^{+}$ decays, where $K_{L}^{0}$ is missing. Projecting the events
for which the identified muon momentum being in the range from 0.8 to 1.1
GeV/$c$ onto the horizontal scale yields the $M^{2}_{miss}$ distribution as
shown in Fig. 3(c). From this plot, we can see that the difficultly suppressed
backgrounds from $D^{+}\rightarrow K_{L}^{0}\pi^{+}$ decays in CLEO-c
measurement [8] are effectively suppressed due to that they use the MUC
measurements to identify the muon. Detailed Monte Carlo studies show that
there are $47.7\pm 2.3\pm 1.3$ background events in $425$ candidates for
$D^{+}\rightarrow\mu^{+}\nu_{\mu}$ decays, where the first error is the MC
statistical error and second is the systematic, which arises from
uncertainties in the branching fractions or production cross sections for the
source modes. After subtracting the number of background events, $377.3\pm
20.6\pm 2.6$ signal events for $D^{+}\rightarrow\mu^{+}\nu_{\mu}$ decay are
retained, where the first error is statistical and the second systematic
arising from the uncertainty of the background estimation.

The overall efficiency for observing the decay
${D^{+}\rightarrow\mu^{+}\nu_{\mu}}$ is obtained by analyzing full MC
simulated events of $D^{+}\rightarrow\mu^{+}\nu_{\mu}$ versus $D^{-}$ tags and
combining with $\mu^{+}$ reconstruction efficiency of the MUC. The $\mu^{+}$
reconstruction efficiency of the MUC is measured with muon samples selected
from the same data taken at 3.773 GeV. The overall efficiency is $0.6382~{}\pm
0.0015$.

With $1565953$ singly tagged $D^{-}$ mesons, $377.3\pm 20.6\pm 2.6$
$D^{+}\rightarrow\mu^{+}\nu_{\mu}$ decay events and with a reconstruction
efficiency of $0.6382~{}\pm 0.0015$, the BES-III collaboration obtain the
branching fraction

$B(D^{+}\to\mu^{+}\nu_{\mu})=(3.74\pm 0.21\pm 0.06)\times 10^{-4}~{}~{}({\rm
BESIII~{}Preliminary}),$

where the first error is statistical and the second systematic. This measured
branching fraction is consistent within error with world average of
$B(D^{+}\to\mu^{+}\nu_{\mu})=(3.82\pm 0.33)\times 10^{-4}$ [2], but with
better precision.

The decay constant $f_{D^{+}}$ can be obtained by inserting the measured
branching fraction, the mass of the muon, the mass of the $D^{+}$ meson, the
CKM matrix element $|V_{\rm cd}|=0.2252\pm 0.0007$ from the CKMFitter [2],
$G_{F}$ and the lifetime of the $D^{+}$ meson [2] Eq.(2), which yields

$f_{D^{+}}=(203.91\pm 5.72\pm 1.97)~{}~{}\rm MeV~{}~{}({\rm
BESIII~{}Preliminary}),$

where the first error is statistical and the second is systematic, which
arises mainly from the uncertainties in the measured branching fraction
($1.7\%$), the CKM matrix element $|V_{\rm cd}|$ ($0.3\%$), and the lifetime
of the $D^{+}$ meson ($0.7\%$) [2]. The total systematic error is $1.0\%$.

##  4 Leptonic decays of $D_{s}^{+}$ meson

The BES-I and CLEO-c experiments have published results on $f_{D_{s}^{+}}$ and
BES-III will publish their experiment results of the purely leptonic decay of
$D^{+}_{s}$ meson soon.

###  4.1 BES-I experiment near $D_{s}^{+}D_{s}^{-}$ threshold

The first absolute measurements of decay branching factions of
$D_{s}^{+}\rightarrow\mu^{+}\nu$, $D_{s}^{+}\rightarrow\tau^{+}\nu$ and the
decay constant were made by BES-I experiment in 1995. They found 3 events of
both the $D_{s}^{+}\rightarrow\tau^{+}\nu$ and
$D_{s}^{+}\rightarrow\mu^{+}\nu$ decays in the recoil side of $94.3\pm 12.5$
singly tagged $D_{s}^{-}$ mesons by analyzing 22.3 pb-1 of data taken at 4.03
GeV. They measured the decay branching fractions of
$B(D_{s}^{+}\rightarrow\tau^{+}\nu)=(15^{+13+3}_{-6-2})\%$ and
$B(D_{s}^{+}\rightarrow\mu^{+}\nu)=(1.5^{+1.3+0.3}_{-0.6-0.2})\%$,
respectively, and the decay constant of $f_{D_{s}^{+}}=(430^{+150}_{-130}\pm
40)$ MeV [11].

###  4.2 CLEO-c experiment near $D_{s}^{+}D_{s}^{*-}$ threshold

CLEO-c collected 600 pb-1 of data at 4.17 GeV in $e^{+}e^{-}$ annihilation.
With this data sample, they published 3 papers to present the studies of the
purely leptonic decays of $D^{+}_{s}$ by reconstructing the $D^{+}_{s}$ with
various decay modes [12, 13, 14]. By averaging the results obtained by
analyzing different leptonic decay modes of $D^{+}_{s}$ meson, CLEO-c gave
$f_{D_{s}^{+}}=(259.0\pm 6.2\pm 3.0)$ MeV [14].

##  5 Summary

Since the first attempt to search for the $D^{+}$ leptonic decay by the MARK-
III experiment in 1988, many experiments have been making great efforts to
search for and study the $D^{+}$ and $D_{s}^{+}$ leptonic decays. At present,
the most precise measurements of the decay constant of $f_{D^{+}}$ is made by
BES-III, which was first reported at Charm2012 [15]. They measured the
branching fraction $B(D^{+}\rightarrow\mu^{+}\nu)=(3.74\pm 0.21\pm 0.06)\times
10^{-4}$, $f_{D^{+}}=(203.9\pm 5.7\pm 2.0)$ MeV and $|V_{\rm cd}|=(0.222\pm
0.006\pm 0.005)$. In addition, more precise results on
$D^{+}\to\tau^{+}\nu_{\tau}$, $e^{+}\nu_{e}$ will be determined from the full
BES-III data taken at 3.773 GeV in the near future, and results on
$D^{+}_{s}\to\tau^{+}\nu_{\tau}$, $\mu^{+}\nu_{\mu}$ will be reported from the
BES-III recent selected data at 4.010 GeV soon.

## Acknowledgement

I would like to thank the colleagues of BES-III collaboration, the staff of
BEPCII and the computing center of IHEP for their hard efforts and excellent
works. This work is partly supported by National Key Basic Research Program
(973 by MOST) (2009CB825200) and National Natural Science Foundation of China
(NSFC) under Contract No. 10935007.

## References

  * [1] Francis Halzen, Alan D. Martin, Quarks & Leptons (John Wiley & Sons, New York, Chichester, Brisbane, Toronto, Singapore, 1984).
  * [2] K. Nakamura $et~{}al.$ J. Phys. G 37, 075021 (2010).
  * [3] J. Adler $et~{}al.$ (The MARK III Collaboration), Phys. Rev. Lett. 60, 1375 (1998).
  * [4] J.Z. Bai $et~{}al.$ (BES Collaboration), Phys. Lett. B429, 188 (1998).
  * [5] G. Rong (for BES Collaboration), Proceeding of the XXXIXth RENCONTRES DE MORIOND, March 21–28, 2004, edited by J. Tran Thanh Van; M. Ablikim $et~{}al.$ (BES Collaboration), Phys. Lett. B610, 183 (2005).
  * [6] G. Bonvicini $et~{}al.$ (CLEO Collaboration), Phys. Rev. D 70, 112004 (2004).
  * [7] M. Artuso $et~{}al.$ (CLEO Collaboration), Phys. Rev. Lett. 95, 251801 (2005).
  * [8] B.I. Eisenstein $et~{}al.$ (CLEO Collaboration), Phys. Rev. D 78, 052003 (2008).
  * [9] M. Ablikim $et~{}al.$ (BESIII Collaboration), Nucl. Instrum. Methods Phys. Res., Sect. A A 614, 345 (2010)
  * [10] J. Z. Bai $et~{}al.$ (BES Collaboration), Nucl. Instrum. Methods Phys. Res. A 458, 627 (2001).
  * [11] J.Z. Bai $et~{}al.$ (BES Collaboration), Phys. Rev. Lett.74, 4599 (1995).
  * [12] J.P. Alexander $et~{}al.$ (CLEO Collaboration), Phys. Rev. D 79, 052001 (2009).
  * [13] P.U.E. Onyisi $et~{}al.$ (CLEO Collaboration), Phys. Rev. D 79, 052002 (2009).
  * [14] P. Naik $et~{}al.$ (CLEO Collaboration), Phys. Rev. D 80, 112004 (2009).
  * [15] G. Rong (for BES-III Collaboration), arXiv:hep-ph/12090085.