#  ManiTweet: A New Benchmark for Identifying
Manipulation of News on Social Media

Kung-Hsiang Huang1 Hou Pong Chan2 Kathleen McKeown3 Heng Ji1
1University of Illinois Urbana-Champaign 2University of Macau 3Columbia
University
1{khhuang3<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Considerable advancements have been made to tackle the misrepresentation of
information derived from reference articles in the domains of fact-checking
and faithful summarization. However, an unaddressed aspect remains—the
identification of social media posts that manipulate information within
associated news articles. This task presents a significant challenge,
primarily due to the prevalence of personal opinions in such posts. We present
a novel task, identifying manipulation of news on social media, which aims to
detect manipulation in social media posts and identify manipulated or inserted
information. To study this task, we have proposed a data collection schema and
curated a dataset called ManiTweet, consisting of 3.6K pairs of tweets and
corresponding articles. Our analysis demonstrates that this task is highly
challenging, with large language models (LLMs) yielding unsatisfactory
performance. Additionally, we have developed a simple yet effective basic
model that outperforms LLMs significantly on the ManiTweet dataset. Finally,
we have conducted an exploratory analysis of human-written tweets, unveiling
intriguing connections between manipulation and the domain and factuality of
news articles, as well as revealing that manipulated sentences are more likely
to encapsulate the main story or consequences of a news outlet.

Figure 1: Two illustrative examples that highlight the challenge of
identifying manipulation of news on social media. For the first example, while
the associated article does not explicitly discuss the importance of
maintaining good hand hygiene, the tweet does not distort the information
within the article. Conversely, in the second example, a tweet falsely asserts
that COVID-19 is non-contagious, directly contradicting the content of the
reference article. Hence, the second tweet misrepresents the information
contained in the reference article.

##  1 Introduction

Detecting texts that misrepresent information within reference texts is
crucial for combating misinformation. Previous research has primarily tackled
this issue in the context of fact-checking Thorne et al. (2018); Wadden et al.
(2020), where the goal is to debunk unsupported claims using relevant
passages, and in summarization Kryscinski et al. (2020); Fabbri et al. (2022),
where the focus is on assessing the faithfulness of generated summaries to the
reference articles. However, none of the previous work has specifically
addressed the identification of social media posts that manipulate information
within the reference articles. This poses a significant challenge due to the
prevalence of personal opinions in social media posts. To effectively tackle
this problem, models must be able to discern between personal opinions and
sentences that distort information in social media posts. Examples of tweets
that only express personal opinions and tweets that manipulate information can
be found in Figure 1.

In this paper, we introduce a new task called identifying manipulations of
news on social media. To explore this problem, we repurposed news articles
from FakeNewsNet Shu et al. (2020) and constructed a fully-annotated dataset,
ManiTweet, consisting of 3.6K tweets accompanied by their corresponding news
outlets. One main challenge of collecting data for this task is the imbalanced
tweet distributions, with the majority of tweets not manipulating the
associated article. Randomly sampling human-written tweets for annotation will
result in poor annotation cost-efficiency. To ensure efficient annotation, we
propose a two-round annotation scheme. In the first round, human annotators
are assigned the task of validating tweets generated by large language models
(LLMs). The data collected from these rounds is subsequently utilized to train
a sequence-to-sequence model for identifying manipulation within tweets
authored by humans. In the second round of annotation, these human-authored
tweets are labeled accordingly. The 0.5K human-written tweets annotated in the
second round are used as the test set for evaluation. Conversely, the 3.1K
machine-generated tweets collected in the first round are used for our
training and development set.

Our study aims to address three main research questions. First, we investigate
the comparison between the fine-tuning paradigm and the in-context learning
paradigm for this task. Using our curated dataset, we evaluate the performance
of the fine-tuned sequence-to-sequence model discussed earlier in comparison
to state-of-the-art LLMs. Surprisingly, we discover that our much smaller
fine-tuned model outperforms LLMs prompted with zero-shot or two-shot
exemplars on the proposed task. In fact, we find that LLMs do not achieve
satisfactory performance on our task when only provided with a few exemplars.
Second, we explore the impact of various attributes of a news article on its
susceptibility to manipulation. To conduct this analysis, we employ the
previously described sequence-to-sequence model to analyze a vast collection
of over 1M tweets and their associated articles. Our findings reveal a higher
likelihood of manipulation in social media posts when the associated news
articles exhibit low trustworthiness or pertain to political topics. Finally,
we investigate the role of manipulated sentences within a news outlet. To
address this question, we perform discourse analysis on the test set of
ManiTweet. Through this analysis, we uncover that manipulated sentences within
a news article often encompass the primary narrative or consequential aspects
of the news outlet.

Our contributions can be summarized as follows:

  * •

We introduce and define the new task of identifying manipulations of news on
social media.

  * •

We propose a novel annotation scheme for this task. Using this scheme, we
construct a dataset consisting of 3.6K samples, carefully annotated by human
experts. Each sample comprises a Twitter post paired with a reference news
article.

  * •

Through our analysis, we have demonstrated that this dataset serves as a
rigorous testbed for tackling the identification of manipulation in social
media. Specifically, we showcased the inadequate performance of large language
models in effectively addressing this challenge.

  * •

We also unveiled that low-trustworthiness and political news are more likely
to be manipulated and that manipulated sentences are more likely to contain
the main story or consequence of a news outlet.

##  2 Identifying Manipulations of News on Social Media

The goal of our task is to identify whether a social media post is manipulated
and what information is being manipulated given the associated reference
article. The models are tasked to understand whether a tweet manipulates the
reference article (Section 2.1), which newly introduced information in the
tweet is used for manipulation (Section 2.2), and which original information
in the reference article is being manipulated (Section 2.3). In the following
subsections, we provide detailed task formulation for each sub-task.

###  2.1 Sub-task 1: Tweet Manipulation Detection

Given a tweet and its associated news article, the first subtask is to
classify the manipulation label $l$ of this tweet, where
$l\in\\{\textsc{Mani},\textsc{NoMani}\\}$. A tweet is considered Mani as long
as there is at least one sentence that comments on the content of the
associated article, but this sentence contains manipulated or inserted
information. Otherwise, this tweet is NoMani.

###  2.2 Sub-task 2: Manipulating Span Localization

Once a tweet is classified as Mani, the next step is determining which
information in the reference article was manipulated in the tweet. We refer to
the information being manipulated as the pristine span, and the newly
introduced information as the manipulating span. Both pristine span and
manipulating span are represented as a text span in the reference article and
the tweet, respectively. Identifying both information can help provide
interpretability on model outputs and enable finer-grained analysis that
provides more insights, as demonstrated in Section 6. Using Figure 1 as an
example, the manipulating span is COVID-19 is not contagious at all!.

###  2.3 Sub-task 3: Pristine Span Localization

Similar to the second task, in this task, the model should output the pristine
span that is being manipulated. In cases where the manipulating span is simply
inserted, and no pristine span is manipulated, models should output a null
span or an empty string. Using Figure 1 as an example, the pristine span is
The novel COVID-19 is highly contagious.

##  3 The ManiTweet Dataset

Our dataset consists of 3,636 tweets associated with 2,688 news articles. Each
sample is annotated with (1) whether the tweet manipulates information
presented in the associated news article, (2) which new information is being
introduced, and (3) which information is being manipulated. We refer to this
dataset as the ManiTweet dataset. The following sections describe our corpus
collection and annotation process.

###  3.1 News Article Source

To facilitate the analysis of human-written tweets, we created ManiTweet by
repurposing from a fake news detection dataset, FakeNewsNet Shu et al. (2020).
FakeNewsNet contains news articles from two fact-checking websites,
PolitiFact111https://www.politifact.com/ and
GossipCop222https://www.gossipcop.com/, where each news article is annotated
with a factuality label. In addition, for each news article, FakeNewsNet also
consists of user engagement data, such as tweets, retweets, and likes, on
Twitter. We reused the news content, the factuality label, and the associated
tweets from FakeNewsNet for our ManiTweet dataset.

During the early stage of the experiment, we observe that some news articles
in FakeNewsNet are inappropriate for our study due to insufficient textual
context. For example, some articles only contain a news title, a video, and a
caption. To avoid such content, we remove news pieces containing less than 300
tokens.

###  3.2 Tweet Collection

A naive approach to collecting manipulating tweets is to request annotators to
annotate a subset of human-written tweets from FakeNewsNet. However, this
approach suffers from imbalanced tweet distributions, with the majority of
tweets not manipulating the associated article, resulting in poor annotation
cost-efficiency. Moreover, such an annotation task is challenging as
annotators must verify every information unit between the news article and the
tweet, leading to an inefficient annotation process. To address these issues,
we have devised a three-stage data collection pipeline. In the initial two
rounds of annotation, we utilize ChatGPT333GPT-3.5-turbo to generate both Mani
and NoMani tweets in a controllable manner. Human annotators are then tasked
with validating the generated tweets for their validity. In the third round of
annotation, we train a model on the data collected from the previous two
rounds and employ this model to identify Mani human-written tweets for human
annotation. This approach ensures that annotators are not overwhelmed with a
large number of NoMani tweets, resulting in significant improvements in time
and cost efficiency compared to the aforementioned naive method.

####  3.2.1 Tweet Generation

We first used Stanza444https://stanfordnlp.github.io/ to extract Location,
People, and Event named entities from all news articles. Then, we prompted
ChatGPT to generate NoMani and Mani tweets for each news article. The span of
these entities are denoted as $S=\\{S_{0},S_{1},...,S_{n}\\}$. The prompts
used for generating these tweets are as follows:

> NoMani: This is a news article: NEWS_ARTICLE. Write a tweet that comments on
> this article. Keep it within 280 characters:
>

>> Mani: This is a news article: NEWS_ARTICLE. Write a tweet that comments on
this article but changes PRISTINE_SPAN to NEW_SPAN and includes NEW_ENTITY in
your tweet. Keep it within 280 characters:  Here, PRISTINE_SPAN is a span
randomly sampled from the spans of all named entities belonging to
NEWS_ARTICLE , whereas NEW_SPAN is another span sampled from $S$ with the same
entity type as PRISTINE_SPAN. We have also experimented with other prompt
templates. While the overall generation quality does not differ much, these
prompt templates most effectively prevent ChatGPT from generating undesirable
sequences such as "As an AI language model, I cannot …".

>>

>> In addition to generating Mani tweets where new information is manipulated
from the original information contained in the associated article, we also
produce Mani tweets where new information is simply inserted into the tweet
using the following prompt:

>>

>>> This is a news article: NEWS_ARTICLE. Summarize the article into a tweet
and comment about it. Include NEW_SPAN in your summarization but do not
include NEW_SPAN in the hashtag555We instruct ChatGPT not to include  NEW_SPAN
in the hashtag. Otherwise, ChatGPT very likely does not insert  NEW_SPAN into
the main tweet.. Keep it within 280 characters:

>>

>> To further improve data quality and reduce costs in human validation, we
only keep NoMani tweets that contain at least one sentence inferrable from the
corresponding article. Concretely, we use DocNLI Yin et al. (2021), a
document-level entailment model, to determine the entailment probability
between the reference article and each tweet sentence. A valid consistent
tweet must have at least one sentence with an entailment probability greater
than 50%. Additionally, we remove Mani tweets that do not contain the
corresponding NEW_SPAN specified in the corresponding prompts.

>>

>> ####  3.2.2 Our Proposed Annotation Process

>>

>> We use Amazon’s Mechanical Turk (AMT) to conduct annotation. Annotators
were provided with a reference article and a corresponding generated tweet,
along with labels indicating whether the tweet manipulates the article, and
whether the predicted NEW_SPAN and PRISTINE_SPAN are accurate. In the first
two rounds of annotation, annotators were presented with tweets generated by
ChatGPT. The labels for these tweets were naively derived from the data
generation process, where we determined the manipulation label, NEW_SPAN, and
PRISTINE_SPAN before prompting ChatGPT to generate a tweet. In the final round
of annotation, human-written tweets were annotated, and the predicted labels
for these tweets were obtained from a model (see below paragraphs) trained on
the data collected in the previous two annotation rounds. For detailed
information regarding annotation guidelines and the user interface, please
refer to Appendix A. The following paragraphs provide an overview of our
annotation process.

>>

>> ##### First Round

>>

>> The first round of annotation is for curating machine-generated tweets,
which are used as our training set and development set. Initially, for
annotator qualification, three annotators worked on each of our HITs. We used
the first 100 HITs to train annotators by instructing them where their
annotations were incorrect. Then, the next 100 HITs were used to compute the
inter-annotator agreement (IAA). At this stage, we did not provide further
instructions to the annotators. Using Fleiss’ $\kappa$ Fleiss (1971), we
obtain an average IAA of 90.4% across all tasks, indicating a high level of
agreement. Finally, we selected the top 15 performers as qualified annotators.
These annotators were chosen based on how closely their annotations matched
the majority vote for each HIT.

>>

>> Since the annotators already achieved a reasonably high IAA, we assigned
each HIT to a single annotator to improve annotation efficiency for the
remainder of the machine-generated tweets. In addition to being annotated by
an MTurk worker, each annotation is also re-validated by a graduate student.
The average agreement between the graduate student and the MTurk worker is
93.1% per Cohen’s $\kappa$ Cohen (1960), implying a high agreement. We only
keep samples where the validation done by the graduate student agrees with the
annotation done by the worker. After two rounds of annotations, we collected
3,116 human-validated samples.

>>

>> Split # Mani # NoMani # Doc Tweet Author Train 1,465 851 1,963 Machine Dev
482 318 753 Machine Test 294 226 299 Human

>>

>> Table 1: Statistics of our ManiTweet dataset.  Model | Learning Method | Sub-task 1 | Sub-task 2 | Sub-task 3
>> ---|---|---|---|---
>> F1 | EM | F1 | RL | EM | F1 | RL
>> Vicuna | Zero-shot | 47.09 | 1.50 | 5.11 | 6.07 | 4.00 | 6.21 | 7.06
>> ChatGPT | Zero-shot | 52.49 | 1.50 | 13.30 | 15.96 | 4.50 | 7.46 | 8.35
>> ChatGPT | Two-shot | 65.28 | 1.00 | 7.62 | 8.87 | 12.50 | 13.91 | 14.18
>> Ours | Fine-tuned | 72.62∗ | 26.73∗ | 29.25∗ | 29.68∗ | 13.65∗ | 14.46 | 14.53
>> Table 2: Performance (%) of different models on the ManiTweet test set. EM
denotes Exact Match, and RL denotes ROUGE-L. Statistical significance over
best-performing LLMs computed with the paired bootstrap procedure Berg-
Kirkpatrick et al. (2012) are indicated with ∗ ($p<.01$).

>>

>> ##### Second Round

>>

>> Using the 3K examples we collected, we train a sequence-to-sequence model
that learns to tackle all three tasks jointly. Concretely, we split the
collected data into 2,316: 800 for training and validation. Model details are
described in the next paragraph. Once the model was trained, we applied it to
identify manipulation in the human-written tweets that are associated with the
articles in FakeNewsNet. Then, we randomly sampled from predicted Mani and
NoMani examples to be further validated by MTurk workers. The inter-annotator
agreement between the graduate student and the MTurk worker is 73.0% per
Cohen’s $\kappa$ Cohen (1960). While the agreement is moderately high, it is
much lower than that in the previous round. This suggests that manipulation in
human-written tweets is more challenging to identify. The user interface of
each round of annotation is shown in Section A.1. Finally, we have curated the
ManiTweet dataset. The dataset statistics are shown in Table 1.

>>

>> ##### Model

>>

>> In this paragraph, we describe the model we used to facilitate the second
round of annotation. Motivated by the advantages of generative models over
sequence-tagging models Li et al. (2021); Huang et al. (2021); Hsu et al.
(2022), we trained a sequence-to-sequence model based on LongFormer-Encoder-
Decoder (LED)666https://huggingface.co/allenai/led-base-16384 Beltagy et al.
(2020) that learns to solve the three tasks jointly. Concretely, the input to
our model is a concatenation of a tweet and a reference article:

>>

>>> Tweet: TWEET \
> Reference article: REF_ARTICLE
>>

>> If the article is NoMani, the model should output:

>>

>>> No manipulation

>>

>> Otherwise, the model should output the following:

>>

>>> Manipulating span: NEW_SPAN \
> Pristine span:
>  PRISTINE_SPAN
>>

>> For cases where  NEW_SPAN is merely inserted into the tweet, the model will
output “None” for  PRISTINE_SPAN. Using this formulation, our model is learned
to optimize the maximum likelihood estimation loss. We set identical weights
for all tokens in the outputs.

>>

>> To learn the model, we use a learning rate of 5e-5. The maximum input and
output sequence length are 1024 and 32 tokens, respectively. The model is
optimized using the AdamW optimizer Loshchilov and Hutter (2019) with a batch
size of 4 and a gradient accumulation of 8. During inference time, we use beam
search as the decoding method with a beam width of 4.

>>

>> ##  4 Experimental Setup

>>

>> ###  4.1 Evaluation Metrics

>>

>> Subtask 1 involves a binary classification problem, and thus, the Macro F1
score serves as the evaluation metric. For subtasks 2 and 3, in addition to
Exact Match, we use Macro Overlap F1 score Rajpurkar et al. (2016) and ROUGE-L
Lin (2004) as the metrics to more accurately assess model performance by
allowing models to receive partial credit for correctly identifying some parts
of the information, even if they fail to output the entire text span.

>>

>> ###  4.2 Baselines

>>

>> In addition to the model discussed in Section 3.2.2, we also recently
tested various released large language models (LLMs), including
Vicuna777Vicuna-13b is evaluated in our experiment. vic (2023) and ChatGPT,
which have demonstrated superior language understanding and reasoning
capabilities. ChatGPT is an improved version of InstructGPT Ouyang et al.
(2022) that was optimized for generating conversational responses. On the
other hand, Vicuna is a LLaMA model Touvron et al. (2023) fine-tuned on
ShareGPT888https://sharegpt.com/ data, and has exhibited advantages compared
to other open-source LLMs, such as LLaMA and Alpaca Taori et al. (2023). We
tested both the zero-shot and two-shot performance of ChatGPT, where the in-
context exemplars are randomly chosen from our training set for the two-shot
experiment. For Vicuna, we only evaluated its zero-shot ability as we found
that it often outputs undesirable texts when exemplars are provided. The
details of our prompts for these LLMs can be found in Table 3.

>>

>> Figure 2: The percentage of tweets that manipulate the associated articles
across different factuality and domains.

>>

>> ##  5 Performance Analysis

>>

>> ###  5.1 Performance on ManiTweet

>>

>> Table 2 presents a summary of the main findings from our evaluation on the
ManiTweet test set. We have made several interesting observations: First, all
LLMs we tested performed poorly across the three proposed tasks. This
indicates that simply prompting LLMs, whether with or without exemplars, is
not sufficient to effectively address problem of identifying manipulation of
news on social media. Second, despite its simplicity and smaller size compared
to the LLMs, our proposed method outperformed them significantly in
identifying social media manipulation across all three tasks. This outcome
highlights the value and importance of our training data and suggests that a
fine-tuned smaller model can outshine larger models when tackling challenging
tasks. Finally, it was surprising to discover that the two-shot exemplars
decrease ChatGPT’s ability to identify manipulating information. One possible
explanation for this unexpected result is that the in-context exemplars we
sampled from the training set do not represent the test set well, especially
on Task 2. Further investigation into the robustness of different exemplars
will be left for future research.

>>

>> ###  5.2 Remaining Challenges

>>

>> Although our proposed model exhibits significant improvements in
identifying manipulation within social media posts, there is still room for
further enhancement. To gain insights into the additional modeling and
reasoning capabilities required for effectively addressing the task of social
media manipulation, we manually compare 50 errors made by our model with
ground-truth labels and analyze the sources of errors. The distribution of
errors is illustrated in Figure 3. Notably, the most prevalent error arises
from the model’s inability to extract the correct pristine span from the
reference article that underwent manipulation. Among the 18 erroneous
predictions in this category, 16 cases result from the model outputting an
empty string. This indicates that the model considers the manipulating
information to be inserted when, in reality, it is manipulated from the
information present in the reference articles. This could be attributed to the
presence of 368 instances where the original information is an empty string,
while the alternative answers for the original information only occur 1-2
times in other instances. This can be solved by scaling down the loss for
these samples with an empty string as the label for original information.
Additionally, another common type of error involves the model’s failure to
identify opinions expressed in the tweet. In these instances, the model
considers the tweet to be manipulating information from the article, whereas
the tweet primarily expresses opinions. Examples of these errors are presented
in Appendix C.

>>

>> Figure 3: Distributions of errors.

>>

>> ##  6 Exploratory Analysis

>>

>> The learned model enables us to perform a large-scale study of manipulation
on the ManiTweet test set and the 1M human-authored tweets associated with the
news articles from the FakeNewsNet dataset. In this section, we explore how an
article is Mani and how different properties of a news article, such as domain
and factuality affect manipulation.

>>

>> ##### Insight 1: Low-trustworthiness and political news are more likely to
be manipulated.

>>

>> Figure 2 shows the percentage of the 1M human-written tweets that are
manipulated across different domains and factuality.999The domain and
factuality labels of each news article are already annotated in the
FakeNewsNet dataset. We first observe that tweets associated with False news
are more likely to be manipulated. One possible explanation is that audience
of low-trustworthy news media may pay less attention to facts. Hence, they are
more likely to manipulate information from the reference article accidentally
when posting tweets. In addition, we also see that tweets associated with
Politics news are more frequently manipulated than those with Entertainment
articles. This could be explained by the fact that people have a stronger
incentive to manipulate information for political tweets due to elections or
campaigns.

>>

>> ##### Insight 2: Manipulated sentences are more likely to contain the main
story or consequence of a news outlet.

>>

>> To discover the role of the sentence being manipulated in the reference
article, we conducted discourse analysis on these sentences. We only conducted
the analysis on our test set instead of the entire 1M human-written tweets
since Concretely, we formulate the discourse classification task as a
sequence-to-sequence problem and train a LED-based model on the NewsDiscourse
dataset Choubey et al. (2020) using a similar strategy discussed in Section
3.2.2. The learned discourse classification model achieves a Micro F1 score of
67.7%, which is on par with the state-of-the-art method Spangher et al.
(2021). Upon the discourse classification model being trained, we applied it
to all the sentences in the reference article to analyze the discourse
distribution. As shown in Figure 4, compared to other sentences, sentences
that were manipulated are much more likely to contain Main or Cause discourse,
which corresponds to the primary topic being discussed and the underlying
factor that led to a particular situation, respectively. Examples of the
manipulated sentences with a Main or Cause discourse can be found in Appendix
D.

>>

>> Figure 4: Results of discourse analysis. Manipulated sentences within news
articles tend to encompass the main story (Main) or convey the consequential
aspects (Cause) of the corresponding news outlet.

>>

>> ##  7 Related Work

>>

>> This proposed task is mostly relevant to faithfulness-related tasks. In the
following sections, we discuss important prior work in faithfulness and other
related tasks.

>>

>> ###  7.1 Faithfulness

>>

>> Faithfulness is often referred to as the factual consistency between the
inputs and outputs. This topic has mainly been studied in the field of
summarization. Prior work on faithfulness can be divided into two categories:
evaluation and enhancement, the former of which is more relevant to our study.
One line of faithfulness evaluation work developed entailment-based metrics by
training document-sentence entailment models on synthetic data Kryscinski et
al. (2020); Yin et al. (2021) or using traditional natural language inference
(NLI) models at the sentence level Laban et al. (2022). Another line of
studies evaluates faithfulness by comparing information units extracted from
the summaries and input sources using question answering (QA) Wang et al.
(2020); Deutsch et al. (2021). Recently, a study combined QA with entailment
by employing the outputs of QA as features for an entailment model Fabbri et
al. (2022).

>>

>> Our task differs from faithfulness evaluation in two key ways. Firstly, for
our task to be completed effectively, models must possess the additional
capability of distinguishing tweet sentences that relate to the reference
article from those that simply express opinions. In contrast, models
evaluating faithfulness only need to identify whether each sentence in the
output is inferable from the input. Secondly, we require models to not only
identify which original information is being manipulated by the new
information, but also to provide interpretability as to why a tweet has been
manipulated. While Kryscinski et al. (2020) trained a span extractor to
highlight mistakes in the generated summaries, they did not identify which
part of the input contained the correct information. Furthermore, this task
has not been included in any prior faithfulness evaluation work.

>>

>> ###  7.2 Citation Contextualization

>>

>> The task of citation contextualization Cohan et al. (2015); Jaidka et al.
(2017) involves identifying sections or excerpts in a cited document that are
pertinent to a specific citation in a citing document. This task is most
similar to our third subtask Section 2.3, recognizing the original information
in the input article. Our second subtask poses a more difficult challenge as
the reference article and the tweet typically do not share similar semantics
due to manipulation.

>>

>> ###  7.3 Fact-checking

>>

>> Fact-checking is a task that determines the veracity of an input claim
based on some evidence passages. Some work assumes the evidence candidates are
provided, such as in the FEVER dataset Thorne et al. (2018) and the SciFact
dataset Wadden et al. (2020). Approaches for this category of fact-checking
tasks often involve a retrieval module to retrieve relevant evidence from the
given candidate pool, followed by a reasoning component that determines the
compatibility between a piece of evidence and the input claim Yin and Roth
(2018); Pradeep et al. (2021). Other work focuses on the open-retrieval
setting, where evidence candidates are not provided, such as in the Liar
dataset Wang (2017) and the X-Fact dataset Gupta and Srikumar (2021). For this
task formulation, one of the main challenges is to determine where and how to
retrieve evidence. Some approaches determine the veracity of a claim based
solely on the claim itself and the information learned by language models
during the pre-training stage Lee et al. (2021), other methods leverage a
retrieval module to look for evidence on the internet Gupta and Srikumar
(2021) or a set of trustworthy sources Huang et al. (2022). Similar to the
faithfulness task, the key distinction between fact-checking and our proposed
task lies in the additional requirement for models to possess the capability
of discerning between tweet sentences that pertain to the reference article
and those that merely express opinions.

>>

>> ##  8 Conclusion

>>

>> In this study, we have introduced and defined a novel task called
identifying manipulation of news on social media, which aims to determine
whether and how a social media post manipulates the associated news article.
To address this challenge, we meticulously collected a dataset named
ManiTweet, composed of both human-written and machine-generated tweets. Our
analysis revealed that existing large language models (LLMs) prompted with
zero-shot and two-shot exemplars do not yield satisfactory performance on our
dataset, highlighting avenues for future research. We believe that the
resources presented in this paper can serve as valuable assets in combating
the dissemination of false information on social media, particularly in
tackling the issue of manipulation.

>>

>> ##  9 Limitations

>>

>> There are two main limitations in our work. Firstly, despite our best
efforts to minimize the gap between the training set and test set of
ManiTweet, some discrepancies remain due to the training set being generated
by machines and the test set being produced by humans. This limitation is
primarily attributed to budget constraints. In the future, with additional
resources, we aim to create an additional training set consisting entirely of
human-written tweets. By comparing the performance of models trained on this
human-written training set with those trained on the machine-generated
training set, we can gain further insights. Secondly, in our experiments
involving prompting LLMs, we only explored up to one-shot in-context
exemplars, and our prompts were not meticulously optimized. There is a
possibility that LLMs can achieve better performance when provided with more
in-context exemplars and when prompted in a more refined manner, such as
employing Chain-of-Thought reasoning Wei et al. (2022) .

>>

>> ##  10 Ethical Considerations

>>

>> The primary ethical consideration in our work pertains to the presence of
false information in two aspects: tweets that manipulate the associated news
articles and the inclusion of false news from the FakeNewsNet dataset. As with
other fact-checking and fake news detection research, it is important to
acknowledge the dual-use concerns associated with the resources presented in
this work. While our resources can contribute to combating false information,
they also possess the potential for misuse. For instance, there is a risk that
malicious users could utilize the manipulating tweets or fake news articles to
train a text generator for creating deceptive content. We highlight
appropriate and inappropriate uses of our dataset in various scenarios:

>>

>>   * •

>>

>> Appropriate: Researchers can use our framework to study the manipulation
issue on social media and develop stronger models for identifying social media
posts that manipulate information.

>>

>>   * •

>>

>> Inappropriate: The fake news and manipulating tweets in ManiTweet cannot be
used to train text generators for malicious purposes.

>>

>>   * •

>>

>> Inappropriate: Use the manipulation prompts discussed in this paper to
generate tweets and spread false information.

>>

>>   * •

>>

>> Inappropriate: The fake news in ManiTweet should not be used as evidence
for fact-checking claims.

>>

>>

>>

>> Furthermore, the privacy of tweet users is another aspect that warrants
consideration, given that we are releasing human-written tweets. However, we
assure that the dataset does not pose significant privacy concerns. The tweets
in our dataset are anonymized, and it is important to note that all the
associated news articles were already publicly available. Therefore, the
release of this dataset should not have adverse implications for privacy.

>>

>> ## References

>>

>>   * vic (2023) 2023\.  Vicuna: An open-source chatbot impressing gpt-4 with
90% chatgpt quality.

>>   * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan.
2020.  Longformer: The long-document transformer.  _arXiv preprint
arXiv:2004.05150_.

>>   * Berg-Kirkpatrick et al. (2012) Taylor Berg-Kirkpatrick, David Burkett,
and Dan Klein. 2012.  An empirical investigation of statistical significance
in NLP.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in
Natural Language Processing and Computational Natural Language Learning_ ,
pages 995–1005, Jeju Island, Korea. Association for Computational Linguistics.

>>   * Choubey et al. (2020) Prafulla Kumar Choubey, Aaron Lee, Ruihong Huang,
and Lu Wang. 2020.  Discourse as a function of event: Profiling discourse
structure in news articles around the main event.  In _Proceedings of the 58th
Annual Meeting of the Association for Computational Linguistics_ , pages
5374–5386, Online. Association for Computational Linguistics.

>>   * Cohan et al. (2015) Arman Cohan, Luca Soldaini, and Nazli Goharian.
2015.  Matching citation text and cited spans in biomedical literature: a
search-oriented approach.  In _Proceedings of the 2015 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies_ , pages 1042–1048, Denver, Colorado. Association for
Computational Linguistics.

>>   * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal
scales.  _Educational and psychological measurement_ , 20(1):37–46.

>>   * Deutsch et al. (2021) Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth.
2021.  Towards question-answering as an automatic metric for evaluating the
content quality of a summary.  _Transactions of the Association for
Computational Linguistics_ , 9:774–789.

>>   * Fabbri et al. (2022) Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and
Caiming Xiong. 2022.  QAFactEval: Improved QA-based factual consistency
evaluation for summarization.  In _Proceedings of the 2022 Conference of the
North American Chapter of the Association for Computational Linguistics: Human
Language Technologies_ , pages 2587–2601, Seattle, United States. Association
for Computational Linguistics.

>>   * Fleiss (1971) Joseph L Fleiss. 1971.  Measuring nominal scale agreement
among many raters.  _Psychological bulletin_ , 76(5):378.

>>   * Gupta and Srikumar (2021) Ashim Gupta and Vivek Srikumar. 2021.
X-fact: A new benchmark dataset for multilingual fact checking.  In
_Proceedings of the 59th Annual Meeting of the Association for Computational
Linguistics and the 11th International Joint Conference on Natural Language
Processing (Volume 2: Short Papers)_ , pages 675–682, Online. Association for
Computational Linguistics.

>>   * Hsu et al. (2022) I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott
Miller, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. 2022.  DEGREE: A data-
efficient generation-based event extraction model.  In _Proceedings of the
2022 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies_ , pages 1890–1908,
Seattle, United States. Association for Computational Linguistics.

>>   * Huang et al. (2021) Kung-Hsiang Huang, Sam Tang, and Nanyun Peng. 2021.
Document-level entity-based extraction as template generation.  In
_Proceedings of the 2021 Conference on Empirical Methods in Natural Language
Processing_ , pages 5257–5269, Online and Punta Cana, Dominican Republic.
Association for Computational Linguistics.

>>   * Huang et al. (2022) Kung-Hsiang Huang, ChengXiang Zhai, and Heng Ji.
2022.  CONCRETE: Improving cross-lingual fact-checking with cross-lingual
retrieval.  In _Proceedings of the 29th International Conference on
Computational Linguistics_ , pages 1024–1035, Gyeongju, Republic of Korea.
International Committee on Computational Linguistics.

>>   * Jaidka et al. (2017) Kokil Jaidka, Muthu Kumar Chandrasekaran, and Min-
Yen Kan, editors. 2017.  _Proceedings of the Computational Linguistics
Scientific Summarization Shared Task (CL-SciSumm 2017) organized as a part of
the 2nd Joint Workshop on Bibliometric-enhanced Information Retrieval and
Natural Language Processing for Digital Libraries (BIRNDL 2017) and co-located
with the 40th International ACM SIGIR Conference on Research and Development
in Information Retrieval (SIGIR 2017), Tokyo, Japan, August 11, 2017_, volume
2002 of _CEUR Workshop Proceedings_. CEUR-WS.org.

>>   * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming
Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of
abstractive text summarization.  In _Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346,
Online. Association for Computational Linguistics.

>>   * Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N. Bennett,
and Marti A. Hearst. 2022.  SummaC: Re-visiting NLI-based models for
inconsistency detection in summarization.  _Transactions of the Association
for Computational Linguistics_ , 10:163–177.

>>   * Lee et al. (2021) Nayeon Lee, Yejin Bang, Andrea Madotto, and Pascale
Fung. 2021.  Towards few-shot fact-checking via perplexity.  In _Proceedings
of the 2021 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies_ , pages 1971–1981,
Online. Association for Computational Linguistics.

>>   * Li et al. (2021) Sha Li, Heng Ji, and Jiawei Han. 2021.  Document-level
event argument extraction by conditional generation.  In _Proceedings of the
2021 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies_ , pages 894–908,
Online. Association for Computational Linguistics.

>>   * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic
evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81,
Barcelona, Spain. Association for Computational Linguistics.

>>   * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.
Decoupled weight decay regularization.  In _International Conference on
Learning Representations_.

>>   * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida,
Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina
Slama, Alex Ray, et al. 2022\.  Training language models to follow
instructions with human feedback.  _Advances in Neural Information Processing
Systems_ , 35:27730–27744.

>>   * Pradeep et al. (2021) Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, and
Jimmy Lin. 2021.  Scientific claim verification with VerT5erini.  In
_Proceedings of the 12th International Workshop on Health Text Mining and
Information Analysis_ , pages 94–103, online. Association for Computational
Linguistics.

>>   * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin
Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine
comprehension of text.  In _Proceedings of the 2016 Conference on Empirical
Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas.
Association for Computational Linguistics.

>>   * Salehi et al. (2015) Niloufar Salehi, Lilly C. Irani, Michael S.
Bernstein, Ali Alkhatib, Eva Ogbe, Kristy Milland, and Clickhappier. 2015.  We
are dynamo: Overcoming stalling and friction in collective action for crowd
workers.  In _Proceedings of the 33rd Annual ACM Conference on Human Factors
in Computing Systems_ , CHI ’15, page 1621–1630, New York, NY, USA.
Association for Computing Machinery.

>>   * Shu et al. (2020) Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon
Lee, and Huan Liu. 2020.  Fakenewsnet: A data repository with news content,
social context, and spatiotemporal information for studying fake news on
social media.  _Big data_ , 8(3):171–188.

>>   * Spangher et al. (2021) Alexander Spangher, Jonathan May, Sz-Rung
Shiang, and Lingjia Deng. 2021.  Multitask semi-supervised learning for class-
imbalanced discourse classification.  In _Proceedings of the 2021 Conference
on Empirical Methods in Natural Language Processing_ , pages 498–517, Online
and Punta Cana, Dominican Republic. Association for Computational Linguistics.

>>   * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann
Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.
2023.  Stanford alpaca: An instruction-following llama model.
https://github.com/tatsu-lab/stanford_alpaca.

>>   * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos
Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for
fact extraction and VERification.  In _Proceedings of the 2018 Conference of
the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New
Orleans, Louisiana. Association for Computational Linguistics.

>>   * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard,
Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman
Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient
foundation language models.  _arXiv preprint arXiv:2302.13971_.

>>   * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu
Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact
or fiction: Verifying scientific claims.  In _Proceedings of the 2020
Conference on Empirical Methods in Natural Language Processing (EMNLP)_ ,
pages 7534–7550, Online. Association for Computational Linguistics.

>>   * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.
Asking and answering questions to evaluate the factual consistency of
summaries.  In _Proceedings of the 58th Annual Meeting of the Association for
Computational Linguistics_ , pages 5008–5020, Online. Association for
Computational Linguistics.

>>   * Wang (2017) William Yang Wang. 2017.  “liar, liar pants on fire”: A new
benchmark dataset for fake news detection.  In _Proceedings of the 55th Annual
Meeting of the Association for Computational Linguistics (Volume 2: Short
Papers)_ , pages 422–426, Vancouver, Canada. Association for Computational
Linguistics.

>>   * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.
Chain of thought prompting elicits reasoning in large language models.  In
_Advances in Neural Information Processing Systems_.

>>   * Yin et al. (2021) Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021.
DocNLI: A large-scale dataset for document-level natural language inference.
In _Findings of the Association for Computational Linguistics: ACL-IJCNLP
2021_ , pages 4913–4922, Online. Association for Computational Linguistics.

>>   * Yin and Roth (2018) Wenpeng Yin and Dan Roth. 2018.  TwoWingOS: A two-
wing optimization strategy for evidential claim verification.  In _Proceedings
of the 2018 Conference on Empirical Methods in Natural Language Processing_ ,
pages 105–114, Brussels, Belgium. Association for Computational Linguistics.

>>

>>

>> Figure 5: MTurk user interface for the initial two rounds of data
annotation. Figure 6: MTurk user interface for the third round of data
annotation.

>>

>> ##  Appendix A Annotation Details

>>

>> In this section, we describe the details of our annotation process. For
better control of the annotation quality, we required that all annotators be
from the U.S. and have completed at least 10,000 HITs with 99% acceptance on
previous HITs. The reward for each HIT is $1, complying with the ethical
research standards outlined by AMT Salehi et al. (2015). Annotation interfaces
are shown below.

>>

>> ###  A.1 User Interface

>>

>> Figure 5 and Figure 6 display the annotation interface for the first two
rounds and the third round of annotation, respectively. The only difference is
that for the third round of annotation, we asked annotators to correct errors
made by our basic model discussed in Section 3.2.2. Samples that do not
receive “yes” on all three questions for the first two rounds of annotation
will be discarded. The rationale behind this design stems from three key
reasons: Firstly, the data generated for the initial two rounds of annotation
is automatically generated, enabling a relatively cost-effective approach to
discard invalid samples and generate new ones, as opposed to requesting
annotators to correct errors. Secondly, the data generated in these two rounds
is predominantly valid, which eliminates the need for annotators to rectify
errors and consequently accelerates the annotation process. Lastly, in the
third round of annotation, by instructing annotators to identify errors made
by our model, we can effectively identify the challenges faced by the model.

>>

>> ##  Appendix B Prompts for LLMs

>>

>> The zero-shot and two-shot prompt template to LLMs for the experiments
discussed in Section 4.2 is shown in Table 3. The in-context exemplars for the
two-shot experiments are randomly sampled from the training set of ManiTweet.

>>

>> Role |  Utterance
>> ---|---
>> User |  You are tackling a social manipulation problem. You will be giving a tweet and an article, and your task is to identify which information from the article is misrepresented by which information in the tweet. You should answer in the following format “Manipulating span: manipulating_span Pristine span: pristine_span” in a single line. Here, {manipulating_span} is the new information introduced in the tweet and original_concept is the original information in the article. If the tweet simply inserts information, {original_concept} should be "None". If the tweet does not manipulate the article, answer “No manipulation”. You do not need to output other information such as an explanation. You don’t need to provide code. In the following utterances, you will be presented a pair of tweet and news article.
>> LLM |  I understand the task. Please provide me with the tweet and the corresponding article, and I will identify the manipulation.
>> User |  Tweet: Sad news for Ben Affleck and Lindsay Shookus as they split after a year of dating. It looks like Ben has been getting cozy with Playboy model Shauna Sexton, but sources say it’s just light and fun. Best of luck to all involved. #BenAffleck #LindsayShookus #ShaunaSexton
>> Article: Ben Affleck and Lindsay Shookus Split After a Year of Dating Ben
Affleck and Lindsay Shookus have broken up after over a year of dating, a
source tells ET…
>> =======================================================================
>> For this example, you should answer “No manipulation”
>> LLM |  No manipulation
>> User |  Tweet: Meghan’s Lush Malibu Home Worth $16M, which comes with 8 bedrooms, 8 bathrooms, a private beach, a pool, a home cinema - they even want to spend an extra $1 million to decorate it to their own taste. Unfortunately Caboolture scores 0 here!
>> Article: Take A Look Inside Prince Harry And Meghan Markle’s Lush Malibu
Home If you’re set to marry royalty, like Meghan Markle you would expect to
move into a palace, and live out your Disney Princess fantasy…
>> ========================================================================
>> For this example, you should answer “Manipulating span: Caboolture Pristine
span: None”
>> LLM |  Manipulating span: Caboolture Pristine span: None
>> User |  Tweet: Input Tweet
>> Article: Input Article
>> Table 3: The zero-shot and two-shot prompt template to the LLMs for
producing the results in Table 2. Texts marked in gray indicate that they are
only used in the two-shot experiments.

>>

>> ##  Appendix C Additional Qualitative Examples

>>

>> Opinion Identification Error
>> ---
>> Input |  Tweet: Ariana Grande’s New Lavender Hair Color Just Inspired Our Next Trip To The Salon
>> Article: Ariana Grande Dyed Her Hair, And This Is Our Favorite Color
Transformation Yet Ariana Grande is giving us whiplash with her hairstyles
lately, and we honestly love it. On July 18th, Grande took to Instagram to
debut her latest hair transformation. She’s now sporting pastel lavender locks
and good god (is a woman), it looks amazing on her…
>> Prediction |  Is manipulated: Yes
>> Manipulating span: Salon
>> Pristine span: None
>> Pristine Span Extraction Failure
>> Input |  Tweet: Transcript: Democratic Presidential Debate in Brooklyn view more
>> Article: The Democratic Debate in Cleveland This is rightly a big issue in
Ohio. And I have laid out my criticism, but in addition my plan, for actually
fixing NAFTA. Again, I have received a lot of incoming criticism from Senator
Obama. And the Cleveland Plain Dealer examined Senator Obama’s attacks on me
regarding NAFTA and said they were erroneous. So I would hope that, again, we
can get to a debate about what the real issues are and where we stand because
we do need to fix NAFTA. It is not working. It was, unfortunately, heavily
disadvantaging many of our industries, particularly manufacturing. …
>> Prediction |  Is manipulated: Yes
>> Manipulating span: Cleveland
>> Pristine span: None
>> Table 4: Example outputs from our baseline model where it produces
erroneous outputs.

>>

>> Table 4 presents two instances where our baseline model makes errors. In
the first example, our model was not able to identify that “Inspired Our Next
Trip To The Salon” is an expression of opinion, resulting in the model
incorrectly classifying this sample as Mani. In the second example, although
our model accurately predicts the example as Mani and extracts the correct
manipulating span, it fails to extract the pristine text span correctly,
likely due to the nature of the training set, as discussed in Section 5.2.

>>

>> ##  Appendix D Discourse Analysis Examples

>>

>> Table 5 shows examples of manipulated sentences associated with a Main or
Cause discourse. A main discourse implies that the sentence conveys the main
story of an article, whereas a cause discourse indicates that the sentences
discuss the consequential aspect of the main story.

>>

>> Main Discourse
>> ---
>> Tweet |  #Zuckerbergtestimony Mark Zuckerberg’s testimony before the House Energy and Commerce Committee is over.
>> Article |  … U.S. Rep. Joe Barton, R-Texas, chairman of the House Energy and Commerce Committee, made the following statement today during the full committee hearing on the Administrationś FY 0́7 Health Care Priorities: "Good afternoon.. Let me begin by welcoming Secretary Michael Leavitt today to the Energy and Commerce Committee. We look forward to hearing him testify about the Administrationś Fiscal Year 2007 Health Care Priorities …
>> Cause Discourse
>> Tweet |  Thank you, Rep. Johnson, for your service! Weekly Republican Address: Rep. Sam Johnson (R-TX) … via @YouTube
>> Article |  … In the address, Boehner notes that this is a new approach that hasn’t been tried in Washington – by either party – and it is at the core of the Pledge to America, a governing agenda Republicans built by listening to the people. Leader Boehner recorded the weekly address earlier this week from Ohio, where he ran a small business and saw first-hand how Washington can make it harder for employers and entrepreneurs to meet a payroll and create jobs. Following is a transcript …
>> Table 5: Examples of manipulated sentences with a Main discourse and a
Cause discourse. The manipulated sentences are marked in boldface. The
manipulating and pristine spans are marked in red and blue, respectively.