# Contrastive News and Social Media Linking using BERT for Articles and Tweets
across Dual Platforms

Jan Piotrowski<EMAIL_ADDRESS>University of Warsaw,MIM
SolutionsPoland ,  Marek Wachnicki<EMAIL_ADDRESS>MIM SolutionsPoland
,  Mateusz Perlik<EMAIL_ADDRESS>University of Warsaw,MIM
SolutionsPoland ,  Jakub Podolak<EMAIL_ADDRESS>University of
AmsterdamThe Netherlands ,  Grzegorz Rucki<EMAIL_ADDRESS>MIM
SolutionsPoland ,  Michał Brzozowski<EMAIL_ADDRESS>MIM
SolutionsPoland ,  Paweł Olejnik<EMAIL_ADDRESS>University of
WarsawPoland ,  Julian Kozłowski<EMAIL_ADDRESS>University of
WarsawPoland ,  Tomasz Nocoń<EMAIL_ADDRESS>University of
WarsawPoland ,  Jakub Kozieł<EMAIL_ADDRESS>Warsaw University of
TechnologyPoland ,  Stanisław Giziński<EMAIL_ADDRESS>University of WarsawPoland and  Piotr Sankowski<EMAIL_ADDRESS>University
of Warsaw, IDEAS NCBR,MIM SolutionsPoland

(2018)

###### Abstract.

X (formerly Twitter) has evolved into a contemporary agora, offering a
platform for individuals to express opinions and viewpoints on current events.
The majority of the topics discussed on Twitter are directly related to
ongoing events, making it an important source for monitoring public discourse.
However, linking tweets to specific news presents a significant challenge due
to their concise and informal nature. Previous approaches, including topic
models, graph-based models, and supervised classifiers, have fallen short in
effectively capturing the unique characteristics of tweets and articles.

Inspired by the success of the CLIP model in computer vision, which employs
contrastive learning to model similarities between images and captions, this
paper introduces a contrastive learning approach for training a representation
space where linked articles and tweets exhibit proximity. We present our
contrastive learning approach, CATBERT (Contrastive Articles Tweets BERT),
leveraging pre-trained BERT models. The model is trained and tested on a
dataset containing manually labeled English and Polish tweets and articles
related to the Russian-Ukrainian war. We evaluate CATBERT’s performance
against traditional approaches like LDA, and the novel method based on OpenAI
embeddings, which has not been previously applied to this task. Our findings
indicate that CATBERT demonstrates superior performance in associating tweets
with relevant news articles. Furthermore, we demonstrate the performance of
the models when applied to finding the main topic – represented by an article
– of the whole cascade of tweets. In this new task, we report the performance
of the different models in dependence on the cascade size.

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

X (formerly Twitter) has emerged as a contemporary agora in recent years,
serving as a platform where individuals express their opinions and viewpoints
on current events. According to Kwak et al. (2010), more than 85% of the
topics discussed on this social media platform are directly linked to ongoing
news. However, a challenge arises as only a limited number of tweets establish
explicit connections to specific events. Public opinion researchers,
institutions engaged in countering disinformation, and other entities
responsible for monitoring public discourse are compelled to heavily rely on
the active participation of humans in identifying novel keywords and emerging
events. Hence, the development of automatic tools that could link tweets to
related press articles would allow for indirect identification of
disinformation and thus have possible applications in fighting the spread of
fake news. Disinformation on X is present in various contexts, e.g., medical,
political, and societal, and is present in all regions of the world. However,
a new wave was observed after the aggression of Russia on Ukraine. This wave
is not restricted to the English language but is clearly present in other
local languages, e.g., Ukrainian, Polish, or Czech. Many examples of fake news
in this context describe non-existing events. In order to study this
phenomenon and ultimately provide tools to counter it, we concentrate on
studying related news and tweets in two languages: Polish and English.

However, the significant challenge in linking tweets to real events lies in
the fact that tweets differ in nature from press articles. They are concise,
with a maximum length of 280 characters, and are generally more informal in
tone. Initially, topic models (Zhao et al., 2011) were employed to address
this challenge. However, these models proved inadequate in capturing the
differences between tweets and articles, as they treated both domains
uniformly. Another approach utilized a graph-based latent variable model based
on Named Entity Recognition (Guo et al., 2013), but it fell short by focusing
primarily on NER while neglecting a significant portion of semantic
information from articles and tweets. Another study (Krestel et al., 2015)
focused on a supervised trained classifier, but it faced limitations as the
classifier was designed for a fixed set of articles and lacked the ability to
build a general representation, essential for analyzing new topics. In
contrast, Mogadala et al. (2017) proposed the use of two embedding spaces
based on GloVe (Pennington et al., 2014) that were transformed into a new
space, where related tweets and news articles would be positioned in close
proximity to each other. In particular, none of these papers applied deep
neural networks for this task, thus leveraging cutting-edge natural language
processing capabilities of existing transformer networks. In this paper, we
commence such a study and successfully demonstrate the capabilities of a novel
neural network model designed especially for this task. In order to
demonstrate the capabilities of the proposed model, we create new large
datasets which allow for reliable comparison of our approach with existing
ones.

The first contribution of the paper are new annotated datasets linking tweets
and articles. We hope that these datasets will serve as a benchmark and
reference for further study of this topic. We constructed four datasets:

  * •

the _Tweets – PAP News Dataset_ – comprising tweet-article pairs in Polish
related to the war in Ukraine,

  * •

the _Tweets – BBC News Dataset_ – contains tweet-article pairs in English
related to the war in Ukraine,

  * •

the _Cascades – PAP News Dataset_ – in this dataset we expand individual
tweets coming from _Tweets – PAP News Dataset_ into related information
cascades, i.e., quotes and replies,

  * •

the _Cascades – BBC News Dataset_ – here full information cascades have been
downloaded on the topic of the war in Ukraine.

The _Tweets_ datasets served the purpose of training, prototyping, and
benchmarking models. The _Cascades_ datasets are created for the purpose of
testing whether it is possible to more accurately find the related article for
the whole information cascades.

Inspired by the success of the CLIP model (Radford et al., 2021) in computer
vision, which is based on a contrastive learning approach and models
similarities between images and captions, we propose a similar contrastive
learning approach to train a representation space where transformed linked
articles and tweets exhibit proximity. As the second contribution of this
paper, we introduce our contrastive learning approach with pre-trained BERT
(Devlin et al., 2019) named _CATBERT_ (Contrastive Articles Tweets BERT). An
important aspect of contrastive learning is that it was designed to handle
automatically generated data and requires only a limited number of manual
annotations – typically only for the creation of the test set. Indeed, we
apply it to a dataset where matches are generated through heuristics that are
not flawless. Nonetheless, the implementation of contrastive learning, which
was demonstrated to be robust in various contexts (Zhong et al., 2023),
applies as well to our case and proves to be a valuable tool in mitigating the
limitations associated with manual data curation.

In our experiments, we are comparing the _CATBERT_ model with 3 other
solutions, which generate vector representation of texts, i.e., Latent
Dirichlet Allocation (LDA), Term Frequency-Inverse Document Frequency (TF-
IDF), and OpenAI embeddings generated via API calls. After computing such
vector representations for both articles and tweets, the best matching pairs
are computed using cosine similarity. First, we demonstrate that _CATBERT_
performed significantly better than other methods on the _Tweets – PAP News
Dataset_. This clearly demonstrates the strength of contrastive learning in
this novel context. Second, we study the cascade datasets with the aim to show
new approaches for aggregating information over a set of related tweets. With
this respect, we concentrate on the study of the English language dataset,
which due to its more diverse nature is more challenging. We observe that all
developed methods perform relatively weakly. Where the best results are
obtained using OpenAI embeddings aggregated appropriately with the max
function. Furthermore, our findings demonstrate that aggregating indeed
increases the average precision of the methods and it is easier to find
related articles for the whole information cascades than for single tweets.

##  2\. Related work

###  2.1. Different Sources of News Information

The linkage between tweets and news articles represents only one example of
analyzing two distinct mediums of information. In this context, Guo et al.
(2013) focuses on leveraging correlations between textual elements, employing
features such as hashtags, named entities, and temporal information to
establish connections. They further propose a graph-based latent variable
model to facilitate this linkage. In a similar vein, Suarez et al. (2018b)
incorporates Named Entity Recognition and article text summarization
techniques in their methodology for linking tweets and news articles.
Expanding upon this approach, Syed et al. (2021) augments the use of Named
Entity Recognition with a graph-based representation.

Furthermore, several studies have considered the inclusion of diverse news
providers to detect and track emerging events. For instance, Mele et al.
(2019) extends this line of research to encompass topic detection and
tracking. In a different vein, Spitz and Gertz (2018) utilizes named entities
as a means of aggregating news from multiple streams. Moreover, they employ a
graph-based framework to represent the entirety of the content, enabling
further downstream analysis.

However, all of the above studies suffer from small test sample sizes, which
limits the possibility of rigorously validating the methods.

###  2.2. Multimodal Representation Learning

Multimodal models play a pivotal role in mapping inputs from diverse data
modalities, such as images and text, into a shared representation space (Zhang
et al., 2022; Radford et al., 2021). This framework of data integration has
garnered significant interest and excitement in the research community.
Nevertheless, the idea of mapping images and texts into a shared embedding
space has been explored in earlier works (Socher and Fei-Fei, 2010). There
have been recent efforts in formulating image and text embeddings as metric
learning (Frome et al., 2013), and multilabel classification (Joulin et al.,
2015). In the context of our work, these approaches could also be used
assuming that one modality is the text of the articles and the other is the
text of the tweets.

##  3\. Datasets

We constructed four datasets: the _Tweets – PAP News Dataset_ , the _Tweets –
BBC News Dataset_ , the _Cascades – PAP News Dataset_ , and the _Cascades –
BBC News Dataset_. The _Tweets – PAP News Dataset_ served the purpose of
training, prototyping, and benchmarking models. The _Cascades – PAP News
Dataset_ served the purpose of benchmarking cascade aggregation functions. The
_Tweets – BBC News Dataset_ was used to train models on English texts.
Finally, we evaluated the models’ performance on the _Cascades – BBC News
Dataset_.

We share all of those datasets for public use. Due to legal reasons, the
tweets are represented by their IDs, and news articles by their URLs – their
content is not included. The datasets are available under
https://anonymous.4open.science/r/cascades_news_dataset-2211.

###  3.1. Tweets – PAP News Dataset

The Russian invasion of Ukraine in 2022 got a lot of attention on social
media. Twitter became a key place to share news, especially stories that
hadn’t been covered by newspapers or TV yet. Later on, some of these stories
were proven wrong and called fake news. Identifying news that has not been
reported by traditional media is an intriguing endeavor and holds the
potential to build models identifying fake news. This unique situation
inspired us to document the event by creating a dataset that can be analyzed
and researched further. Consequently, we chose to focus on Polish tweets and
news from 2022 related to the war in Ukraine. To achieve our goal, we had to
choose the source of news, fetch tweets with Twitter API, and choose a
strategy to create training and test datasets.

####  3.1.1. News

We decided to use news scraped from the PAP (Polish Press Agency) website. PAP
was chosen as the most reliable and the least biased source of information in
Polish. Moreover, these articles are concise and informative. To download news
only related to the events in Ukraine, we fetched only those containing
specific keywords (like ”Ukraine”, and ”Russia”, see A.1 for the full list) –
resulting in 8,153 articles in the selected timeframe (January 1 – November 5,
2022).

####  3.1.2. Training set

Inspired by CLIP (Radford et al., 2021) training process, we wanted to find a
natural connection between tweets and articles, similar to images with
captions. In the case of tweets - these are tweets with URLs to articles; this
way we can be sure those tweets refer to the mentioned article. Unfortunately,
the number of tweets with a direct link to the PAP article was scarce (around
1,000). We extended this dataset with all replies and quotes (including
replies to quotes) to the tweets with PAP URLs - ultimately 6,049 tweet-
article pairs. The vast majority of tweets come from the official PAP account
on Twitter 111https://twitter.com/PAPinformacje or replies to tweets from that
account.

####  3.1.3. Test and validation sets

To create gold-standard tweet-article pairs, we presented two sets of news-
tweet pairs to our data labeler:

  * •

Pairs containing an article and a tweet with a hyperlink to the article, or a
tweet that is a quote of, or a reply to, the mentioned tweet.

  * •

Pairs created by finding relevant tweets to an article by news-related
keywords chosen by the data labeler.

Subsequently, data labelers were required to assign one of the following three
labels to each pair:

  * •

match \- the tweet and the article both pertain to the same event,

  * •

no match \- the tweet and the article describe distinct events,

  * •

skip \- it is indeterminable whether the tweet relates to any event.

The outcome yielded a dataset comprising 335 news articles and 2648 tweets.
This dataset was subsequently partitioned equally by news into two subsets:
the test and validation datasets. The test dataset encompasses 168 news
articles and 1260 tweets with 593 match and 668 no match labels, whereas the
validation dataset contains 167 news articles and 1388 tweets with 594 match
and 800 no match labels. It is important to note that no news article or tweet
is present in both datasets.

####  3.1.4. Inter-Annotator agreement

To evaluate the efficacy of our annotation process, we enlisted 13 people to
assess the relevance of 30 tweet-news pairs drawn from the _Tweets – PAP News
Test Dataset_. The Fleiss’ kappa is 0.113, which means slight agreement
between voters (Landis and Koch, 1977). It suggests that the task is non-
trivial. These findings align with similar conclusions drawn by Suarez et al.
(2018a).

###  3.2. Tweets – BBC News dataset

In order to see how our findings transfer to other languages, we created an
analogous dataset with English tweets and news, both from September 2023.

We chose BBC as the news source, as it is generally considered to be impartial
and informative. It also allowed us to scrape news related to the war in
Ukraine easily, as there is a tab containing articles on this
topic222https://www.bbc.com/news/world-60525350. We downloaded 101 news
articles in total, in the time range from September 1 to September 26, 2023.
In this dataset, we used only 45 articles, for which we were able to find
relevant tweets.

We downloaded all tweets, with their quotes and replies, from September 15 to
September 26 (we were unable to download older tweets due to recent
constraints in the Twitter API (Calma, 2023)) containing a URL to one of the
downloaded articles. The total number of downloaded tweets is 5,851; 717 of
which contained one of the URLs. Tweets with a link to a downloaded article,
together with replies and quotes, are automatically labeled as matching,
giving us an English training dataset.

###  3.3. Cascades

In studying Twitter interactions, we use the cascade model. Within this
paradigm, we define a cascade as a directed acyclic graph, where each node
represents an individual tweet. Edges between these nodes are added if one
tweet is a reply or a quote of another tweet. The initial tweet is the edge
source, and the reply or quote is the target. This structure not only maps the
flow of information but also encapsulates the hierarchical and temporal
character of communication on the platform.

#### Cascades construction

Cascades datasets are constructed in a recursive manner. Starting from the
root we check whether it has children (quotes or replies) and if so we add
them to the cascade graph. We repeat this operation on newly added nodes
recursively until we reach the leaves of the graph. In this manner, the
created cascade is a tree.

###  3.4. Cascades – PAP News Dataset

Based on the PAP test and validation datasets, we created new datasets, which
contained partial cascades with root tweets from the original dataset.
Partial, because constructed using previously gathered tweets, as we were
unable to download these cascades due to changes in Twitter API policies
(Calma, 2023). In the process, all tweets from a given cascade have the same
corresponding article as the root tweet. The goal of this dataset is to find
the best cascade aggregation function, taking advantage of well-labeled data.

###  3.5. Cascades – BBC News Dataset

Due to our interest in cascades of varying sizes, we decided to download whole
cascades on the topic of the war in Ukraine, focusing on those bigger than a
few tweets. For this purpose, we found all tweets from around a week (from
September 20 to September 28, 2023) containing keywords concerning the war
(see A.2 for the full list), which were recognized as the most relevant by
Twitter API. After removing the tweets with potentially very large cascades
(we set the limit to 400 replies and quotes in total), we were left with 458
cascades containing 61,912 tweets in total, with a median of 45.5, a mean of
135, and a maximum of 1,318 tweets.

For the news part, we used all the 101 BBC articles that were already
downloaded when creating the _Tweets – BBC News Dataset_.

#### Data labelling

Firstly, we limited the dataset to the 100 largest cascades. Then we assigned
an appropriate news article to each cascade, if such is present in the
collection of those we gathered from BBC. Based on this assignment, we
constructed the labeled dataset, marking every cascade-article pair as
unrelated, unless the article was assigned to the cascade.

###  3.6. Data processing

The datasets were processed suitably to the model used and the dataset
language. The data processing pipeline could be composed of:

  * •

Cleaning – applied to both tweets and articles, performs basic text cleaning,
including: lowercasing, removing unnecessary characters, such as multiple
whitespaces, numerical values, and punctuation, removing URLs, short words,
and user tags, and processing emojis;

  * •

Lemmatization – applied to both tweets and articles, lemmatizes the given
text; has both Polish and English variants.

##  4\. Models

In an effort to address the challenge of detecting semantic relationships
between tweets and articles, we adopt a methodology that involves encoding
information from both sources into vector representations and subsequently
comparing them using the cosine similarity measure. Thus, the similarity score
for each tweet-article pair is a value between -1 and 1. By obtaining the
score for every possible pair, we get the similarity matrix, in which the rows
represent tweets and the columns represent articles. To produce the
classification matrix (with binary values), we evaluate whether the values in
the similarity matrix exceed or fall below the similarity threshold.

We investigate two distinct approaches to accomplish the task under
investigation. The first approach employs a single embedding model, wherein
tweets and articles are treated as texts originating from a common domain. As
a result, we make use of a single encoder model to embed both tweets and
articles into their respective vector representations.

In contrast, the second approach utilizes a dual embedding model – _CATBERT_.
This model employs two distinct encoder models to separately embed articles
and tweets, acknowledging the inherent differences in their nature and
characteristics. By using separate encoders, we aim to more effectively
capture the unique features and nuances of each text type.

To evaluate the performance of the first approach, we employ encoding methods
such as Latent Dirichlet Allocation (LDA), Term Frequency-Inverse Document
Frequency (TF-IDF), and OpenAI embeddings. They serve as a basis for
comparison with the second approach.

Lastly, we introduce models that leverage the aforementioned encoding methods,
but with a different focus: instead of working with tweet-article pairs, these
models operate on cascades datasets. These models utilize an aggregation
strategy.

###  4.1. LDA based model

In this approach, we use Latent Dirichlet Allocation (LDA), a generative
probabilistic model for collections of discrete data such as text corpora.
After transforming input text with the LDA model, the output might be
interpreted as a vector of likelihoods that the given input is related to the
corresponding topics. In this way, we generate embeddings for texts from both
tweets and news articles.

To achieve the best possible results, we employ data processing involving
cleaning and lemmatization (see 3.6 for the details).

###  4.2. TF-IDF based method

In the center of this approach, there is a TF-IDF vectorizer. TF stands for
term frequency, whereas IDF for inverse document frequency. TF-IDF is a method
used in information retrieval for assigning each word its importance in a
document in the context of whole corpora.

This method involves an extra step in our data preparation process besides its
cleaning and lemmatization. Instead of fitting the vectorizer to the whole,
possibly lengthy article, we aimed to represent it in a summary. After
reviewing the structure of PAP articles, we observed that its first paragraph
- always written in bold by journalists - could quite adequately serve as a
summary, by containing the key information from the article.

Such prepared summaries of articles, jointly with the tweets’ texts, form the
corpora for the TF-IDF vectorizer, which is trained on them collectively. The
fitted vectorizer is used to transform documents (tweets and articles’
summaries) into vectors of weights of words found in a given document. The
last step is a pairwise comparison of tweets and summaries using the cosine
similarity of beforehand prepared embeddings.

###  4.3. OpenAI embeddings

In this approach, we gather the tweet and news embeddings through the OpenAI
API calls. Specifically, we use the embeddings endpoint with the text-
embedding-ada-002 model (Greene et al., 2022). This is a very powerful
GPT3-based model trained for diverse NLP tasks on large datasets. It
outperforms most of the models listed in the Massive Text Embedding Benchmark
(MTEB) Leaderboard (Face, 2023).

###  4.4. CATBERT

In our approach, we create a model that consists of two embedding networks,
where each of them computes embedding for different text types (tweets – short
texts, articles) into joint representation space. This way, similarities
between tweets and articles can be measured by calculating the cosine
similarity of their representation. For the embeddings of both tweets and
articles, we utilize pre-trained BERT-based models tailored to the language of
the task. Specifically, for English, we employ Roberta (Liu et al., 2019) for
articles and Twitter Roberta Base pre-trained on Twitter data (Barbieri et
al., 2020) for tweets. For Polish, we use HerBERT (Mroczkowski et al., 2021)
for articles and TrelBERT (Szmyd et al., 2023) for tweets. Finally, we utilize
the CLS token to generate embeddings.

Similarly to CLIP (Radford et al., 2021), we employ a contrastive learning
approach to train the model. Given pairs of matching and non-matching tweet-
article combinations, we aim to bring together the representations of matching
pairs while pushing apart the representations of non-matching pairs. As a
learning objective cosine embedding loss was used, which is computed given
input embedding tensors $x_{1},\,x_{2}$ and label tensor $y$ that states
whether the inputs are _similar_ or _dissimilar_ as follows (here cos is the
cosine similarity function of two embeddings):

$\text{loss}(x,y)=\begin{cases}1-\cos(x_{1},x_{2})&\text{if $y=1$}\\\
\max(0,\cos(x_{1},x_{2}))&\text{if $y=-1$}\end{cases}$

The _Tweets – PAP News Training Dataset_ has only positive tweet-article
pairs. Therefore, to generate training pairs with negative examples, we sample
articles from the dataset that are unrelated to the current tweet for each
tweet. The ratio of positive-to-negative examples serves as a hyperparameter
in our approach, and it is independent of the batch size (in contrast to CLIP
(Radford et al., 2021)).

A challenge we faced was handling long texts, as BERT has a context window
limited to 512 tokens, while tweets have their constraints and articles do
not. To address this issue, we explored several methods and evaluated their
performance in different experimental configurations.

####  4.4.1. BERT with truncation

The most straightforward approach involves truncating texts that exceed the
512-token limit.

####  4.4.2. BERT with mean embeddings (BELT-like)

We split tokens of the whole text into chunks. For each chunk, we add special
tokens at the beginning and the end and ensure uniform chunk lengths. We stack
the chunks and obtain one embedding for each chunk. The final text embedding
is obtained by taking the mean of all the chunks’ embeddings. This approach is
inspired by BELT (Brzozowski and Wachnicki, 2023).

####  4.4.3. BERT with data augmentation

We implement a data augmentation strategy, which involves creating additional
examples by splitting articles into parts. Specifically, we take the first 256
tokens of an article as a header, which conveys the main idea, and divide the
remaining text into 256-token parts.

####  4.4.4. Longformer

We explored the use of the Polish Longformer (Beltagy et al., 2020; Dadas,
2022) model, which is designed to handle longer text sequences and has a
context window limited to 4096. This surpasses BERT’s context window by a
factor of 8 and thus can easily fit the whole news article text.

##  5\. Experiments

###  5.1. Evaluation metrics

To measure model performance, we use standard metrics, such as average
precision, accuracy, precision, recall, and F1-score, using their standard
sklearn implementations (Pedregosa et al., 2011). The first one is score-
based, therefore the similarity matrix returned by the model is used, while
the rest use the model’s classification matrix. In each case, the model’s
matrix is flattened, as well as the ground truth matrix.

The ground truth matrix in each cell contains one of three possible values:

  * •

1 – the corresponding tweet and news article match;

  * •

-1 – the corresponding tweet and news article don’t match;

  * •

0 – it is not known whether the corresponding tweet and news article match.

Before calculating metrics, the flattened matrices have removed cells
corresponding to the 0 values of the ground truth matrix.

###  5.2. Human annotators’ performance

To have a good starting point, we check how well human annotations (described
in 3.1.4) perform when evaluated on a subset of the _Tweets – PAP News Test
Dataset_. We define the similarity score for the tweet-article pairs as an
average of assessment values (assuming the match label corresponds to 1, and
no match to 0). The threshold for the pair matching was 0.5.

Based on this, we obtain the performance metrics presented in Table 1.

Metric | Value
---|---
Average precision | 0.996
Accuracy | 0.900
Precision | 1.000
Recall | 0.889
F1-score | 0.941
Table 1. Metrics of human annotators’ performance.

###  5.3. Optimization of model parameters

For each model tested in this paper, we perform hyperparameter optimization.
For this purpose, we train the model on the _Tweets – PAP News Training
Dataset_ , then evaluate it on the _Tweets – PAP News Validation Dataset_. Our
aim is to maximize the average precision. The optimization process was aided
by Optuna (Akiba et al., 2019), a hyperparameter optimization framework which
automates the task.

In the next step, the similarity threshold is chosen. We iterate over possible
values until we find the one maximizing the F1-score of the model on the
_Tweets – PAP News Validation Dataset_.

Various training configurations of _CATBERT_ were evaluated. The results of
these evaluations are presented in Table 2. In subsequent sections of the
paper, when referring to _CATBERT_ , we will specifically be referring to the
model trained with a truncation configuration.

Configuration name | Average precision | Accuracy | Precision | Recall | F1-score
---|---|---|---|---|---
BERT with truncation | 0.715 | 0.680 | 0.638 | 0.740 | 0.685
BERT with mean embeddings | 0.681 | 0.665 | 0.618 | 0.756 | 0.680
BERT with data augmentation | 0.661 | 0.628 | 0.577 | 0.784 | 0.665
Longformer | 0.547 | 0.530 | 0 | 0 | 0
Table 2. Evaluation of different _CATBERT_ configurations.

Final model results on the _Tweets – PAP News Test Dataset_ can be seen in
Table 3.

Model | Average precision | Accuracy | Precision | Recall | F1-score
---|---|---|---|---|---
TF-IDF | 0.639 | 0.596 | 0.576 | 0.530 | 0.552
LDA | 0.589 | 0.578 | 0.541 | 0.685 | 0.604
CATBERT | 0.715 | 0.680 | 0.638 | 0.740 | 0.685
OpenAI Embeddings | 0.738 | 0.650 | 0.618 | 0.671 | 0.643
Table 3. Results on the _Tweets – PAP News Test Dataset_ for each model.

###  5.4. Linking cascades to articles

In order to see how cascades relate to news articles, we used models developed
to detect a linkage between a single tweet and a news article and took a two-
step approach. First, we ran a model on each tweet from a cascade and
calculated its similarity value with each article, forming a collection of
similarity vectors. Then we aggregated these similarity vectors to produce a
single vector of news similarity for the cascade. We conducted experiments
with the following three aggregation functions: mean, median, and max (see
Table 4). For each model, we chose the optimal aggregation method for further
experiments.

Model | mean | median | max
---|---|---|---
TF-IDF | 0.559 | 0.557 | 0.567
LDA | 0.602 | 0.600 | 0.595
CATBERT | 0.624 | 0.622 | 0.620
Table 4. Average precision on the _Cascades – PAP News Validation Dataset_ for
different models and aggregation functions.

###  5.5. Cascades size impact on predictions

We conducted an experiment whose goal was to check how the size of cascades
affects the quality of model predictions. For this purpose, we used the
_Cascades – BBC News Dataset_.

#### Training models

To adjust the models to the language switch from Polish to English, we trained
the models again, this time on the _Tweets – BBC News Dataset_. The previously
optimized hyperparameters were used in the training process, with the hope
that the newly trained models would not perform significantly worse than their
counterparts trained on Polish data. In the case of the _CATBERT_ model, we
replaced the pre-trained BERT-based models as described in 4.4.

#### Cascade cutting

Given a cascade of size $N$, the cut cascade of size $n\leq N$ is the original
cascade with only the $n$ oldest tweets. This approach ensures that each cut
cascade represents a real cascade at a certain point in time.

Using cascade cutting, we compare model results on cascades with the same
starting tweet and differing sizes. It allows us to accurately measure the
impact of the number of tweets in the cascade on the quality of model
predictions.

###  5.6. Results

_CATBERT_ outperforms all the other models in terms of accuracy, precision,
recall, and F1-score on the _Tweets – PAP News Test Dataset_. The average
precision for OpenAI embeddings is slightly higher.

The basic approach of text truncation to train the _CATBERT_ model, closely
followed by the BELT-inspired, emerged as the most effective method in terms
of both metrics’ results and training time. Employing the large Longformer
model showed the least success in terms of both aspects.

Unfortunately, switching the language from Polish to English was an
unsuccessful attempt with all the metrics dropping significantly. The OpenAI
model was expected to perform similarly since it is a versatile multi-language
model, so this may be caused by the strict approach to labeling the _Cascades
– BBC News Dataset_ and thus, the difficulty of this dataset. However, LDA,
TF-IDF, and _CATBERT_ models performed markedly worse, while achieving
comparable results on the _Tweets – PAP News Test Dataset_.

OpenAI embeddings with max aggregation is the only model that benefits from
analyzing larger cascades. It achieves the best performance on cascades of
size of around 150 tweets. There is a slight drop in average precision for
cascades of size from 10 to 70 though. Other models visibly yield the best
results for cascades of size 1 (single tweets). The root tweets are usually
the most informative and it can be inferred that the additional noise
adversely impacts the performance.

Figure 1. Results on the _Cascades – BBC News Dataset_ for LDA with mean
aggregation. Figure 2. Results on the _Cascades – BBC News Dataset_ for TF-IDF
with max aggregation. Figure 3. Results on the _Cascades – BBC News Dataset_
for OpenAI embeddings with max aggregation. Figure 4. Results on the _Cascades
– BBC News Dataset_ for OpenAI embeddings with mean aggregation. Figure 5.
Results on the _Cascades – BBC News Dataset_ for _CATBERT_ with mean
aggregation. Figure 6. Average precision results on the _Cascades – BBC News
Dataset_.

##  6\. Discussion

In this paper, we introduce an approach, named _CATBERT_ , for associating
news articles with relevant tweets. Our method leverages contrastive learning
to bridge the gap between these two sources of information. To facilitate a
comprehensive evaluation of our proposed approach, we have curated datasets in
both Polish and English languages. To the best of our knowledge, this is the
first attempt to utilize neural networks for this task.

We study different variants of _CATBERT_ on both Polish and English language
datasets. Furthermore, we showcase the effectiveness of our models when
applied to identifying the main topic represented by an article within a
cascade of tweets.

The dataset accompanying this paper is notably larger than those previously
available, providing a resource for more rigorous benchmarking. The
availability of a larger dataset will enable the development and evaluation of
more robust solutions.

Additionally, the challenge we have addressed in this paper can be further
explored from the perspective of the heterogeneity gap. The dataset utilized
in this study poses a challenge for the methods we have evaluated. However,
the creation of such a dataset provides the opportunity to benchmark
approaches for reducing the heterogeneity gap (as exemplified by (Liang et
al., 2022)), between mediums not previously considered in such benchmarks.

##  7\. Acknowledgements

This work was supported by the National Science Center (NCN) grant no.
2020/37/B/ST6/04179 and the ERC CoG grant TUgbOAT no 772346.

## References

  * (1)
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019.  Optuna: A Next-generation Hyperparameter Optimization Framework. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_.
  * Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020.  TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 1644–1650.  https://doi.org/10.18653/v1/2020.findings-emnlp.148
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The Long-Document Transformer.  arXiv:2004.05150 [cs.CL]
  * Brzozowski and Wachnicki (2023) Michał Brzozowski and Marek Wachnicki. 2023.  BELT (BERT For Longer Texts).  Retrieved 2023-10-13 from https://github.com/mim-solutions/bert_for_longer_texts
  * Calma (2023) Justine Calma. 2023.  Twitter just closed the book on academic research.  Retrieved 2023-10-09 from https://www.theverge.com/2023/5/31/23739084/twitter-elon-musk-api-policy-chilling-academic-research
  * Dadas (2022) Sławomir Dadas. 2022.  Polish Longformer.  Retrieved 2023-10-13 from https://huggingface.co/sdadas/polish-longformer-base-4096/tree/main
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  arXiv:1810.04805 [cs.CL]
  * Face (2023) Hugging Face. 2023.  MTEB Leaderboard.  Retrieved 2023-10-11 from https://huggingface.co/spaces/mteb/leaderboard
  * Frome et al. (2013) Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. 2013.  DeViSE: A Deep Visual-Semantic Embedding Model. In _Advances in Neural Information Processing Systems_ , C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (Eds.), Vol. 26. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2013/file/7cce53cf90577442771720a370c3c723-Paper.pdf
  * Greene et al. (2022) Ryan Greene, Ted Sanders, Lilian Weng, and Arvind Neelakantan. 2022.  New and improved embedding model.  Retrieved 2023-10-11 from https://openai.com/blog/new-and-improved-embedding-model
  * Guo et al. (2013) Weiwei Guo, Hao Li, Heng Ji, and Mona Diab. 2013.  Linking Tweets to News: A Framework to Enrich Short Text Data in Social Media. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Sofia, Bulgaria, 239–249.  https://aclanthology.org/P13-1024
  * Joulin et al. (2015) Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. 2015.  Learning Visual Features from Large Weakly Supervised Data.  arXiv:1511.02251 [cs.CV]
  * Krestel et al. (2015) Ralf Krestel, Thomas Werkmeister, Timur Pratama Wiradarma, Gjergji Kasneci, and Gjergji Kasneci. 2015.  Tweet-Recommender: Finding Relevant Tweets for News Articles.  _The Web Conference_ (2015).  https://doi.org/10.1145/2740908.2742716
  * Kwak et al. (2010) Haewoon Kwak, Changhyun Lee, Hosung Park, and Sue Moon. 2010.  What is Twitter, a Social Network or a News Media?. In _Proceedings of the 19th International Conference on World Wide Web_ (Raleigh, North Carolina, USA) _(WWW ’10)_. Association for Computing Machinery, New York, NY, USA, 591–600.  https://doi.org/10.1145/1772690.1772751
  * Landis and Koch (1977) J. Richard Landis and Gary G. Koch. 1977.  The Measurement of Observer Agreement for Categorical Data.  _Biometrics_ 33, 1 (mar 1977), 159.  https://doi.org/10.2307/2529310
  * Liang et al. (2022) Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. 2022.  Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning.  Retrieved 2023-10-13 from https://openreview.net/forum?id=S7Evzt9uit3
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  arXiv:1907.11692 [cs.CL]
  * Mele et al. (2019) Ida Mele, Seyed Ali Bahrainian, and Fabio Crestani. 2019.  Event mining and timeliness analysis from heterogeneous news streams.  _Information Processing & Management_ 56, 3 (2019), 969–993.  https://doi.org/10.1016/j.ipm.2019.02.003
  * Mogadala et al. (2017) Aditya Mogadala, Dominik Jung, and Achim Rettinger. 2017.  Linking Tweets with Monolingual and Cross-Lingual News using Transformed Word Embeddings.  _CoRR_ abs/1710.09137 (2017).  arXiv:1710.09137 http://arxiv.org/abs/1710.09137
  * Mroczkowski et al. (2021) Robert Mroczkowski, Piotr Rybak, Alina Wróblewska, and Ireneusz Gawlik. 2021.  HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish. In _Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing_. Association for Computational Linguistics, Kiyv, Ukraine, 1–10.  https://www.aclweb.org/anthology/2021.bsnlp-1.1
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12 (2011), 2825–2830.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global Vectors for Word Representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Doha, Qatar, 1532–1543.  https://doi.org/10.3115/v1/D14-1162
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning Transferable Visual Models From Natural Language Supervision.  arXiv:2103.00020 [cs.CV]
  * Socher and Fei-Fei (2010) Richard Socher and Li Fei-Fei. 2010.  Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. 966–973.  https://doi.org/10.1109/CVPR.2010.5540112
  * Spitz and Gertz (2018) Andreas Spitz and Michael Gertz. 2018.  Exploring Entity-centric Networks in Entangled News Streams. In _Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW ’18_. ACM Press, Lyon, France, 555–563.  https://doi.org/10.1145/3184558.3188726
  * Suarez et al. (2018a) Axel Suarez, Dyaa Albakour, David Corney, Miguel Martinez, and José Esquivel. 2018a.  A Data Collection for Evaluating the Retrieval of Related Tweets to News Articles.  In _Advances in Information Retrieval_ , Gabriella Pasi, Benjamin Piwowarski, Leif Azzopardi, and Allan Hanbury (Eds.). Vol. 10772. Springer International Publishing, Cham, 780–786.  https://doi.org/10.1007/978-3-319-76941-7_76 Series Title: Lecture Notes in Computer Science.
  * Suarez et al. (2018b) Axel Suarez, Dyaa Albakour, David Corney, Miguel Martinez, Miguel Martinez, Miguel Martinez, Miguel Martinez, Miguel Ángel Martínez, and José Esquivel. 2018b.  A Data Collection for Evaluating the Retrieval of Related Tweets to News Articles.  _European Conference on Information Retrieval_ (2018).  https://doi.org/10.1007/978-3-319-76941-7_76
  * Syed et al. (2021) Munira Syed, Daheng Wang, Meng Jiang, Meng Jiang, Oliver Conway, Vishal Juneja, Sriram Subramanian, Sriram Subramanian, Nitesh V. Chawla, Nitesh V. Chawla, Nitesh V. Chawla, and Nitesh V. Chawla. 2021.  Unified Representation of Twitter and Online News Using Graph and Entities.  _Frontiers in Big Data_ (2021).  https://doi.org/10.3389/fdata.2021.699070
  * Szmyd et al. (2023) Wojciech Szmyd, Alicja Kotyla, Michał Zobniów, Piotr Falkiewicz, Jakub Bartczuk, and Artur Zygadło. 2023.  TrelBERT: A pre-trained encoder for Polish Twitter. In _Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023)_. Association for Computational Linguistics, Dubrovnik, Croatia, 17–24.  https://aclanthology.org/2023.bsnlp-1.3
  * Zhang et al. (2022) Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. 2022.  Contrastive Learning of Medical Visual Representations from Paired Images and Text.  arXiv:2010.00747 [cs.CV]
  * Zhao et al. (2011) Wayne Xin Zhao, Jing Jiang, Jianshu Weng, Jing He, Ee-Peng Lim, Hongfei Yan, and Xiaoming Li. 2011.  Comparing Twitter and Traditional Media Using Topic Models. In _Advances in Information Retrieval_ , Paul Clough, Colum Foley, Cathal Gurrin, Gareth J. F. Jones, Wessel Kraaij, Hyowon Lee, and Vanessa Mudoch (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 338–349.
  * Zhong et al. (2023) Yuanyi Zhong, Haoran Tang, Junkun Chen, Jian Peng, and Yu-Xiong Wang. 2023.  Is Self-Supervised Contrastive Learning More Robust Than Supervised Learning?  https://openreview.net/forum?id=FPdDFUVYVPl

##  Appendix A Keywords used for tweet search

###  A.1. Polish keywords

ukraina, ukraiński, rosja, rosyjski, putin, sowiecki, kreml/kremlowski, mińsk,
NATO, kijów, moskwa, zełeński, sankcje, rubel, donbas, UKR, RUS, #ukraine,
#ukraina, #russia, #rosja, #war, #wojna, #warinukraine, #wojnawukrainie,
#wojnanaukrainie, #standwithukraine, #ukrainerussiawar, #putin, #putin,
#ukrainewar, #putinwarcrimes, #ukraineunderattack, #russianaggression

###  A.2. English keywords

ukraine, ukrainian, russia, russian, putin, soviet, kremlin, minsk, NATO,
kyiv/kiev, moscow, zelensky, sanctions, ruble, donbas, UKR, RUS, #ukraine,
#russia, #war, #warinukraine, #standwithukraine, #ukrainerussiawar, #putin,
#ukrainewar, #putinwarcrimes, #ukraineunderattack, #russianaggression