# Temporal Validity Change Prediction

Georg Wenzel
University of Innsbruck, Austria
<EMAIL_ADDRESS>
&Adam Jatowt
University of Innsbruck, Austria
<EMAIL_ADDRESS>

###### Abstract

Temporal validity is an important property of text that is useful for many
downstream applications, such as recommender systems, conversational AI, or
story understanding. Existing benchmarking tasks often require models to
identify the temporal validity duration of a single statement. However, in
many cases, additional contextual information, such as sentences in a story or
posts on a social media profile, can be collected from the available text
stream. This contextual information may greatly alter the duration for which a
statement is expected to be valid. We propose _Temporal Validity Change
Prediction_ , a natural language processing task benchmarking the capability
of machine learning models to detect contextual statements that induce such
change. We create a dataset consisting of temporal target statements sourced
from Twitter and crowdsource sample context statements. We then benchmark a
set of transformer-based language models on our dataset. Finally, we
experiment with temporal validity duration prediction as an auxiliary task to
improve the performance of the state-of-the-art model.

##  1 Introduction

In human communication, temporal properties are frequently underspecified when
authors assume that the recipient can infer them via commonsense reasoning.
For example, when reading “I am moving on Saturday”, a reader is likely to
assume the person will be busy for most of the day. On the other hand, when
reading “I will make a sandwich on Sunday”, this is likely to only take up a
fraction of the author’s day and may not impact other plans. Such reasoning is
referred to as _temporal commonsense_ (TCS) _reasoning_ (Wenzel and Jatowt,
2023).

_Temporal validity_ (Almquist and Jatowt, 2019; Hosokawa et al., 2023; Lynden
et al., 2023) is a property that is vital for the proper understanding of a
text. The temporal validity of a statement, i.e., whether it contains valid
information at a given time, often requires TCS reasoning to resolve. For
example, in determining whether a statement like “I am driving home from work”
is still valid after five hours, we may use our prior understanding of the
typical duration of related events, such as commuter traffic. While the amount
of research into TCS and, to a degree, temporal validity, has risen over the
past years (Wenzel and Jatowt, 2023), there are still several properties of
temporal validity that have not been considered in previous research. One such
property is the impact of _context_ on the temporal validity duration of a
statement. For example, the sentence “I am driving home from work” may be
valid for a longer time when followed by a statement such as “There is a
massive traffic jam”.

Figure 1: A visualization of the TVCP task

To model this problem, we propose a new NLP task format called _Temporal
Validity Change Prediction_ (TVCP), which requires reasoning over whether a
context statement changes the temporal validity duration of a target
statement. The task is visualized in Figure 1. We propose the following
applications for such a system.

Timeline Prioritization: Social media services such as Twitter rely on
recommender systems to prioritize the vast amount of content that their users
produce. One possible way to improve the prioritization of content is to
consider its temporal validity (Takemura and Tajima, 2012; Koul et al., 2022),
as users are likely to care about current and relevant information over more
general, stationary statements. TVCP can be used to leverage the stream of
social media posts by a given user as possible context to better estimate the
temporal validity duration of a previously observed post.

User Status Tracking: Similarly, the content of a user’s posts on social media
could be utilized for other analytical or business purposes, such as
predicting revenue streams (Asur and Huberman, 2010; Deng et al., 2011; Lassen
et al., 2014; Lu et al., 2014) or identifying trends in a community’s or an
individual user’s behaviour (Li et al., 2018; Abe et al., 2018; Shen et al.,
2020). TVCP could be used to identify posts that refer to previous temporal
information, to detect chains of thought about topics that may not be self-
contained.

Conversational AI: Foundation models, such as ChatGPT (Ouyang et al., 2022)
and Bard (Manyika, 2023), could use the temporal validity of statements
provided by the user to keep track of knowledge that is still relevant to the
conversation. Using TVCP, new messages by the user could be evaluated to
adjust the expected temporal validity period of previously learned facts. This
is especially relevant as initial reports indicate that foundation models may
struggle with TCS reasoning (Bian et al., 2023).

Our main contributions are the following:

  1. 1.

We define a novel NLP task (TVCP). This ternary classification task requires
models to predict the impact of a context statement on a target statement’s
temporal validity duration.

  2. 2.

We build a dataset of tuples consisting of time-sensitive _target statements_
, as well as _follow-up statements_ that act as context for our task.

  3. 3.

We evaluate the performance of existing pre-trained _language models_ (LMs) on
our dataset, including models fine-tuned on other TCS tasks as well as
ChatGPT.

  4. 4.

We propose an augmentation to the training process that leverages temporal
validity duration information to help improve the performance of the state-of-
the-art classifier.

##  2 Related Work

###  2.1 Temporal Commonsense Reasoning

TCS reasoning is often considered one of several categories of commonsense
reasoning (Storks et al., 2019a; Bhargava and Ng, 2022). A major driver of
research specifically into TCS appears to have been the transformer
architecture (Vaswani et al., 2017) and resulting LMs. In recent years,
several datasets that specifically aim to benchmark TCS understanding have
been published (Zhou et al., 2019; Ning et al., 2020; Zhang et al., 2020; Qin
et al., 2021; Zhou et al., 2021), while ROCStories (Mostafazadeh et al., 2016)
appears to be the only dataset focussing on this type of reasoning before the
publication of the transformer architecture. Small adjustments to transformer-
based LMs are often proposed as state-of-the-art solutions for these datasets
(Pereira et al., 2020; Yang et al., 2020; Zhou et al., 2020; Pereira et al.,
2021; Kimura et al., 2021; Zhou et al., 2021, 2022; Cai et al., 2022; Yu et
al., 2022). Similarly, temporalized transformer models are popular solutions
for tasks such as document dating or semantic change detection (Rosin and
Radinsky, 2022; Rosin et al., 2022; Wang et al., 2023).

The TCS taxonomy defined by Zhou et al. (2019) is frequently referenced. It
contains the five dimensions of _duration_ (how long an event takes),
_temporal ordering_ (typical order of events), _typical time_ (when an event
happens), _frequency_ (how often an event occurs) and _stationarity_ (whether
a state holds for a very long time or indefinitely).

###  2.2 Temporal Validity

Method | Task | Data Source | Duration Bias | Model | # Samples
---|---|---|---|---|---
Takemura and Tajima (2012) | $\textsc{TV}_{d}$ | Twitter | N/A | SVC | 9,890
Almquist and Jatowt (2019) | $\textsc{TV}_{d}$ | Blogs, News, Wikipedia | _years_ | SVC | 1,762
Hosokawa et al. (2023) | TNLI | Image Captions |  _seconds_ 111Based on analysis of a sample. $\textsc{TV}_{d}$ labels are not available for the full dataset. | LM | 10,659
Lynden et al. (2023) | $\textsc{TV}_{d}$ | WikiHow | _hours_ | LM | 339,184
Ours | TVCP | Twitter | _hours_ | LM | 5,055
Table 1: Summary of related work

Compared to TCS reasoning, temporal validity in text is a less well-researched
field. It effectively combines three dimensions of the taxonomy by Zhou et al.
(2019): _Stationarity_ , to reason about whether a statement contains temporal
information, _typical time_ , to reason about when the temporal information
occurs, and _duration_ , to reason about how long the temporal information
takes to resolve.

Takemura and Tajima (2012) classify the lifetime duration of tweets, i.e., the
informational value of a tweet over time. They use handcrafted, domain-
specific features to train a _support vector classifier_ (SVC) on supervised
samples.

Almquist and Jatowt (2019) similarly design features to estimate the temporal
validity duration of sentences collected from news, blog posts, and Wikipedia
using SVCs. Their features contain general properties such as the word- or
sentence length, but also more complex ones, such as latent semantic analysis.

Hosokawa et al. (2023) define the _Temporal Natural Language Inference_ (TNLI)
task. The goal of TNLI is to determine whether the temporal validity of a
given hypothesis sentence is supported by a premise sentence.

Lynden et al. (2023) build a large dataset of human annotations specifying the
duration required to perform various actions on WikiHow as well as their
respective temporal validity durations.

###  2.3 Comparison with Related Work

Table 1 shows the most closely related research. As noted, our dataset is
based on the proposed TVCP task, whereas previous work was based on the
$\textsc{TV}_{d}$ and TNLI tasks. All tasks are described in more detail in
Section 3.

Another prominent distinctive attribute is the text source and the resulting
temporal validity duration bias. For example, sentences sourced from news or
Wikipedia articles often appear to be valid for years or longer. On the other
hand, image captions may only be valid for a few seconds or minutes. We
decided to source our sentences from Twitter due to its alignment with our
downstream use cases. Similar to Lynden et al. (2023), our collected temporal
information tends to be valid for a few hours.

We follow recent research by evaluating our dataset using transformer-based
LMs, whereas earlier approaches relied on methods such as SVCs.

Except for the CoTAK dataset (Lynden et al., 2023), the datasets tend to be
relatively small. As crowdsourcing is used in all datasets referenced in Table
1 to annotate text spans with commonsense information, the costs of dataset
creation can quickly escalate. In addition, we ask participants to create
examples of follow-up statements that cause temporal validity change. This
approach further restricts the overall size of our dataset due to the relative
difficulty of the task.

##  3 Task

###  3.1 Defining Temporal Validity

Temporal validity, in essence, is simply the time-dependent validity of a
text. As shown in Equation 1, the temporal validity of a statement $s$ at a
time $t$ is a binary value that determines whether the information in $s$ is
valid at the given time.

$\textsc{TV}(s,t)=\begin{cases}\text{True}&\text{if information in $s$ is
valid at $t$},\\\ \text{False}&\text{otherwise}\end{cases}$ (1)

In some previous research (Hosokawa et al., 2023; Lynden et al., 2023), the
scope of evaluated temporal information is limited to actions, such as “I am
_baking bread_ ”. However, we note that other types of temporal information
exist, such as events (e.g., in the sentence “ _Job interview_ tomorrow”) or
temporary states (e.g., in the sentence “It _is nice out_ today”). In an
analysis of a subset of our collected statements, shown in Figure 2, we find
that these alternative types of temporal information constitute a significant
portion (28%) of samples. Additionally, one-third of sampled statements
contained at least two distinct pieces of temporal information with differing
temporal validity spans. This indicates that the true scope of determining the
temporal validity of a text may exceed what current datasets are benchmarking.

Figure 2: Distribution of different types of temporal information in a sample
of our dataset

We assume that the temporal validity of stationary statements is constant for
any given timestamp $t$. A stationary statement may be continuously true
(e.g., “Japan lies in Asia”), or continuously false (e.g., “Japan lies in
Europe”). This includes information that is fully contained in the past (e.g.,
“I went to the bank yesterday”). In general, we do not expect the validity of
such a statement to change.

For contemporary or future information, we assume the statement is valid from
the moment of sentence conception until the information is no longer ongoing.
We include the duration of the information, rather than just its occurrence
time, as humans are likely to still consider durative information relevant
while it is ongoing. For example, we may reason that the statement “I will
take a shower at 8 p.m.” still has informational value at 8:05 p.m., as it
allows us to infer the current action of the author.

###  3.2 Formalizing Existing Tasks

####  3.2.1 Temporal Validity Duration Estimation

_Temporal Validity Duration Estimation_ ($\textsc{TV}_{d}$) is the primary
task that is evaluated in temporal validity research (Takemura and Tajima,
2012; Almquist and Jatowt, 2019; Lynden et al., 2023). The goal is to estimate
the duration for which the statement is valid, starting at the statement
creation time. We formalize this task in Equation 2, where $t_{s}$ is the
timestamp at which the statement $s$ is created.

$\textsc{TV}_{d}(s)=\max_{t\geq
t_{s}}\\{t\,|\,\textsc{TV}(s,t)=\text{True}\\}$ (2)

The $\textsc{TV}_{d}$ task is useful in downstream applications such as social
media, where information on the posting time of a statement is readily
available and can be used to infer the span during which the statement is
valid.

####  3.2.2 Temporal Natural Language Inference

The goal of TNLI (Hosokawa et al., 2023) is to infer whether a statement is
temporally valid, given additional context, using typical NLI terminology
(MacCartney, 2009; Storks et al., 2019b). TNLI requires a _hypothesis
statement_ (that we call _target statement_ , or $s_{t}$) and a _premise
sentence_ (that we call _follow-up statement_ , or $s_{f}$). Implicitly, the
inference takes place at $t_{s_{f}}$, that is, the posting time of the follow-
up statement, but no explicit duration information is required to solve this
task. Formally, we define TNLI in Equation 3 (SUO = supported, INV =
invalidated, UNK = unknown), where $\textsc{TV}^{c}(s,t)$ is the temporal
validity of a statement $s$ at a time $t$ given context $c$. The UNK class is
assigned in cases where $\textsc{TV}^{s_{f}}(s_{t},t_{s_{f}})$ is neither
clearly supported nor invalidated by the context.

$\textsc{TNLI}(s_{t},s_{f})=\begin{cases}\textsc{SUO}&\textsc{TV}^{s_{f}}(s_{t},t_{s_{f}})=\text{True}\\\
\textsc{INV}&\textsc{TV}^{s_{f}}(s_{t},t_{s_{f}})=\text{False}\\\
\textsc{UNK}&\textsc{TV}^{s_{f}}(s_{t},t_{s_{f}})=\text{Unclear}\end{cases}$
(3)

Unlike $\textsc{TV}_{d}$, this task format lends itself to downstream
applications such as story understanding, wherein a larger text stream of
individual statements is provided with no clear explicit notion of time
passing between each sentence (e.g., in a book).

###  3.3 Temporal Validity Change Prediction

We propose _Temporal Validity Change Prediction_ (TVCP), which combines ideas
from both the inference- and duration-based tasks. Like TNLI, we require
$s_{t}$ and $s_{f}$ for classification, and determine a ternary label that
provides information about the impact of $s_{f}$ on $s_{t}$. Unlike TNLI, our
goal is to predict a _change_ in the temporal validity _duration_ of $s_{t}$.

We consider TVCP a necessary step in accurately determining a statement’s
temporal validity. Simply estimating the duration of the statement alone may
not yield very precise results when it is, as in many use cases, extracted
from a rich context, such as a book, a story, a news article, a step-by-step
guide, or a social media profile. In these cases, surrounding information may
provide additional context that could lead us to a different $\textsc{TV}_{d}$
estimate. Simply concatenating $s_{t}$ and $s_{f}$ may lead to the
classification of temporal information within $s_{f}$, which is undesired. Our
segmentation of $s_{t}$ and $s_{f}$ into different semantic roles, similar to
TNLI, prevents this issue.

Formally, we define TVCP in Equation 4 (DEC = decreased, UNC = unchanged, INC
= increased), where $\textsc{TV}_{d}^{c}(s)$ is the temporal validity duration
of a statement $s$ given context $c$. Figure 3 shows a concrete comparative
example of the goal of all three tasks.

$\textsc{TVCP}(s_{t},s_{f})=\begin{cases}\textsc{DEC}&\textsc{TV}_{d}(s_{t})>\textsc{TV}_{d}^{s_{f}}(s_{t})\\\
\textsc{UNC}&\textsc{TV}_{d}(s_{t})=\textsc{TV}_{d}^{s_{f}}(s_{t})\\\
\textsc{INC}&\textsc{TV}_{d}(s_{t})<\textsc{TV}_{d}^{s_{f}}(s_{t})\end{cases}$
(4)

Figure 3: An example of $\textsc{TV}_{d}$, TNLI and TVCP

Since TVCP is a signal measuring the difference between $\textsc{TV}_{d}$
with- and without $s_{f}$, respectively, a more fine-grained $\textsc{TV}_{d}$
classification increases the number of TVCP instances that can be detected. On
the other hand, evaluating $\textsc{TV}_{d}$ on a very fine-grained scale may
be more difficult for both models and humans (Honda et al., 2022), and the
resulting uncertainty and inaccuracies could lead to a degradation of the
system as a whole.

In our sample analysis, we find that temporal validity change generally occurs
along two axes, shown in Figure 4. The first dimension is _implicit_ versus
_explicit_ change. For example, an appointment may be postponed, which is an
explicit change. On the other hand, the author may note in a follow-up
statement that the appointment is in a sleep laboratory, which may cause us to
re-evaluate for how long the original statement is valid, although the
information itself has not changed.

Figure 4: Dimensions of temporal validity change. The frequency of each
category for DEC and INC classes in our sample is appended.

The second dimension is a change to the _occurrence time_ versus the
_duration_ of the information. For example, a flight may be delayed, in which
case the occurrence time changes. Alternatively, the flight might have to be
re-routed mid-air due to bad weather, in which case the duration changes.

In our sample, we find that all four categories are present to a reasonable
degree in both the DEC and INC classes. Generally, changes to the duration
tend to be slightly more frequent than changes to the occurrence time. This
makes sense, as the occurrence time is a dimension that is only present when
the information occurs in the future, whereas the duration of temporal
information can change irrespective of the occurrence time.

##  4 Dataset

We create a dataset for training and benchmarking TVCP, where each sample is a
quintuple
$<s_{t},s_{f},\textsc{TV}_{d}(s_{t}),\textsc{TV}_{d}^{s_{f}}(s_{t}),\textsc{TVCP}(s_{t},s_{f})>$.

$s_{t}$ is collected by querying the Twitter API for tweets with no external
context (e.g., no tweets that are retweets or replies, or tweets containing
media). We apply several pre-processing steps to remove tweets whose content
may not be self-contained. We aim to minimize spam and offensive content by
applying publicly available LMs and word-list-based filters. To decrease the
number of stationary statements, we employ an ensemble classifier based on the
Almquist2019 (Almquist and Jatowt, 2019) and CoTAK datasets and select the
most likely statements to contain temporal information. Finally, crowdworkers
can tag any remaining stationary samples during the annotation process. A
summary of our pre-processing pipeline is shown in Figure 5. Our code,
including all preprocessing steps, is published under the Apache 2.0 licence.

Figure 5: A summary of our tweet collection pipeline

For each target statement, we ask two crowdworkers to estimate
$\textsc{TV}_{d}(s_{t})$ from the logarithmic class design shown in Equation
5, which is modelled after human timeline understanding (Jatowt and Au Yeung,
2011; Varshney and Sun, 2013; Howard, 2018). If the annotators disagreed, we
supplied a third vote. We discarded any tweets that were annotated as _less
than one minute_ , _more than one month_ , or _no time-sensitive information_
(i.e., stationary), as well as tweets where no majority agreement could be
reached. Of 2,996 annotated target tweets, 571 were discarded without a third
annotation, 867 were added without a third annotation, 546 were discarded
after providing a third vote, and 1,012 were added after providing a third
vote. The distribution of resulting $\textsc{TV}_{d}$ labels before temporal
validity change is shown in Figure 6.

$\begin{split}t\in\\{<\textrm{1 minute},\textrm{1-5 minutes},\textrm{5-15
minutes},\\\ \textrm{15-45 minutes},\textrm{45 minutes-2 hours},\textrm{2-6
hours},\\\ \textrm{more than 6 hours},\textrm{1-3 days},\textrm{3-7 days},\\\
\textrm{1-4 weeks},\textrm{more than 1 month}\\}\\\ \end{split}$ (5)

Figure 6: Distribution of $\textsc{TV}_{d}$ labels (before temporal validity
change) in our dataset

Both $s_{f}$ and $\textsc{TV}^{s_{f}}_{d}(s_{t})$ were provided by a separate
set of crowdworkers, given $s_{t}$ and $\textsc{TV}_{d}(s_{t})$ as an input.
In total, we collected 5,055 samples from 1,685 target statements. In Figure
7, we plot the _temporal validity change delta_ , which is the class distance
between the original and the updated $\textsc{TV}_{d}$ estimate. We find that,
in most cases, the temporal validity duration of a target statement is shifted
only by one class.

Figure 7: Temporal validity change delta distribution

All crowdsourcing tasks were set up on Amazon Mechanical Turk, using
qualification tests, participation criteria, and manual verification of
results to ensure high-quality samples (see Appendix A). We publish the
resulting dataset for public use under the CC BY 4.0 licence. In accordance
with the Twitter developer
policy222https://developer.twitter.com/en/developer-terms/policy, accessed
12.10.2023, we only publish the Tweet IDs of sourced statements. This also
means original tweet authors retain the ability to delete their content,
effectively removing it from the dataset.

##  5 Experiments

###  5.1 Language Models

We evaluate a set of transformer-based LMs on our dataset. We test four
different archetypes in total:

  * •

TransformerClassifier: Builds a hidden representation from the sentence-
embedding token of the concatenation of $s_{t}$ and $s_{f}$.

  * •

SiameseClassifier: Builds a hidden representation from the concatenated
embeddings $[h_{s_{t}},h_{s_{f}},h_{s_{t}}-h_{s_{f}},h_{s_{t}}\otimes
h_{s_{f}}]$, where $h_{s_{t}}$ and $h_{s_{f}}$ are the sentence-embedding
tokens of the target- and follow-up statement, respectively (Bromley et al.,
1993; Nandy et al., 2020).

  * •

SelfExplain (Sun et al., 2020): Builds a hidden representation from the
embeddings of spans between arbitrary tokens in either $s_{t}$ or $s_{f}$,
selected by the model.

  * •

ChatGPT: A chain-of-thought (Wei et al., 2022) reasoning prompt based on few-
shot learning (one sample per TVCP class), passed to the gpt-3.5-turbo model
via the OpenAI API. 333This call uses the most recent GPT3.5 model. We
collected ChatGPT responses in July 2023.

For the TransformerClassifier and SiameseClassifier pipelines, we evaluate
BERT-base-uncased (Kenton and Toutanova, 2019; 110M parameters) and RoBERTa-
base (Liu et al., 2019; 125M parameters) embeddings. For SelfExplain, we only
test the original implementation with RoBERTa-base embeddings. To evaluate
transfer learning from other TCS tasks, we test the TransformerClassifier
pipeline on regular BERT-base-uncased pre-training weights as well as two
variants TacoLM (Zhou et al., 2020) and CoTAK (Lynden et al., 2023), which
have the same underlying architecture, but use weights fine-tuned on existing
TCS datasets.

We use the AdamW optimizer (Loshchilov and Hutter, 2018) with
$\varepsilon=\textrm{1e-8}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$,
$\textrm{weight\\_decay}=0.01$. We optimize for cross-entropy loss.
SelfExplain adds an additional loss parameter in the form of squared span-
attention weights, to encourage the model to more sharply choose which spans
should be used to build the hidden representation.

We set the dropout probabilities and learning rates as defined in Table 2 as a
result of our hyperparameter optimization (see Appendix C). For all models,
the hidden embedding size is 768. For some RoBERTa-based models, we freeze
embedding layers (i.e., only fine-tune intermediate and classification
weights), as training all parameters leads to poor performance.

Model | Dropout | LR | Frozen
---|---|---|---
TF - BERT | 0.25 | 1e-4 | False
S - BERT | 0.25 | 1e-4 | False
TF - RoBERTa | 0.25 | 1e-3 | True
S - RoBERTa | 0.10 | 1e-4 | True
SelfExplain | 0.00 | 2e-5 | False
Table 2: Hyperparameter settings for different models. TF =
TransformerClassifier, S = SiameseClassifier

###  5.2 Multitask Implementation

For all archetypes except ChatGPT, we provide a second implementation, in
which we add two regression layers that aim to respectively predict
$\textsc{TV}_{d}(s_{t})$ and $\textsc{TV}^{s_{f}}_{d}(s_{t})$ from the same
hidden representation. For these layers, we calculate the mean squared error
between a single output neuron and a linear mapping of the $\textsc{TV}_{d}$
class index to the range $[0,1]$. Our intuition is that embeddings with an
understanding of $\textsc{TV}_{d}$ may be better suited for TVCP. Inspiration
for this approach are models that utilize the interplay between temporal
dimensions to improve the TCS reasoning performance in LMs, such as SymTime
(Zhou et al., 2021) or SLEER (Cai et al., 2022).

###  5.3 Evaluation

We evaluate two metrics, accuracy and _exact match_ (EM). Accuracy is simply
the fraction of correctly classified samples. EM is the fraction of _target
statements_ for which all three samples were correctly classified. This metric
punishes inconsistency in the model more strictly, thus providing a better
view of the true performance and task understanding of each model (Wenzel and
Jatowt, 2023), while disincentivizing shallow reasoning behaviours commonly
seen in transformer models (Helwe et al., 2021; Tan et al., 2023).

We report the mean EM and accuracy across a five-fold cross-validation split.
Each evaluation consists of 70% training data, 10% validation data, and 20%
test data. If the validation EM does not exceed the best previously observed
value for 5 consecutive epochs, we stop training. The model with the best
validation EM is used for evaluation on the test set. The results are shown in
Table 3.

Model |  $\overline{\textrm{Acc}}$ (+ MT) |  $\overline{\textrm{EM}}$ (+ MT)
---|---|---
TF - RoBERTa |  $64.0$ $(+1.5)$ |  $21.2$ $(+2.5)$
ChatGPT |  $66.3\;\;$ (N/A) |  $29.3\;\;$ (N/A)
S - RoBERTa |  $78.7$ $(+1.1)$ |  $48.2$ $(+2.1)$
TF - CoTAK |  $83.2$ $(+0.6)$ |  $58.2$ $(+1.4)$
S - BERT |  $83.8$ $(-0.3)$ |  $59.1$ $(-1.5)$
TF - TacoLM |  $83.5$ $(+1.4)$ |  $59.1$ $(+2.9)$
TF - BERT |  $84.8$ $(-0.2)$ |  $61.2$ $(+0.9)$
SelfExplain |  $88.5$ $(+1.1)$ |  $69.8$ $(+2.8)$
Table 3: Model evaluation results, sorted by mean EM score. TF =
TransformerClassifier, S = SiameseClassifier, MT = Multitask Implementation

We note a positive impact on the EM score from implementing multitasking in
all models except the Siamese architecture with BERT-based embeddings. We use
bootstrapping to calculate the statistical significance of implementing
multitask learning on the best-performing model, SelfExplain, evaluating the
number of bootstrap samples in which the multitask implementation outperforms
regular SelfExplain. We find $p=0.0027$ for accuracy, with a 95% confidence
interval of $[0.0036,0.0192]$. For EM, $p=0.0025$, with a 95% confidence
interval of $[0.0089,0.0487]$.

The use of weights from other TCS tasks does not seem to have a positive
impact on the performance of the TransformerClassifier pipeline. It is
possible that, although the resulting embeddings are more aligned with
temporal properties (Zhou et al., 2020), other important information in the
embeddings is lost, leading to an overall decreased performance.

Due to some RoBERTa-based models having frozen embedding layers, the baseline
performance by RoBERTa is much worse, but it improves much more when switching
to the SiameseClassifier implementation. We hypothesize that RoBERTa’s
sentence embedding token, <s>, may contain less information about the full
sequence than BERT’s [SEP] token due to the lack of a next-sentence-prediction
task during pre-training.

ChatGPT ranks among the lower-performing models, which is consistent with
other studies on TCS understanding (Bian et al., 2023). Its shortcomings may
be due to the few-shot learning approach and a lack of knowledge about dataset
specifics traits, which a trained classifier could leverage. Additionally, we
do not specify our class design in the ChatGPT prompt, which could make it
harder for ChatGPT to isolate the UNC class.

To evaluate the impact of training data quantity on classifier performance, we
train our best-performing classifier (SelfExplain with multitasking, which we
dub MultiTask) on a single train-val-test split (80%/10%/10%) with different
amounts of training data. The results can be seen in Figure 8. We find that
performance increases as more data is used for training, but this effect
starts to diminish as we approach 100% of our training data.

Figure 8: Training data vs. performance metrics in MultiTask

In testing SelfExplain and MultiTask on various temporal validity change
deltas (Figure 9), we find they perform comparably on the UNC class, but
MultiTask slightly outperforms SelfExplain on all delta values greater than
zero. While ChatGPT’s subpar performance in the UNC class can partially be
attributed to prompt design, it continues to lag far behind other models in
the DEC and INC classes.

Figure 9: Temporal validity change delta vs. accuracy in MultiTask,
SelfExplain and ChatGPT

All models were trained and evaluated on an MSI GeForce RTX 3080 GAMING X TRIO
10G GPU using CUDA 11.7. Training and evaluation of all final models as well
as hyperparameter tests took around 15 GPU hours.

##  6 Conclusion and Future Work

In this work, we have introduced TVCP, an NLP task, to aid in the accurate
determination of the temporal validity duration of text by incorporating
surrounding context. We create a benchmark dataset for our task and provide a
set of baseline evaluation results for our dataset. We find that the
performance of most classifiers can be improved by explicitly incorporating
the temporal validity duration as a loss signal during training to improve the
resulting embeddings. Despite the impressive feats performed by foundation
models, we report, similar to previous work (Bian et al., 2023), poor
performance in the TCS domain. These findings show that users should carefully
evaluate whether a model like ChatGPT properly understands a given task before
choosing it over smaller, fine-tuned LMs. We hypothesize that the performance
of all models could further increase with additional training data.

Possible future work includes using the provided dataset and classifiers to
collect a larger number of TVCP samples and annotating them with an updated
temporal validity duration. A comparison of context-aware $\textsc{TV}_{d}$
classifiers with prior models, like those by Almquist and Jatowt (2019), would
shed light on the significance of accurate semantic segmentation between
target and context. Similarly, the use of our dataset for generative
approaches could be explored, for example, in the context of generative
adversarial networks. For our multitasking implementation, directions for
future work could be changes to hyperparameters such as the weight of the
auxiliary loss, changes to the definition of the auxiliary task (e.g., log-
scaled regression or ordinal classification), or even entirely new auxiliary
tasks. Finally, current methods face limitations due to the effort of manual
removal of stationary samples (Almquist and Jatowt, 2019; ours) or altering
task definitions to avoid them (Hosokawa et al., 2023; Lynden et al., 2023).
Research into models differentiating temporal and stationary information could
enhance the development and definition of future TCS reasoning tasks.

## Limitations

Although we focus on creating a reproducible training- and evaluation
environment, some variables are out of our control. For example, bit-wise
reproducibility is only guaranteed on the same CUDA toolkit version and when
executed on a GPU with the same architecture and the same number of streaming
multiprocessors. This means that an exact reproduction of the models discussed
in this article may not be possible. Nevertheless, we expect trends to remain
the same across GPU architectures.

The use of ChatGPT as an example of foundation model performance may be
limiting due to its black box design. In the future, open-source models, such
as Llama 2 (Touvron et al., 2023), could be evaluated to improve the
reproducibility of foundation model performance claims. We chose to benchmark
ChatGPT due to its common use as a baseline and in end-user scenarios, but the
evaluation results may not be transferrable to other foundation models or even
other versions of ChatGPT.

One of the major limitations of our approach is likely the dataset size.
Although a relatively small dataset size is common in TCS reasoning, we find
that our model performance still increases with the amount of training data
used. The existing synthesized context statements in our dataset could be used
to bootstrap an approach for automatically extracting additional samples from
social media to alleviate this issue.

The data we collect is not personal in nature. However, the possibility of
latent demographic biases in our data exists, for example, with respect to
certain language structures or expressions used in the creation of follow-up
statements. This could lead to the propagation of any such bias when the
dataset is used to bootstrap further data collection, which should be
considered in future work.

Our external validity is mainly threatened by two factors. First, our context
statements are crowdsourced. While we apply several steps to ensure the
produced context is sensible, it is unclear whether downstream context, such
as on social media platforms, manifests in similar structures as in our
dataset, with respect to traits such as sentence length, grammaticality, and
phrasing.

Second, similar to how pre-training weights from other TCS tasks do not seem
to improve the classifier performance on our dataset, the weights generated as
part of our training process are likely very task-specific, and may not
generalize well to other tasks or text sources.

Overall, we recommend the use of the TVCP dataset and classifiers for
bootstrapping further research into combining the duration- and inference-
based temporal validity tasks, as well as research into directly predicting
updated temporal validity durations and improving the generalizability to
different text sources, rather than for a direct downstream task application.

## References

  * Abe et al. (2018) Shun Abe, Masumi Shirakawa, Tatsuya Nakamura, Takahiro Hara, Kazushi Ikeda, and Keiichiro Hoashi. 2018.  Predicting the occurrence of life events from user’s tweet history.  In _2018 IEEE 12th International Conference on Semantic Computing (ICSC)_ , pages 219–226. IEEE.
  * Almquist and Jatowt (2019) Axel Almquist and Adam Jatowt. 2019.  Towards content expiry date determination: Predicting validity periods of sentences.  In _European Conference on Information Retrieval_ , pages 86–101. Springer.
  * Asur and Huberman (2010) Sitaram Asur and Bernardo A Huberman. 2010.  Predicting the future with social media.  In _2010 IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology_ , volume 1, pages 492–499. IEEE.
  * Bhargava and Ng (2022) Prajjwal Bhargava and Vincent Ng. 2022.  Commonsense knowledge reasoning and generation with pre-trained language models: a survey.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 12317–12325.
  * Bian et al. (2023) Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. 2023.  Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models.  _arXiv preprint arXiv:2303.16421_.
  * Bromley et al. (1993) Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993.  Signature verification using a" siamese" time delay neural network.  _Advances in neural information processing systems_ , 6.
  * Cai et al. (2022) Bibo Cai, Xiao Ding, Bowen Chen, Li Du, and Ting Liu. 2022.  Mitigating reporting bias in semi-supervised temporal commonsense inference with probabilistic soft logic.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 10454–10462.
  * Deng et al. (2011) Shangkun Deng, Takashi Mitsubuchi, Kei Shioda, Tatsuro Shimada, and Akito Sakurai. 2011.  Combining technical analysis with sentiment analysis for stock price prediction.  In _2011 IEEE ninth international conference on dependable, autonomic and secure computing_ , pages 800–807. IEEE.
  * Helwe et al. (2021) Chadi Helwe, Chloé Clavel, and Fabian M Suchanek. 2021.  Reasoning with transformer-based models: Deep learning, but shallow reasoning.  In _3rd Conference on Automated Knowledge Base Construction_.
  * Honda et al. (2022) Hidehito Honda, Rina Kagawa, and Masaru Shirasuna. 2022.  On the round number bias and wisdom of crowds in different response formats for numerical estimation.  _Scientific Reports_ , 12(1):1–18.
  * Hosokawa et al. (2023) Taishi Hosokawa, Adam Jatowt, Masatoshi Yoshikawa, and Kazunari Sugiyama. 2023.  Temporal natural language inference: Evidence-based evaluation of temporal text validity.  _Proceedings of the 45th European Conference on Information Retrieval (ECIR 2023), Springer LNCS_.
  * Howard (2018) Marc W Howard. 2018.  Memory as perception of the past: compressed time inmind and brain.  _Trends in cognitive sciences_ , 22(2):124–136.
  * Jatowt and Au Yeung (2011) Adam Jatowt and Ching-man Au Yeung. 2011.  Extracting collective expectations about the future from large text collections.  In _Proceedings of the 20th ACM international conference on Information and knowledge management_ , pages 1259–1264.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of naacL-HLT_ , volume 1, page 2.
  * Kimura et al. (2021) Mayuko Kimura, Lis Kanashiro Pereira, and Ichiro Kobayashi. 2021.  Towards a language model for temporal commonsense reasoning.  In _Proceedings of the Student Research Workshop Associated with RANLP 2021_ , pages 78–84.
  * Koul et al. (2022) Yashasvi Koul, Kanishk Mamgain, and Ankit Gupta. 2022.  Lifetime of tweets: a statistical analysis.  _Social Network Analysis and Mining_ , 12(1):101.
  * Lassen et al. (2014) Niels Buus Lassen, Rene Madsen, and Ravi Vatrapu. 2014.  Predicting iphone sales from iphone tweets.  In _2014 IEEE 18th International Enterprise Distributed Object Computing Conference_ , pages 81–90. IEEE.
  * Li et al. (2018) Pengfei Li, Hua Lu, Nattiya Kanhabua, Sha Zhao, and Gang Pan. 2018.  Location inference for non-geotagged tweets in user timelines.  _IEEE Transactions on Knowledge and Data Engineering_ , 31(6):1150–1165.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Lu et al. (2014) Yafeng Lu, Robert Krüger, Dennis Thom, Feng Wang, Steffen Koch, Thomas Ertl, and Ross Maciejewski. 2014.  Integrating predictive analytics and social media.  In _2014 IEEE Conference on Visual Analytics Science and Technology (VAST)_ , pages 193–202. IEEE.
  * Lynden et al. (2023) Steven Lynden, Mehari Heilemariam, Kyoung-Sook Kim, Adam Jatowt, Akiyoshi Matono, Hai-Tao Yu, Xin Liu, and Yijun Duan. 2023.  Commonsense temporal action knowledge (cotak) dataset.  In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM 2023)_.
  * MacCartney (2009) Bill MacCartney. 2009.  _Natural language inference_.  Stanford University.
  * Manyika (2023) James Manyika. 2023.  An overview of bard: an early experiment with generative ai.
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 839–849.
  * Nandy et al. (2020) Abhilash Nandy, Sushovan Haldar, Subhashis Banerjee, and Sushmita Mitra. 2020.  A survey on applications of siamese neural networks in computer vision.  In _2020 International Conference for Emerging Technology (INCET)_ , pages 1–5. IEEE.
  * Ning et al. (2020) Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020.  Torque: A reading comprehension dataset of temporal ordering questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1158–1172.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pereira et al. (2021) Lis Pereira, Fei Cheng, Masayuki Asahara, and Ichiro Kobayashi. 2021.  Alice++: Adversarial training for robust and effective temporal reasoning.  In _Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation_ , pages 373–382.
  * Pereira et al. (2020) Lis Pereira, Xiaodong Liu, Fei Cheng, Masayuki Asahara, and Ichiro Kobayashi. 2020.  Adversarial training for commonsense inference.  In _Proceedings of the 5th Workshop on Representation Learning for NLP (RepL4NLP-2020)_ , pages 55–60. Association for Computational Linguistics.
  * Qin et al. (2021) Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021.  Timedial: Temporal commonsense reasoning in dialog.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7066–7076.
  * Rosin and Radinsky (2022) Guy Rosin and Kira Radinsky. 2022.  Temporal attention for language models.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1498–1508.
  * Rosin et al. (2022) Guy D Rosin, Ido Guy, and Kira Radinsky. 2022.  Time masking for temporal language models.  In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ , pages 833–841.
  * Shen et al. (2020) Lingfeng Shen, Zhuoming Liu, and Xiongtao Zhou. 2020.  Forecasting people’s action via social media data.  In _2020 IEEE International Conference on Big Data (Big Data)_ , pages 5254–5259. IEEE.
  * Storks et al. (2019a) Shane Storks, Qiaozi Gao, and Joyce Y Chai. 2019a.  Commonsense reasoning for natural language understanding: A survey of benchmarks, resources, and approaches.  _arXiv preprint arXiv:1904.01172_ , pages 1–60.
  * Storks et al. (2019b) Shane Storks, Qiaozi Gao, and Joyce Y Chai. 2019b.  Recent advances in natural language inference: A survey of benchmarks, resources, and approaches.  _arXiv preprint arXiv:1904.01172_.
  * Sun et al. (2020) Zijun Sun, Chun Fan, Qinghong Han, Xiaofei Sun, Yuxian Meng, Fei Wu, and Jiwei Li. 2020.  Self-explaining structures improve nlp models.  _arXiv preprint arXiv:2012.01786_.
  * Takemura and Tajima (2012) Hikaru Takemura and Keishi Tajima. 2012.  Tweet classification based on their lifetime duration.  In _Proceedings of the 21st ACM international conference on Information and knowledge management_ , pages 2367–2370.
  * Tan et al. (2023) Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023.  Towards benchmarking and improving the temporal reasoning capability of large language models.  _arXiv preprint arXiv:2306.08952_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Varshney and Sun (2013) Lav R Varshney and John Z Sun. 2013.  Why do we perceive logarithmically?  _Significance_ , 10(1):28–31.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2023) Jiexin Wang, Adam Jatowt, Masatoshi Yoshikawa, and Yi Cai. 2023.  Bitimebert: Extending pre-trained language representations with bi-temporal information.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 812–821.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wenzel and Jatowt (2023) Georg Wenzel and Adam Jatowt. 2023.  An overview of temporal commonsense reasoning and acquisition.  _arXiv preprint arXiv:2308.00002_.
  * Yang et al. (2020) Zonglin Yang, Xinya Du, Alexander M Rush, and Claire Cardie. 2020.  Improving event duration prediction via time-aware pre-training.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3370–3378.
  * Yu et al. (2022) Changlong Yu, Hongming Zhang, Yangqiu Song, and Wilfred Ng. 2022.  Cocolm: Complex commonsense enhanced language model with discourse relations.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1175–1187.
  * Zhang et al. (2020) Li Zhang, Qing Lyu, and Chris Callison-Burch. 2020.  Reasoning about goals, steps, and temporal ordering with wikihow.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4630–4639.
  * Zhou et al. (2019) Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019.  “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics.
  * Zhou et al. (2020) Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020.  Temporal common sense acquisition with minimal supervision.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7579–7589.
  * Zhou et al. (2021) Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2021.  Temporal reasoning on implicit events from distant supervision.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1361–1371.
  * Zhou et al. (2022) Bo Zhou, Yubo Chen, Kang Liu, Jun Zhao, Jiexin Xu, Xiaojian Jiang, and Qiuxia Li. 2022.  Generating temporally-ordered event sequences via event optimal transport.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1875–1884.

##  Appendix A Crowdsourcing Definitions

In this section, we provide details on the crowdsourcing implementation. As
noted, we use Amazon Mechanical Turk to collect crowdsourced data from
participants.

###  A.1 Temporal Validity Duration Estimation

We assume the average layman is not familiar with the term _temporal
validity_. Thus, we define the task as “determining how long the information
within the tweet remains relevant after its publication”, i.e., for how long
the user would consider the tweet timely and relevant. We provide the option
_no time-sensitive information_ , which should be selected when tweets do not
contain any information, when information is not expected to change over time,
or when it is fully contained in the past.

The task is otherwise a relatively straightforward classification task. We
split our dataset into batches of 10 samples that are grouped into a single
_human intelligence task_ (HIT). For each HIT, we offer a compensation of
$\textrm{USD}0.25$, based on an estimated 6-9 seconds of processing time per
individual statement (i.e., 60-90 seconds per HIT). Figures 10 to 13 show the
crowdsourcing layout.

Figure 10: The interface of the temporal validity duration estimation task
Figure 11: The summary section of the temporal validity duration estimation
task guidelines Figure 12: The detailed description of the temporal validity
duration estimation task guidelines Figure 13: The examples section of the
temporal validity duration estimation task guidelines

###  A.2 Follow-Up Content Generation

Compared to the temporal validity duration estimation task, the follow-up
content generation task requires a much more robust understanding of the
overall concept of temporal validity and the respective semantic roles of the
target- and follow-up statements. Hence, we focus on providing a more detailed
explanation of the task. Figures 14 to 16 show the crowdsourcing setup. The
detailed instructions tab is not listed due to its length, but contains
instructions that can also be found in the public repository.

Notably, we labelled the target statement as _context tweet_ rather than
_target tweet_ in this crowdsourcing task to emphasize that participants
should not alter this statement directly, as this was a problem that occurred
somewhat frequently during pilot tests. This contrasts with our formal
definition of TVCP, where providing context is the role of the follow-up
statement.

Each HIT requires participants to provide three follow-up statements, one for
each TVCP class (DEC, UNC, INC). For each HIT, we offer a compensation of
$\textrm{USD}0.35$. We base our compensation on an estimated 30–40 seconds of
processing time per follow-up statement (i.e., 90–120 seconds per HIT) due to
the creative writing involved .

Figure 14: The interface of the follow-up content generation task Figure 15:
The summary section of the follow-up content generation task guidelines Figure
16: The examples section of the follow-up content generation task guidelines

###  A.3 Discouraging Dishonest Activity

In initial pilot runs, we find that many submissions are the result of spam,
dishonest activity, or a complete lack of task understanding, with many
provided annotations being inexplicable by common sense in any possible
interpretation of the statement.

To increase the quality of work on both tasks, we introduced three measures.
First, we required participants to have an overall approval rate of 90% on the
platform, as well as 1,000 approved HITs. Without these requirements, the
amount of blatant spam (e.g., copy-pasted content) increases significantly.

Second, we devised qualification tests for both tasks. Participants had to
determine the temporal validity durations for a set of sample statements to
work on the temporal validity duration estimation task, and determine the
correctness of follow-up statements and their updated duration labels to work
on the follow-up content generation task.

Finally, we vet all participants’ responses individually up to a certain
threshold. For each task, we manually verify the first 20 submissions of each
annotator on their quality. We provide feedback and manually adapt submissions
when they are partially incorrect. If submission quality is appropriate by the
time a participant reaches 20 submitted HITs, we consider them as trusted, and
only spot-check every 5th submission thereafter. If submission quality does
not sufficiently improve at this point, we prohibit the participant from
further working on the task.

Despite these efforts, the follow-up content generation task specifically
still received several low-quality submissions that had to be manually
filtered out and corrected. In future work, a preferable approach may be to
replace the qualification test with an unpaid qualification HIT, in which a
feedback loop between participants and requesters can be established on data
that will not be included in the final dataset, and participants can manually
be assigned a qualification once their quality of work is sufficient.

##  Appendix B ChatGPT Setup

We provide the following system prompt to the ChatGPT API:

> “You are a language model specializing in temporal commonsense reasoning.
> Each prompt contains Sentence A and Sentence B. You should determine whether
> Sentence B changes the expected temporal validity duration of Sentence A,
> i.e., the duration for which the information in Sentence A is expected to be
> relevant to a reader.
>
> To achieve this, in your responses, first, estimate for how long the average
> reader may expect Sentence A to be relevant on its own. Then, consider if
> the information introduced in Sentence B increases or decreases this
> duration. Surround this explanation in triple backticks (“‘).
>
> After your explanation, respond with one of the three possible classes
> corresponding to your explanation: Decreased, Neutral, or Increased.”

After this system prompt, we provide three sample responses, one for each of
the classes. These sample responses are shown in Figure 17.

Sentence A: I’m ready to go to the beach
Sentence B: I forgot all the beach towels are still in the dryer, but I’ll be
ready to go as soon as the dryer’s done running.
Target Class: Increased
Sample Explanation: Going to the beach may take a few minutes to an hour,
depending on the distance. However, if the author first needs to wait on the
dryer to finish in order to retrieve their beach towels, this may take an
additional 30-60 minutes.

Sentence A: taking bad thoughts out of my mind thru grinding my assignments
Sentence B: I just have to get through a short math homework assignment and
memorize a few spelling words so it shouldn’t take long.
Target Class: Decreased
Sample Explanation: Grinding through assignments may take several hours,
depending on the number of assignments to complete. In Sentence B, the author
states they only have a few short assignments remaining, so they may only take
an hour or less to finish them.

Sentence A: Slide to my dm guys, come on
Sentence B: Instagram DMs are such a fun way to communicate.
Target Class: Neutral
Sample Explanation: The author encourages people to direct message them, which
may be relevant for several minutes to a few hours. Sentence B does not change
the duration for which Sentence A is expected to be relevant.

Figure 17: Sample items, target classes, and explanations provided to ChatGPT
for few-shot reasoning

Similar to the crowdsourcing task setup, we use the concept of the _expected
relevance duration_ of the target statement (called Statement A in the ChatGPT
prompt) to explain statement-level temporal validity. Additionally, instead of
prompting the model to classify the sample directly, we ask it to provide an
explanation for its decision based on chain-of-thought reasoning. Wei et al.
(2022) show that chain-of-thought prompting significantly increases several
types of reasoning capabilities, including commonsense, in LLMs. We prompt
ChatGPT to first estimate the temporal validity duration of the target
statement on its own. In a second step, the model should then determine if the
information introduced in the follow-up statement changes this temporal
validity duration. After giving its explanation, the model should respond with
one of the three target classes.

##  Appendix C Hyperparameters

We perform hyperparameter testing regarding dropout probability before the
classification layer ($0.1$, $0.25$, $0.5$), the base learning rate (1e-2,
1e-3, 1e-4), and whether to freeze embedding layers (i.e., training only
intermediary and classification layers). For both BERT and RoBERTa in the
frozen and unfrozen setting, we perform grid-search over the learning rate and
dropout probability. For these benchmarks, we use a predefined train-val-test
split (80%/10%/10%). The remaining setup is the same as in Section 5.

Table 4 shows the three best-performing configurations for BERT and RoBERTa in
the freeze and nofreeze settings, respectively, on the TransformerClassifier
pipeline. Table 5 shows the same results for the SiameseClassifier pipeline.

Model | DO | LR | #Epochs | EM
---|---|---|---|---
BERT-nofreeze | 0.25 | 1e-4 | 5 | 0.613
BERT-nofreeze | 0.10 | 1e-4 | 6 | 0.548
BERT-nofreeze | 0.50 | 1e-4 | 4 | 0.548
BERT | 0.25 | 1e-4 | 17 | 0.321
BERT | 0.10 | 1e-4 | 8 | 0.315
BERT | 0.10 | 1e-3 | 10 | 0.304
RoBERTa | 0.25 | 1e-3 | 14 | 0.262
RoBERTa | 0.10 | 1e-4 | 16 | 0.256
RoBERTa | 0.50 | 1e-3 | 15 | 0.238
RoBERTa-nofreeze | 0.25 | 1e-3 | 1 | 0.000
RoBERTa-nofreeze | 0.50 | 1e-3 | 1 | 0.000
RoBERTa-nofreeze | 0.10 | 1e-4 | 1 | 0.000
Table 4: Best three models for each of the proposed configurations in the TransformerClassifier pipeline Model | DO | LR | #Epoch | EM
---|---|---|---|---
BERT-nofreeze | 0.25 | 1e-4 | 7 | 0.589
BERT-nofreeze | 0.10 | 1e-4 | 4 | 0.577
BERT-nofreeze | 0.50 | 1e-4 | 2 | 0.565
RoBERTa | 0.10 | 1e-4 | 21 | 0.548
RoBERTa | 0.50 | 1e-4 | 13 | 0.518
RoBERTa | 0.25 | 1e-4 | 17 | 0.512
BERT | 0.50 | 1e-4 | 9 | 0.387
BERT | 0.25 | 1e-3 | 8 | 0.357
BERT | 0.25 | 1e-4 | 5 | 0.339
RoBERTa-nofreeze | 0.25 | 1e-3 | 1 | 0.000
RoBERTa-nofreeze | 0.50 | 1e-3 | 1 | 0.000
RoBERTa-nofreeze | 0.10 | 1e-4 | 1 | 0.000
Table 5: Best three models for each of the proposed configurations in the
SiameseClassifier pipeline

The most notable finding appears to be that RoBERTa gets stuck in a false
minimum of predicting a constant class when embedding layers are unfrozen,
leading to an accuracy of 0.33 and an EM of 0. Hence, we freeze embedding
layers for these model types in our main evaluation. As noted in Section 5, a
possible reason for this behaviour could be differences in the embeddings
contained within BERT’s [CLS] and RoBERTa’s <s> token.


# Stance Detection on Social Media with Fine-Tuned Large Language Models

Ilker Gül
EPFL
Lausanne, Switzerland
<EMAIL_ADDRESS>
&Rémi Lebret
EPFL
Lausanne, Switzerland
<EMAIL_ADDRESS>
&Karl Aberer
EPFL
Lausanne, Switzerland
<EMAIL_ADDRESS>

###### Abstract

Stance detection, a key task in natural language processing, determines an
author’s viewpoint based on textual analysis. This study evaluates the
evolution of stance detection methods, transitioning from early machine
learning approaches to the groundbreaking BERT model, and eventually to modern
Large Language Models (LLMs) such as ChatGPT, LLaMa-2, and Mistral-7B. While
ChatGPT’s closed-source nature and associated costs present challenges, the
open-source models like LLaMa-2 and Mistral-7B offers an encouraging
alternative. Initially, our research focused on fine-tuning ChatGPT, LLaMa-2,
and Mistral-7B using several publicly available datasets. Subsequently, to
provide a comprehensive comparison, we assess the performance of these models
in zero-shot and few-shot learning scenarios. The results underscore the
exceptional ability of LLMs in accurately detecting stance, with all tested
models surpassing existing benchmarks. Notably, LLaMa-2 and Mistral-7B
demonstrate remarkable efficiency and potential for stance detection, despite
their smaller sizes compared to ChatGPT. This study emphasizes the potential
of LLMs in stance detection and calls for more extensive research in this
field.

Stance Detection on Social Media with Fine-Tuned Large Language Models

Ilker Gül EPFL Lausanne, Switzerland<EMAIL_ADDRESS>Rémi Lebret EPFL
Lausanne, Switzerland<EMAIL_ADDRESS>Karl Aberer EPFL Lausanne,
Switzerland<EMAIL_ADDRESS>

##  1 Introduction

Stance detection plays a crucial role in analyzing social media content,
aiming to identify an author’s viewpoint—supportive, oppositional, or
neutral—towards various subjects, from political figures to environmental
policies. This process, relying on nuanced textual analysis across platforms
like X (formerly Twitter), presents both opportunities and challenges for
accurate interpretation hasan-ng-2013-stance; 10.1145/3369026;
DBLP:journals/corr/abs-2006-03644, key to gaining insights into public
sentiment and societal trends. These insights are invaluable for research in
data extraction, policy analysis, and beyond siddiqua-etal-2019-tweet;
darwish2017trump; glandt-etal-2021-stance. As natural language processing
(NLP) and social computing fields advance, they provide enhanced tools for
effectively extracting and analyzing stances from social media texts,
reflecting the evolving landscape of online discourse.

Initially, stance detection predominantly utilized rule-based and traditional
machine learning approaches, with support vector machines (SVM) marking a
significant early benchmark anand-etal-2011-cats; walker2012your; mohammad-
etal-2016-semeval. The introduction of deep learning models marked a pivotal
shift, improving over traditional methods due to their ability to process
large datasets and uncover complex patterns wei-etal-2016-pkudblab; zarrella-
marsh-2016-mitre; dey2018topical; huang2018aspect; zhang-etal-2019-aspect. The
significant advancements continued with the development of pre-trained
language models (PLMs) such as BERT, which facilitated a deeper understanding
of textual context devlin-etal-2019-bert; li-etal-2021-p; kawintiranon-
singh-2021-knowledge.

The evolution of Large Language Models (LLMs) like OpenAI’s ChatGPT and Meta
AI’s LLaMa-2 represents a major leap in NLP qin2023chatgpt; touvron2023llama.
These models, trained on vast datasets, are adept at mimicking the subtleties
of human language with remarkable accuracy zhao2023survey; yin-
etal-2023-large. Both open-source and adaptable, LLaMa-2 and Mistral-7B
enhance NLP technology accessibility and facilitate research without
proprietary constraints jiang2023mistral. Unlike their predecessors, LLMs use
flexible prompting strategies, allowing them to perform a wide array of NLP
tasks directly, including stance detection. This capability to navigate the
intricate relationship between textual context and target subjects, accurately
aligning with the author’s intended stance, underscores their advanced
understanding of language brown2020language; chowdhery2022palm; wei2023larger.

The implementation of prompting strategies represents a significant departure
from traditional NLP model training methods. By employing these strategies,
LLMs can generate predictions without the extensive fine-tuning typically
required, showcasing their versatility across various tasks. This
methodological shift has not only simplified the application of LLMs but also
expanded their utility, making them particularly effective for tasks that
require a deep understanding of language nuances, such as stance detection
bang-etal-2023-multitask; ouyang2022training.

Fine-tuning tailors LLMs like ChatGPT, LLaMa-2, and Mistral-7B to specific
tasks, significantly enhancing their precision and relevance for contextually
aware stance detection on social media platforms zhang2023llamaadapter. This
adaptation to the unique language and style of social media discourse enables
these models to outperform traditional methods, demonstrating superior
performance in identifying sentiments and viewpoints.

The flexibility of LLMs in social media analysis is highlighted by their use
of advanced techniques such as fine-tuning, chain of thought prompting
chen2023you, and both zero-shot and few-shot learning zhang2023stance;
cruickshank2023use; aiyappa-etal-2023-trust. These methods help navigate
complex online discourse, enhancing the models’ ability to interpret both
explicit and implicit content effectively gatto2023chainofthought;
lan2023stance.

Our study evaluates the enhanced performance of fine-tuned LLMs using Twitter
datasets, now known as X, to analyze a wide range of user opinions. We aim to
show that fine-tuning significantly improves model understanding of user
viewpoints, offering a deeper insight into online discourse. This research
underscores the advantages of fine-tuning in NLP, particularly in stance
detection, highlighting its superiority over traditional and less-tailored
methods.

##  2 Methods

###  2.1 Datasets and Evaluation Metrics

Datasets. To assess the performance of our fine-tuned LLMs, we employed three
publicly available datasets. The SemEval-2016 Dataset mohammad-
etal-2016-semeval includes diverse societal and political topics, with stances
categorized as Favor, Against, or None. This dataset examines a range of
targets including political figures such as Donald Trump and Hillary Clinton,
as well as broad issues like the Feminist Movement and Climate Change. It
offers a total of 2,914 training instances and 1,956 testing instances.
Detailed breakdowns of the training and testing data for each target are
provided in Table 1, underscoring the dataset’s utility for analyzing varied
viewpoints.

Target | # train | # test
---|---|---
Atheism | 513 | 220
Climate Change is Concern | 395 | 169
Feminist Movement | 664 | 285
Hillary Clinton | 689 | 295
Legalization of Abortion | 653 | 280
Donald Trump | 0 | 707
Total | 2914 | 1956
Table 1: Distribution of SemEval-2016 Dataset

The P-Stance Dataset li-etal-2021-p, on the other hand, narrows its focus to
the political domain, featuring 21,574 labeled tweets that express either
favor or opposition towards political figures Bernie Sanders, Donald Trump,
and Joe Biden. Detailed statistics of this structured dataset are presented in
Table 2, offering a structured dataset for nuanced model training and
analysis.

Target | # train | # validation | # test
---|---|---|---
Bernie Sanders | 5,056 | 634 | 635
Joe Biden | 5,806 | 745 | 745
Donald Trump | 6,362 | 795 | 796
Total | 17,224 | 2,174 | 2,176
Table 2: Distribution of the P-Stance Dataset

The Twitter Stance Election 2020 dataset, presented in Table 3 is derived from
a larger corpus of over 5 million tweets, collected between January and
September 2020 using election-related hashtags, to capture a comprehensive
view of public opinion kawintiranon-singh-2021-knowledge.

Candidate | # Train | # Test
---|---|---
Joe Biden | 875 | 375
Donald Trump | 875 | 375
Total | 1,750 | 750
Table 3: Distribution of the Twitter Stance 2020 Dataset

This dataset is labeled through Amazon Mechanical Turk crowston2012amazon to
ensure a balanced representation, it serves as a pivotal resource for
analyzing the public’s stance towards the candidates in the 2020 US
Presidential election.

Evaluation Metrics. In line with the standards set by previous studies
mohammad-etal-2016-semeval; mohammad2017stance, we adopt $F_{\mathrm{avg}}$ as
our primary evaluation metric, enabling us to assess their precision, recall,
and overall effectiveness in stance detection across varied contexts. This
metric, $F_{\mathrm{avg}}$, computes the average of the $F1$ scores for the
’favor’ and ’against’ classes.

$F_{\text{avg}}=\frac{F_{\text{favor}}+F_{\text{against}}}{2}$

For the analysis of the twitterstance2020, we adhere to the methodology
outlined by the dataset’s original authors kawintiranon-singh-2021-knowledge,
employing the F1-macro score across the three classes. This approach ensures
consistency with the established evaluation framework and facilitates direct
comparison with previous findings regarding this dataset.

###  2.2 Models

In our experiments, we utilized the ChatGPT-1106 (gpt-3.5-turbo-1106) for
fine-tuning, zero-shot, and few-shot tasks, as per OpenAI’s
guidelines111https://platform.openai.com/docs/guides/fine-tuning. We fine-
tuned the model over three epochs, creating the ChatGPT-ft variant. This
choice was governed by the closed-source nature and cost considerations of
ChatGPT’s API.

We also explored open-source alternatives, LLaMa-2 and Mistral-7B, developed
respectively by Meta AI and Mistral AI. For LLaMa-2 models (LLaMa-2-7b and
LLaMa-2-13b), and the Mistral-7b, fine-tuning was adapted to dataset
specifics: three epochs for SemEval-2016 and twitterstance2020, and one epoch
for the larger P-Stance dataset to mitigate overfitting.

Post fine-tuning, the models were designated as LLaMa-2-7b-ft, LLaMa-2-13b-ft,
and Mistral-7b-ft. We employed Low-Rank Adaptation (LoRA) hu2021lora to refine
the self-attention queries and values, through the Lit-GPT framework
litgpt-2023. This approach was aimed at enhancing the stance detection
capabilities of the models.

To detail the fine-tuning parameters, the selected hyperparameters for LoRA
included a rank of 8, an $\alpha$ of 16, and a dropout rate of 0.05. We
adopted a warmup strategy using 10% of the training data, spreading the
training over three epochs, with a learning rate of $3\times 10^{-4}$ and a
batch size of 128. The training was conducted on single NVIDIA A100 GPUs with
40GB of RAM, using bfloat16 precision.

The training sessions for LLaMa-2-7b-ft and Mistral-7b-ft on the SemEval-2016
dataset were completed in approximately 20 minutes, with LLaMa-2-13b-ft
requiring about 30 minutes. This detail ensures the reproducibility of our
results and the assessment of our method’s robustness.

We tested the impact of training data volumes (70%, 50%, and 30% of the total
dataset) on model performance, identifying the optimal training volume that
balances accuracy with resource efficiency, highlighting the effectiveness of
fine-tuning in stance detection.

Our study utilized both the base versions of each model for fine-tuning and
their instruction-tuned variants for the zero-shot and few-shot prompting
evaluations. The instruction-tuned versions, LLaMa-2-7b-chat,
LLaMa-2-13b-chat, and Mistral-7b-Instruct, were enhanced by pre-training on
instruction datasets with over 1 million human annotations, improving their
ability to follow complex instructions touvron2023llama.

For our analysis, we conducted zero-shot and few-shot experiments with ChatGPT
and the instruction-tuned models. In few-shot prompting tests, we utilized
three training samples from different stance categories, selected for their
relevance to our test cases using the “all-MiniLM-L6-v2” model from the
sentence-transformers library 222https://huggingface.co/sentence-transformers.
This method provided insights into each model’s ability to generalize from
minimal examples and revealed their performance across various learning
settings.

###  2.3 Prompting Details

For the ChatGPT model, we employed specific prompting methods for each dataset
following the guideline provided by
OpenAI333https://platform.openai.com/docs/guides/prompt-engineering. For the
LLaMa-2 model, our prompting strategy was inspired by the template samples
available in HuggingFace’s resources444https://huggingface.co/blog/llama2. For
the Mistral model, we devised our prompting approach based on best practices
and recommendations detailed in their
website555https://docs.mistral.ai/guides/prompting-capabilities/. Detailed
specifications of the prompts used for each dataset can be found in the
Appendix LABEL:sec:appendix2.

###  2.4 Baselines

We have selected various stance detection models as our baselines, with
different foundational architectures and approaches. We include BiLSTM and
BiCond models, which utilize bidirectional LSTM layers to capture language
context from both directions, crucial for nuanced stance detection augenstein-
etal-2016-stance. MemNet tang-etal-2016-aspect, from memory networks, retains
key sentiment aspects from text, enhancing context-aware analysis.

AoA huang2018aspect and TAN du2017stance incorporate attention mechanisms; AoA
uses hierarchical attention to prioritize different text parts, while TAN
applies target-specific attention to focus on the target-text relation. ASGCN
zhang2019aspect uses graph neural network techniques to understand textual
interdependencies effectively. AT-JSS-Lex li-caragea-2019-multi integrates
sentiment analysis with stance detection, utilizing a sentiment lexicon to
leverage sentiment signals for better stance prediction. TPDG liang2021target
prioritizes target-dependent representations in its detection approach.

StSQA zhang2023investigating introduces a unique method of stance detection by
teaching models like ChatGPT to perform stance detection with minimal
examples, exemplified through a 1-shot learning scenario. TR-Tweet+COT
gatto2023chainofthought introduces COT Embeddings to enhance Chain-of-Thought
(COT) prompting in LLMs for social media stance detection, integrating them
with RoBERTa to handle implicit stances.

Lastly, the COLA framework lan2023stance employs role-specific LLMs to dissect
and deduce stances from text, addressing the multifaceted nature of stance
detection and culminating in an agent that synthesizes these insights for
accurate stance detection.

##  3 Results

###  3.1 Controversial Topics on Twitter

####  3.1.1 Fine-Tuned Models vs Baselines

| FM | HC | LA | A | CC | DT
---|---|---|---|---|---|---
Baseline models |  |  |  |  |  |
BiCond | 61.4 | 59.8 | 54.5 | - | - | 59.0
MemNet | 57.8 | 60.3 | 61.0 | - | - | -
AoA | 60.0 | 58.2 | 62.4 | - | - | -
TAN | 55.8 | 65.4 | 63.7 | 59.3 | 53.5 | -
ASGCN | 58.7 | 64.3 | 62.9 | - | - | 58.7
AT-JSS-Lex | 61.5 | 68.3 | 68.4 | 69.2 | 59.2 | -
TPDG | 67.3 | 73.4 | 74.7 | - | - | 63.0
TR-Tweet+COT | 70.6 | 78.7 | 63.8 | 72.9 | 54.1 | -
COLA | 69.1 | 75.9 | 71.0 | 62.3 | 64.0 | 71.2
Fine-tuned LLMs |  |  |  |  |  |
ChatGPT-ft | 79.7 | 83.4 | 72.6 | 81.3 | 86.2 | 70.4
LLaMa-2-7b-ft | 73.3 | 84.2 | 71.2 | 78.9 | 69.8 | 72.0
LLaMa-2-13b-ft | 76.0 | 86.5 | 72.5 | 76.9 | 80.4 | 70.9
Mistral-7b-ft | 78.7 | 85.0 | 76.0 | 74.7 | 71.8 | 68.6
Table 4: SemEval-2016 Dataset performance with $F_{avg}$. Note: FM - Feminist
Movement, HC - Hillary Clinton, LA - Legalization of Abortion, Ath - Atheism,
CC - Climate Change is a Real Concern, DT - Donald Trump. Each acronym
represents a specific target within the dataset, assessed with $F_{avg}$
scores.

The analysis of the SemEval-2016 Dataset, presented in Table 4, reveals
significant findings regarding the performance enhancements achieved through
fine-tuning of language models on specific topics such as the Feminist
Movement (FM), Hillary Clinton (HC), and the Legalization of Abortion (LA).
These topics were selected for detailed analysis due to the completeness and
availability of comparative data.

Among the baseline models, TR-Tweet+COT and COLA demonstrated the power of
leveraging advanced language model architectures, setting new benchmarks in
stance detection with notable $F_{avg}$ scores. For instance, TR-Tweet+COT
achieved an $F_{avg}$ of 70.6 for FM and COLA scored 75.9 for HC, illustrating
significant improvements over traditional models. These advancements suggest
that the deeper contextual understanding provided by LLM-based models is
crucial for effectively navigating the complexities inherent in stance
detection tasks.

The superior performance of fine-tuned models such as ChatGPT-ft, which
achieved an $F_{avg}$ score of 79.7 for FM, reflects the effectiveness of
fine-tuning in enhancing model sensitivity to nuanced discussions. This model
outperformed the top baseline model by over 9 points, highlighting the
transformative impact of fine-tuning on model capabilities.

In the case of political figures like Hillary Clinton, LLaMa-2-13b-ft recorded
the highest $F_{avg}$ of 86.5, surpassing all baseline models. This
performance underscores the potential of fine-tuned models to accurately
identify and analyze sentiments, benefiting from the refined model parameters
tailored specifically to the task.

For sensitive topics such as the Legalization of Abortion, Mistral-7b-ft
demonstrated substantial performance improvement, achieving an $F_{avg}$ of
76.0, which indicates the model’s enhanced ability to handle complex discourse
following fine-tuning. This confirms the adaptability of LLMs through fine-
tuning, which enables them to better capture and interpret the subtleties
within the data crucial for accurate stance detection.

####  3.1.2 Fine-Tuning Impact: Comparing Training Scales

Models |  |  | FM | HC | LA
---|---|---|---|---|---
|  |  | Zero-shot Prompting
ChatGPT-1106 |  |  | 74.6 | 82.8 | 59.6
LLaMa-2-7b-chat |  |  | 51.6 | 63.9 | 49.2
LLaMa-2-13b-chat |  |  | 55.0 | 61.5 | 45.9
Mistral-7b-instruct |  |  | 50.7 | 66.3 | 50.1
|  |  | Few-shot Prompting
ChatGPT-1106 |  |  | 75.6 | 82.9 | 62.4
LLaMa-2-7b-chat |  |  | 62.1 | 72.7 | 49.1
LLaMa-2-13b-chat |  |  | 62.6 | 75.4 | 51.1
Mistral-7b-instruct |  |  | 59.7 | 76.6 | 51.2
| Train Size | Fine-Tuning
ChatGPT-ft | 100% |  | 79.7 | 83.4 | 72.6
1-6 LLaMa-2-7b-ft | 100% |  | 73.3 | 84.2 | 71.2
70% |  | 71.5 | 84.4 | 71.1
50% |  | 70.7 | 83.2 | 67.4
30% |  | 67.7 | 82.4 | 58.1
1-6 LLaMa-2-13b-ft | 100% |  | 76.0 | 86.5 | 72.5
70% |  | 76.8 | 86.3 | 71.2
50% |  | 68.3 | 86.1 | 68.3
30% |  | 63.7 | 84.8 | 64.1
1-6 Mistral-7b-ft | 100% |  | 78.7 | 85.0 | 76.0
70% |  | 76.3 | 80.8 | 69.9
50% |  | 71.7 | 84.8 | 72.8
30% |  | 73.8 | 83.1 | 73.7

Table 5:  $F_{\mathrm{avg}}$ scores among different configurations for
SemEval-2016 Dataset

The results presented in Table 3.1.2 demonstrate the impact of different
training strategies on the performance of state-of-the-art language models
across the SemEval-2016 dataset. This table compares zero-shot, few-shot, and
fine-tuned performance, emphasizing the substantial gains achieved through
fine-tuning.

ChatGPT, when fine-tuned, exhibited significant improvements in
$F_{\mathrm{avg}}$ scores, highlighting the effectiveness of adapting pre-
trained models to specific datasets. For instance, the fine-tuned ChatGPT
model achieved an $F_{\mathrm{avg}}$ score of 79.7 for the Feminist Movement
(FM), which was a 5.1 point increase over its zero-shot performance. This
substantial improvement underscores the importance of fine-tuning in
extracting more precise and contextually relevant information from complex
datasets.

LLaMa-2 models, particularly the 7b variant, also demonstrated notable
performance enhancements. The zero-shot configuration scored 51.6 for FM,
which jumped to 73.3 upon fine-tuning—an increase that highlights the model’s
capacity to significantly refine its stance detection capabilities with
comprehensive training.

Analysis of the effects of reduced training data volumes on model performance
revealed a nuanced pattern: while reductions typically led to lower
$F_{\mathrm{avg}}$ scores, the decrease was not always proportional,
indicating efficient learning even from reduced datasets. For example, the
LLaMa-2-7b model trained with 70% of the data managed nearly equivalent
performance to the fully trained model in the HC category (84.4 vs. 84.2),
suggesting that a substantial part of the model’s learning had been achieved
with the reduced data set.

The substantial performance gains in the fine-tuned models as compared to the
zero-shot and few-shot configurations illustrate the critical role of fine-
tuning in aligning the model’s parameters with the intricacies of specific
stance detection tasks. This is particularly evident in complex or
controversial topics such as Legalization of Abortion (LA), where contextual
understanding is crucial for accurate stance detection.

These results not only validate the effectiveness of fine-tuning in enhancing
the performance of language models in stance detection but also highlight the
potential efficiencies that can be realized even with reduced training
volumes. This suggests a promising direction for future research where less
data might be used more strategically to achieve high performance, especially
in resource-constrained settings.

###  3.2 Political Figures on Twitter

####  3.2.1 Fine-Tuned Models vs Baselines

Our study evaluates the efficacy of fine-tuned Large Language Models (LLMs)
against baseline models in capturing nuanced public stances towards political
figures, utilizing two distinct datasets: the P-Stance Dataset and the Twitter
Stance Election Dataset 2020. We begin with an in-depth exploration of the
P-Stance Dataset.

| Bernie | Biden | Trump
---|---|---|---
Baseline Models |  |  |
BiLSTM | 63.9 | 69.5 | 72.0
BiCond | 64.6 | 69.4 | 73.0
MemNet | 72.8 | 77.6 | 77.7
TAN | 72.0 | 77.9 | 77.5
AoA | 71.7 | 77.8 | 77.7
ASGCN | 70.8 | 78.4 | 77.0
StSQA | 80.8 | 82.6 | 85.7
Fine-tuned LLMs |  |  |
ChatGPT-ft | 81.8 | 89.7 | 91.9
LLaMa-2-7b-ft | 79.0 | 87.2 | 89.8
LLaMa-2-13b-ft | 81.0 | 89.0 | 88.9
Mistral-7b-ft | 81.0 | 88.2 | 90.3
Table 6:  P-Stance Dataset performance with $F_{avg}$ scores

The results in Table 6 underscore the substantial impact of fine-tuning on
stance detection accuracy. Notably, ChatGPT-ft demonstrates exceptional
performance, significantly surpassing both baseline models and other fine-
tuned counterparts. This indicates that the fine-tuning process, leveraging
ChatGPT’s extensive pre-trained knowledge, considerably enhances its ability
to discern complex stances across various political figures.

A critical observation from the P-Stance Dataset is the pronounced performance
differential between models, particularly in how fine-tuned LLMs like ChatGPT-
ft and Mistral-7b-ft navigate complex stance-related nuances compared to more
traditional models like StSQA. While StSQA performs robustly, the advanced
adaptation strategies employed in LLMs ensure a deeper alignment with the
nuanced demands of social discourse, enabling them to outperform traditional
models significantly.

Next, we extend our investigation to the Twitter Stance Election Dataset 2020,
focusing on the politically charged atmosphere of the 2020 U.S. presidential
election. This dataset provides a rich context for further evaluating the
refined capabilities of fine-tuned LLMs.

| Biden | Trump
---|---|---
Baseline Models |  |
a-BERT | 75.2 | 77.2
SKEP | 74.8 | 77.2
KE-MLM- | 74.3 | 76.3
KE-MLM | 75.7 | 78.7
Fine-tuned LLMs |  |
ChatGPT-ft | 85.1 | 85.6
LLaMa-2-7b-ft | 73.4 | 74.1
LLaMa-2-13b-ft | 74.1 | 74.8
Mistral-7b-ft | 74.1 | 74.3
Table 7: F1-macro scores for Twitter Stance Election 2020 Dataset

The results from Table 7 reveal significant performance differences between
the fine-tuned models and the baselines. Notably, ChatGPT-ft excels, achieving
F1-macro scores of 85.1 for Biden and 85.6 for Trump, outstripping both
baseline models like a-BERT and SKEP, and other fine-tuned LLMs such as
LLaMa-2 variants and Mistral-7b-ft. This superior performance may be
attributed to ChatGPT-ft having been extensively pretrained on a diverse
dataset that likely includes significant coverage of political figures. This
extensive pre-training enables the model to enhance its performance in tasks
involving political context.

The LLaMa-2 models and Mistral-7b-ft demonstrate notable improvements over
baseline models but still do not reach the impressive performance levels of
ChatGPT-ft. This discrepancy highlights the significant influence of model
architecture and the nuances of pre-training on the effectiveness of
subsequent fine-tuning efforts. Despite undergoing fine-tuning under
comparable conditions, the inherent architectural differences between models
significantly affect their ability to predict accurate stances.

Given the multifaceted nature of social media content, the variation in
performance among models like ChatGPT-ft, LLaMa-2, and Mistral-7b-ft in stance
detection tasks is particularly revealing. It accentuates the necessity for
model-specific fine-tuning strategies that cater to the unique strengths and
limitations of each architecture. This approach underscores the need for
continuous optimization to keep pace with the rapidly evolving landscape of
social media, particularly in the realm of political content, which is
consistently dynamic and continuously evolving.

####  3.2.2 Fine-Tuning Impact: Comparing Training Scales

Building on our initial findings in Section 3.2.1, we now examine the nuanced
impact of fine-tuning on model performance, particularly focusing on the
P-Stance Dataset due to its larger size compared to the Twitter Stance
Election Dataset 2020. This dataset allows for a detailed exploration of
various training strategies, including zero-shot, few-shot, and fine-tuning
across different scales of data.

You are a helpful, respectful, and honest assistant for stance detection for a
given target. Always answer from the possible options given below as helpfully
as possible. Stance detection is the process of determining whether the author
of a tweet is in support of or against a given target. The target may not
always be explicitly mentioned in the text, and the tweet’s stance can be
conveyed implicitly through subtext, regional and cultural references, or
other implicit meanings. The possible stances are:

  * •

support: The tweet has a positive or supportive attitude towards the target,
either explicitly or implicitly.

  * •

against: The tweet opposes or criticizes the target, either explicitly or
implicitly.

  * •

none: The tweet is neutral or doesn’t have a stance towards the target.

</SYS> Tweet: [tweet] Stance towards the target [target]:[/INST]

For this prompt structure, placeholders are utilized: [tweet] and [target].

  * •

[tweet]: Represents the actual tweet being analyzed.

  * •

[target]: Denotes what or whom the tweet’s stance is directed at.

####  A.2.2 P-Stance LlaMa 2 Template

This prompt template is specifically designed for analyzing tweets related to
the US presidential candidates:

> [INST] <<SYS>>

You are a helpful, respectful, and honest assistant for stance detection for
presidential candidates for the USA election. Always answer from the possible
options given below as helpfully as possible. Stance detection is the process
of determining whether the author of a tweet is in favor of or against a given
target. The target may not always be explicitly mentioned in the text, and the
tweet’s stance can be conveyed implicitly through subtext, regional and
cultural references, or other implicit meanings. The possible stances are:

  * •

support: The tweet has a positive or supportive attitude towards the target,
either explicitly or implicitly.

  * •

against: The tweet opposes or criticizes the target, either explicitly or
implicitly.

</SYS> Tweet: [tweet] Stance towards the target [target]:[/INST]

The placeholders [tweet] and [target] are used in a similar manner as
explained for the SemEval-2016 template above.

####  A.2.3 Twitter Stance 2020 LlaMa 2 Template

This prompt template is specifically designed for analyzing tweets related to
the US presidential candidates:

> [INST] <<SYS>>

You are a helpful, respectful, and honest assistant for stance detection for
presidential candidates for the USA election. Always answer from the possible
options given below as helpfully as possible. Stance detection is the process
of determining whether the author of a tweet is in favor of or against a given
target. The target may not always be explicitly mentioned in the text, and the
tweet’s stance can be conveyed implicitly through subtext, regional and
cultural references, or other implicit meanings. The possible stances are:

  * •

support: The tweet has a positive or supportive attitude towards the target,
either explicitly or implicitly.

  * •

against: The tweet opposes or criticizes the target, either explicitly or
implicitly.

  * •

none: The tweet is neutral or doesn’t have a stance towards the target.

</SYS> Tweet: [tweet] Stance towards the target [target]:[/INST]

The placeholders [tweet] and [target] are used in a similar manner as
explained for the SemEval-2016 template above.

##### Note on Terminology:

In the Llama 2 and Mistral templates, we decided to use the term "support"
instead of "favor". This decision was made based on token analysis for Llama 2
and Mistral, revealing that the model had a specific token for "support" but
not for "favor". As a result, for the sake of efficiency, "support" was used
in our prompt.

###  A.3 Mistral Fine-Tuning Prompts

For the Mistral model, we utilized a prompting approach analogous to that of
LLaMa-2, but with adjustments aligned to the guidelines specific for Mistral.
A key difference in our implementation for LLaMa-2 was the introduction of the
<<SYS>> marker across all prompts to signal the system’s response. For
Mistral, it was not required for initiating a response from the model. This
uniform adaptation was applied to each dataset we analyzed. An illustrative
example of a prompt used is as follows:

####  A.3.1 P-Stance Mistral Template

This prompt template is specifically designed for analyzing tweets related to
the US presidential candidates:

> [INST]

You are a helpful, respectful, and honest assistant for stance detection for a
given target. Always answer from the possible options given below as helpfully
as possible. Stance detection is the process of determining whether the author
of a tweet is in support of or against a given target. The target may not
always be explicitly mentioned in the text, and the tweet’s stance can be
conveyed implicitly through subtext, regional and cultural references, or
other implicit meanings. The possible stances are:

  * •

support: The tweet has a positive or supportive attitude towards the target,
either explicitly or implicitly.

  * •

against: The tweet opposes or criticizes the target, either explicitly or
implicitly.

Tweet: [tweet] Stance towards the target [target]:[/INST]