# Named Entity Recognition in Twitter:
A Dataset and Analysis on Short-Term Temporal Shifts

Asahi Ushio1, Leonardo Neves2, Vítor Silva2, Francesco Barbieri2, Jose
Camacho-Collados1
1Cardiff NLP, School of Computer Science and Informatics, Cardiff University,
United Kingdom
<EMAIL_ADDRESS>
2Snap Inc., Santa Monica, CA, United States
<EMAIL_ADDRESS>

###### Abstract

Recent progress in language model pre-training has led to important
improvements in Named Entity Recognition (NER). Nonetheless, this progress has
been mainly tested in well-formatted documents such as news, Wikipedia, or
scientific articles. In social media the landscape is different, in which it
adds another layer of complexity due to its noisy and dynamic nature. In this
paper, we focus on NER in Twitter, one of the largest social media platforms,
and construct a new NER dataset, _TweetNER7_ , which contains seven entity
types annotated over 11,382 tweets from September 2019 to August 2021. The
dataset was constructed by carefully distributing the tweets over time and
taking representative trends as a basis. Along with the dataset, we provide a
set of language model baselines and perform an analysis on the language model
performance on the task, especially analyzing the impact of different time
periods. In particular, we focus on three important temporal aspects in our
analysis: short-term degradation of NER models over time, strategies to fine-
tune a language model over different periods, and self-labeling as an
alternative to lack of recently-labeled data. TweetNER7 is released
publicly111https://huggingface.co/datasets/tner/tweetner7 along with the
models fine-tuned on it222NER models have been integrated into TweetNLP
Camacho-Collados et al. (2022) and can be found at
https://github.com/asahi417/tner/tree/master/examples/tweetner7_paper.

##  § 1 Introduction

Named Entity Recognition (NER) is a long-standing NLP task that consists of
identifying an entity in a sentence or document, and classifying it into an
entity-type from a fixed typeset. One of the most common and successful types
of NER system is achieved by fine-tuning pre-trained language models (LMs) on
a human-annotated NER dataset with token-wise classification Peters et al.
(2018); Howard and Ruder (2018); Radford et al. (2018, 2019); Devlin et al.
(2019). Remarkably, LM fine-tuning based NER models Yamada et al. (2020); Li
et al. (2020) already achieve over 90% F1 score in standard NER datasets such
as CoNLL2003 Tjong Kim Sang and De Meulder (2003) and OntoNotes5 Hovy et al.
(2006). However, NER is far from being solved, specialized domains such as
financial news Salinas Alvarado et al. (2015), biochemical Collier and Kim
(2004), or biomedical (Wei et al., 2015; Li et al., 2016) still pose
additional challenges Ushio and Camacho-Collados (2021). Lower performance in
these domains may be attributed to various factors such as the usage specific
terminologies within those domains, which LMs have not seen while pre-training
Lee et al. (2020).

Among recent studies, social media has been acknowledged as one of the most
challenging domains for NER Derczynski et al. (2016, 2017). Social media texts
are generally more noisy and less formal than conventional written languages
in addition to its vocabulary specificity. In social media, there is another
particular feature that needs to be addressed, which is the presence of
(quick) temporal shifts in the text semantics Rijhwani and Preotiuc-Pietro
(2020), where the meaning of words is constantly changing or evolving over
time. This is a general issue with language models Lazaridou et al. (2021),
but it is especially relevant given the dynamic landscape and immediacy
present in social media Del Tredici et al. (2019). There have been a few
specific approaches to deal with the temporal shifts in social media. For
instance, Loureiro et al. (2022) addressed this issue by pre-training language
models on a large tweet collection from different time period, highlighting
the importance of having an up-to-date language model. Agarwal and Nenkova
(2022) studied the temporal-shift in various NLP tasks including NER and
analyzed methods to overcome the temporal-shift with strategies such as self-
labeling.

In this paper, we propose a new NER dataset for Twitter (_TweetNER7_
henceforth). TweetNER7 contains tweets from diverse topics that are
distributed uniformly from September 2019 to August 2021. It contains 11,382
annotated tweets in total, spanning seven entity types (_person_ , _location_
, _corporation_ , _creative work_ , _group_ , _product_ , and _event_). To the
best of our knowledge, TweetNER7 is the largest Twitter NER datasets with a
high coverage of entity types TTC Rijhwani and Preotiuc-Pietro (2020) contains
about same amount of annotation yet with three entity types, while WNUT17
(Derczynski et al., 2017) has six entity types yet suffer from very small
annotations. The tweets for TweetNER7 were collected by querying tweets with
weekly trending keywords so that the tweet collection covers various topics
within the period, and we further removed near-duplicated tweets and
irrelevant tweets without any specific topics in order to improve the quality
of tweets. We provide baseline results with language model fine-tuning that
showcases the difficulty of TweetNER7, especially when dealing with time
shifts. Finally, we provide a temporal analysis with different strategies
including self-labeling, which does not prove highly beneficial in our
context, and provide insights in the model inner working and potential biases.

##  § 2 Related Work

There is a large variety of NER datasets in the literature. CoNLL2003 Tjong
Kim Sang and De Meulder (2003) and OntoNotes5 Hovy et al. (2006) are widely
used common NER datasets in the literature, where the texts are collected from
public news, blogs, and dialogues. WikiAnn (Pan et al., 2017) and MultiNERD
Tedeschi and Navigli (2022) are both multilingual NER datasets where the
training set is constructed by distant-supervision on Wikipedia and BabelNet.
As far as domain-specific NER datasets are concerned, FIN Salinas Alvarado et
al. (2015) is a NER dataset of financial news, while BioNLP2004 Collier and
Kim (2004) and BioCreative Wei et al. (2015); Li et al. (2016) are both
constructed from scientific documents of the biochemical and biomedical
domains. However, none of these datasets address the same challenges posed by
the social media domain.

In the social media domain, the pioneering Broad Twitter Corpus (BTC) NER
dataset Derczynski et al. (2016) included users with different demographics
with the aim to investigate spatial and temporal shift of semantics in NER.
More recently, the test set of WNUT2017 Derczynski et al. (2017) contained
unseen entities in the training set from broader social media including
Twitter, Reddit, YouTube, and StackExchange. The recent TweeBankNER dataset
Jiang et al. (2022) annotated TweeBank Liu et al. (2018) with entity labels to
investigate the interaction between syntax and NER.

The most similar dataset to ours is the Temporal Twitter Corpus (TTC) NER
dataset. Rijhwani and Preotiuc-Pietro (2020), which was also aimed at
analysing the temporal effects of NER in social media. For this dataset, 2,000
tweets every year from 2014 to 2019 were annotated. In general, however, these
social media datasets suffer from limited data, non-uniform distribution over
time, or limited entity types (see Subsection 3.3 for more details). In this
paper, we contribute with a new NER dataset (TweetNER7) based on recent data
until 2021, which is specifically designed to analyze temporal shifts in
social media.

##  § 3 TweetNER7: Dataset Construction, Statistics and Baselines

In this section, we present our time-aware NER dataset from publicly available
tweets with seven general entity types, which we refer as _TweetNER7_. In the
following subsections, we describe the data collection (Subsection 3.1) and
annotation (Subsection 3.2) processes. We also share relevant statistics
(Subsection 3.3) and baseline results (Subsection 3.4) of our dataset.

###  § 3.1 Data Collection

This NER dataset annotates a similar tweet collection used to construct
TweetTopic Antypas et al. (2022). The main data consists of tweets from
September 2019 to August 2021 with roughly same amount of tweets in each
month. This collection period makes it suitable for our purpose of evaluating
short-term temporal-shift of NER on Twitter. The original tweets were filtered
by leveraging weekly trending topics as well as by various other types of
filtering see Antypas et al. (2022) for more details on the collection and
filtering process). The collected tweets were then split into two periods:
September 2019 to August 2020 (2020-set) and September 2020 to August 2021
(2021-set).

###  § 3.2 Dataset Annotation

Figure 1:  The instructions shown to the annotators during the annotation
phase.

Annotation. To attain named-entity annotations over the tweets, we conducted a
manual annotation on Amazon Mechanical Turk with the interface shown in Figure
1. We split tweets into two periods: September 2019 to August 2020 (2020-set)
and September 2020 to August 2021 (2021-set), and randomly sampled 6,000
tweets from each period, which were annotated by three annotators, collecting
36,000 annotations in total. As the entity types, we employed seven labels:
_person_ , _location_ , _corporation_ , _creative work_ , _group_ , _product_
, and _event_. We followed Derczynski et al. (2017) for the selection of the
first six labels, and additionally included _event_ , as we found a large
amount of entities for events in our collected tweets.

Pre-processing. We pre-process tweets before the annotation to normalize some
artifacts, converting URLs into a special token $\\{\\{\texttt{URL}\\}\\}$ and
non-verified usernames into $\\{\\{\texttt{USERNAME}\\}\\}$. For verified
usernames, we replace its display name with symbols @. For example, a tweet

>
>     Get the all-analog Classic Vinyl Edition
>     of "Takin’ Off" Album from @herbiehancock
>     via @bluenoterecords link below:
>     http://bluenote.lnk.to/AlbumOfTheWeek
>

is transformed into the following text.

>
>     Get the all-analog Classic Vinyl Edition
>     of "Takin’ Off" Album from {@Herbie Hancock@}
>     via {{USERNAME}} link below: {{URL}}
>

We ask annotators to ignore those special tokens but label the verified users’
mentions.

Quality Control. Since we have three annotations per tweet, we control the
quality of the annotation by taking the agreement into account. We disregard
the annotation if the agreement is 1/3, and manually validate the annotation
if it is 2/3, which happens for roughly half of the instances.

###  § 3.3 Statistics

This subsection provides an statistical analysis of (i) our dataset, (ii) our
dataset in comparison with other Twitter NER datasets, and (iii) our dataset
distribution over time.

Statistics of TweetNER7. TweetNER7 contains 5,768 and 5,614 tweets annotated
in each period of 2020 and 2021, which are then split into training /
validation / test sets for each year. Since the 2020-set is for model
development, we consider 80% of the dataset as training set and 10% for
validation and test sets. Meanwhile, the 2021-set is mainly devised for model
evaluation to measure the temporal adaptability, so we take the majority of
the 2021-set (50%) as the test set and split the rest into training and
validation set with the same ratio of training and validation set of the
2020-set. Table 1 summarizes the number of the entities as well as the
instances in each subset of TweetNER7. We can observe a large gap between
frequent entity types such as _person_ and rare entity types as _location_ ,
while the distribution of the entities are roughly balanced across subsets. We
also report entity diversity, which we define as the percentage of unique
entities with respect to the total number of entities. Entity types such as
_product_ contain a relatively large number of duplicates (ranging between
56.2% and 76.4% entity diversity scores), while other types such as creative
work are more diverse (ranging between 80.1% and 93.2%).

Period | 2020-set | 2021-set
---|---|---
Split | Train | Valid | Test | Train | Valid | Test
Number of Entities
\- corporation | 1,700 | 203 | 191 | 902 | 102 | 900
\- creative work | 1,661 | 208 | 179 | 690 | 74 | 731
\- event | 2,242 | 256 | 265 | 968 | 131 | 1,097
\- group | 2,242 | 227 | 311 | 1,313 | 227 | 1,516
\- location | 1,259 | 181 | 165 | 697 | 72 | 716
\- person | 4,666 | 598 | 596 | 2,362 | 283 | 2,712
\- product | 1,850 | 241 | 220 | 926 | 111 | 972
All | 15,620 | 1,914 | 1,927 | 8,864 | 1,000 | 8,644
Entity Diversity
\- corporation | 69.9 | 92.6 | 90.1 | 72.1 | 85.3 | 74.3
\- creative work | 80.1 | 92.8 | 91.6 | 89.0 | 93.2 | 91.0
\- event | 71.1 | 90.6 | 84.2 | 75.9 | 89.3 | 70.9
\- group | 66.7 | 86.8 | 81.7 | 66.0 | 86.3 | 66.2
\- location | 66.4 | 80.7 | 81.2 | 67.9 | 88.9 | 64.9
\- person | 68.4 | 85.6 | 83.6 | 77.3 | 90.1 | 77.7
\- product | 56.2 | 71.4 | 76.4 | 60.3 | 79.3 | 56.6
Number of Tweets | 4,616 | 576 | 576 | 2,495 | 310 | 2,807

Table 1:  Number of entities, tweets, and entity diversity in each data split
and period, where the 2020-set is from September 2019 to August 2020, while
the 2021-set is from September 2020 to August 2021.

Comparison with other Twitter NER Datasets. In Table 2, we compare TweetNER7
against existing NER datasets for Twitter, which highlights the large number
of annotations of TweetNER7 for our covered period. TweetNER7 and TTC are the
overall largest datasets with more than 10k annotations, but TTC covers only
three entities, which may be insufficient for certain practical use cases
given the diversity of text in social media context Derczynski et al. (2017).
In contrast, TweetNER7 has the highest coverage of entity types among all NER
datasets in Twitter, including all the entity types from existing datasets. In
addition to the large amount of annotations and a high coverage of entity
types, TweetNER7 includes recent tweets from 2019 to 2021, from which most
corpus used in pre-training language models do not contain any text Devlin et
al. (2019); Liu et al. (2019); Nguyen et al. (2020). Assuming we tackle NER by
language model fine-tuning, this fact makes the task further challenging,
since language models have never seen the emerging entities from the period
during its pre-training phase.

Dataset | Annotations | Entities | Domain | Year
---|---|---|---|---
BTC | 9,339 | 3 | Twitter | 2009-2015
WNUT2017 | 5,690 | 6 | Twitter+ | 2010-2017
TTC | 11,969 | 3 | Twitter | 2014-2019
TweeBankNER | 3,547 | 4 | Twitter | 2016
TweetNER7 | 11,382 | 7 | Twitter | 2019-2021

Table 2: Number of annotated instances in TweetNER7 and comparison NER
datasets for Twitter.

| Jan | Feb | Mar | Apr | May | Jun
---|---|---|---|---|---|---
BTC | 2,308 | 68 | 502 | 862 | 1,074 | 1,056
TTC | 945 | 1,014 | 1,307 | 1,089 | 764 | 694
TweetNER7 | 957 | 943 | 939 | 937 | 951 | 931
| Jul | Aug | Sep | Oct | Nov | Dec
BTC | 1,321 | 850 | 342 | 419 | 23 | 21
TTC | 760 | 754 | 889 | 958 | 958 | 866
TweetNER7 | 924 | 928 | 956 | 968 | 975 | 973

Table 3:  The number of tweets in each month from BTC, TTC, and our TweetNER7
(the counts are cumulated across years). The normalized standard deviation
across month is 7.5% (BTC), 1.6% (TTC), and 0.2% (TweetNER7).

Distribution over Time. One of the TweetNER7’s focus is the temporal shift in
Twitter similar to BTC and TTC datasets. Retaining uniform distribution over
time is essential for temporal analysis, since the amount of training
instances should have an effect to the metric if it is not uniform. Table 3
shows the distribution of the instances across each month and we can confirm
that TweetNER7 has a very similar amount of tweets each month, while BTC and
TTC have higher variation than TweetNER7. Moreover, Table 4 compares the
number of instances per year for each dataset. TweetNER7 has a an uneven
distribution here due to the the selected range for each period (i.e.,
September 2019 to August 2021), which results in more tweets in 2020 than 2019
and 2021.

| 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015
---|---|---|---|---|---|---|---
BTC | 3 | 5 | 127 | 2,414 | 275 | 6,022 | 0
TTC | 0 | 0 | 0 | 0 | 0 | 2,000 | 2,000
TweetNER7 | 0 | 0 | 0 | 0 | 0 | 0 | 0
| 2016 | 2017 | 2018 | 2019 | 2020 | 2021 |
BTC | 0 | 0 | 0 | 0 | 0 | 0 |
TTC | 2,000 | 2,000 | 2,000 | 2,000 | 0 | 0 |
TweetNER7 | 0 | 0 | 0 | 1,936 | 5,768 | 3,678 |

Table 4: The number of tweets in each year from BTC, TTC, and our TweetNER7
dataset.

###  § 3.4 Baseline Results

Finally, we introduce a couple of baselines with language model fine-tuning on
the TweetNER7 in temporal-shift setup, where we develop models with the
training and the validation set from the 2020-set, and evaluate the models on
the test set of the 2021-set. In this setup, models are required to generalize
to the text from newer period, which the model has not seen in the fine-tuning
phase.

Experimental Setting. We consider masked language model fine-tuning with the
following LMs: BERT Devlin et al. (2019) and RoBERTa Liu et al. (2019) as
general-purpose LMs, and BERTweet Nguyen et al. (2020), and TimeLMs Loureiro
et al. (2022) as Twitter-specific LMs. TimeLMs are based on a RoBERTaBASE
architecture pre-trained on tweets collected continuously until different
years: 2019, 2020, and 2021. Model weights are taken from HuggingFace Wolf et
al. (2020).333 We use bert-base-cased and bert-large-cased for BERT, roberta-
base and roberta-large for RoBERTa, vinai/bertweet-base and vinai/bertweet-
large for BERTweet, and cardiffnlp/twitter-roberta-base-2019-90m,
cardiffnlp/twitter-roberta-base-dec2020, and cardiffnlp/twitter-roberta-base-
dec2021 for TimeLMs.  As evaluation metrics, we consider micro/macro F1 score
and type-ignored F1 score Ushio and Camacho-Collados (2021), in which the
entity type of the prediction is not considered in the evaluation (i.e., this
metric only assesses whether the predicted entity is an entity or not). The F1
scores measure the NER systems’ entire performance, while the type-ignored F1
score measures the ability of identifying whether a span of text is an entity
or not. LM fine-tuning on NER relies on the T-NER library Ushio and Camacho-
Collados (2021) and to find the best combination of hyper-parameters to fine-
tune LMs on NER, we run two-phase grid search. First, we fine-tune a model on
every possible configuration from the search space for 10 epochs. The top-5
models in terms of micro F1 score on the validation set are selected to
continue fine-tuning until their performance plateaus, and then the model that
achieves the highest micro F1 score on the validation set is employed as the
final model. The search space contains 24 configurations, which consist of the
following variations: learning rates from $[0.000001,0.00001,0.0001]$; ratio
of total training step for linear warm up of learning rate from $[0.15,0.3]$;
whether to normalize the gradient norm or not; and whether to add conditional
random field (CRF) on top of the output logit of LM.444Other parameters are
fixed: random seed is $0$ and batch size is $32$.

Model | Micro F1 | Macro F1 | Type-ig. F1
---|---|---|---
2021 / 2020 | 2021 / 2020 | 2021 / 2020
BERTBASE | 60.1 / 60.9 | 54.7 / 56.5 | 75.6 / 72.4
BERTLARGE | 61.4 / 62.2 | 56.1 / 58.1 | 75.9 / 73.8
BERTweetBASE | 64.1 / 66.4 | 59.4 / 62.4 | 77.9 / 77.7
BERTweetLARGE | 64.0 / 65.9 | 59.5 / 62.6 | 78.3 / 77.4
RoBERTaBASE | 64.2 / 64.2 | 59.1 / 60.2 | 77.9 / 74.8
RoBERTaLARGE |  64.8 / 65.7 |  60.0 / 61.9 |  78.4 / 76.1
TimeLM2019 | 64.3 / 65.4 | 59.3 / 61.1 | 77.9 / 76.6
TimeLM2020 | 62.9 / 64.4 | 58.3 / 60.3 | 76.5 / 75.7
TimeLM2021 | 64.2 / 65.4 | 59.5 / 61.1 | 77.4 / 76.4

Table 5: Result of temporal-shift NER on TweetNER7 where micro and macro F1
score as well as type-ignored F1 score on the test set of the 2021-set /
2020-set are reported. The best results in each of the 2021-set / 2020-set are
highlighted in bold character / underline in each metric.

Results. We report the NER results on TweetNER7 in Table 5, where RoBERTaLARGE
is the best across metrics. We should note, however, that the overall metrics
(micro F1 lower than 65% in all cases on the 2021 test set) are lower than
those in standard NER datasets Ushio and Camacho-Collados (2021), which
highlights the difficulty of the social media and temporal-shift components in
TweetNER7. RoBERTa is also the best model among the BASE models but
interestingly the TimeLM2020 performs worse than other RoBERTa models. This
can be explained by the fact that TimeLM2020 was pre-trained over tweets until
the end of 2020. This may have let the model to over-fit to the training
corpus and makes it hard to generalize on the newer test set. Instead,
TimeLM2021 shows a better performance. Table 7 also reports the metrics on the
2020 test set for completeness. While that is not our primary aim, we can find
an interesting result which is the superior performance of BERTweet in this
case. This implies that a model that performs well in the same period of the
training set does not guarantee an equally strong performance on an unseen
period.

Breakdown by entity type. Figure 2 shows a comparison of entity-wise F1 scores
over the language models, and we can see an important gap across entity types.
According to Table 1, _person_ is the most frequent entity type and its F1
score is equally high (around 80%), while _creative work_ and _location_ are
the rarest entity types and hence their F1 scores are relatively low (around
40% for _creative work_ and 60% for _location_). The reason why the
performance for _location_ is better than for _creative work_ may be
attributable to their differences in entity diversity. As we could see from
Table 1, _creative work_ ’s diversity is higher than _location_ , which means
_creative work_ contains more variation of entities than _location_ while
having the same amount of entities in both types, which entails a higher
degree of difficulty. This seems a consistent trend that lower entity
diversity results in lower F1 score as can be seen for _event_ and
_corporation_ as well, which also have a low entity diversity score. To
overcome such entity imbalance, strategies such as balancing the instances of
each class could be explored Li et al. (2020).

Figure 2: Entity-wise F1 score breakdown from the baseline results in the 2021
test set (Table 5).

##  § 4 Temporal Analysis

To better understand the effect of the temporal-shift, we conduct three
additional comparative experiments: (i) temporal vs. random splits, (ii) joint
vs. continuous fine-tuning, and (iii) self-labeling as a solution to deal with
temporal shifts.

###  § 4.1 Short-Term Temporal Effect

If TweetNER7 does not suffer temporal-shift, how is the model performance
changed? This is a question we aim to answer in this analysis, and we create
new training and validation split without temporal-shift for this purpose.
Concretely, temporal-shift usually occurs in a situation where the training
and the validation sets do not contain any texts from the test period, so we
keep the amount of the training/validation split as the same in Subsection
3.4, but randomly sample from the full period of September 2019 to August 2021
instead of the first half period instead. Note that we do not change the test
set and make sure that each month has roughly the same amount of instances at
the sampling of the new training/validation sets, to make it fair comparison
with the temporal-shift result in Subsection 3.4.

Table 6 shows the variations of results between the random and temporal
splits. As expected, the F1 scores on the 2021 test set are generally improved
across all LMs, while the F1 scores on the 2020 test set are decreased. The
increase of accuracy in 2021 is achieved with the inclusion of
training/validation set from 2021, and the decrease of accuracy in 2020 is
caused by the reduced number of the training/validation set from the same 2020
period. This result further highlights the benefit of having a human annotated
training set from the test period, even if the time period differs in a year
only. Interestingly, the results for the time-specific pre-trained TimeLMs
models differ across years. Since in this paper we did not focus on the
analysis of the pre-training corpora, we leave further analysis about this
result for future exploration.

Model | 2021-set | 2020-set
---|---|---
Mi. F1 | Ma. F1 | T-i. F1 | Mi. F1 | Ma. F1 | T-i. F1
BERTBASE | +0.8 | +1.2 | +0.1 | +0.1 | +0.3 | +0.3
BERTLARGE | +1.0 | +1.4 | +0.6 | -0.7 | -1.0 | -0.5
BERTweetBASE | +1.5 | +0.2 | -0.1 | -2.5 | -3.8 | -3.3
BERTweetLARGE | +0.9 | +1.0 | +0.1 | +0.1 | +0.1 | -0.2
RoBERTaBASE | -0.2 | +0.1 | +0.1 | -0.1 | -0.4 | -0.5
RoBERTaLARGE | +1.5 | +1.0 | +0.6 | -1.3 | -1.8 | -0.6
TimeLM2019 | -1.0 | -0.8 | -0.5 | -1.1 | -0.4 | -0.4
TimeLM2020 | +1.8 | +1.7 | +1.8 | +0.3 | +0.2 | +0.2
TimeLM2021 | -1.0 | -1.1 | -0.4 | -1.7 | -1.3 | -1.0

Table 6:  Absolute performance improvement when evaluating on the random split
result over the original temporal split reported in Table 5. Positive
improvements are in blue and negative drops are in red.

|  | Dataset | Micro F1 | Macro F1 | Type-ig. F1
---|---|---|---|---|---
BERT |  BASE | 2020 | 60.1 / 60.9 | 54.7 / 56.5 | 75.6 / 72.4
2021 | 60.7 / 58.4 | 55.5 / 54.2 | 75.7 / 70.9
2020 + 2021 |  62.3 / 62.1 |  57.6 / 57.7 |  76.6 / 73.0
2020 $\rightarrow$ 2021 | 61.8 / 61.4 | 56.8 / 57.1 | 76.5 / 72.5
LARGE | 2020 | 61.4 / 62.2 | 56.1 / 58.1 | 75.9 / 73.8
2021 | 59.7 / 56.6 | 53.9 / 51.0 | 75.0 / 70.7
2020 + 2021 |  63.6 / 62.5 |  59.0 / 58.6 |  77.2 / 73.6
2020 $\rightarrow$ 2021 | 63.2 / 62.5 | 57.7 / 57.9 | 76.0 / 72.5
BERTweet |  BASE | 2020 | 64.1 / 66.4 | 59.4 / 62.4 | 77.9 / 77.7
2021 | 63.1 / 62.1 | 57.4 / 57.2 | 77.9 / 76.0
2020 + 2021 | 65.4 / 65.7 | 60.5 / 61.6 | 79.0 / 76.9
2020 $\rightarrow$ 2021 |  65.8 / 65.2 |  61.0 / 61.4 |  79.1 / 76.8
LARGE | 2020 | 64.0 / 65.9 | 59.5 / 62.6 | 78.3 / 77.4
2021 | 62.9 / 61.6 | 58.1 / 56.8 | 76.5 / 74.5
2020 + 2021 |  66.5 / 66.8 |  61.9 / 63.1 |  79.5 / 77.6
2020 $\rightarrow$ 2021 | 66.4 / 65.9 | 61.7 / 61.8 | 79.0 / 76.4
RoBERTa |  BASE | 2020 | 64.2 / 64.2 | 59.1 / 60.2 | 77.9 / 74.8
2021 | 61.8 / 60.5 | 57.0 / 56.1 | 76.9 / 73.8
2020 + 2021 | 65.2 / 65.3 |  60.8 / 61.7 |  78.9 / 75.2
2020 $\rightarrow$ 2021 |  65.5 / 65.1 | 60.0 / 60.8 | 78.1 / 75.0
LARGE | 2020 | 64.8 / 65.7 | 60.0 / 61.9 | 78.4 / 76.1
2021 | 64.0 / 63.4 | 59.1 / 59.1 | 77.7 / 74.4
2020 + 2021 | 65.7 / 66.3 |  61.2 / 63.0 | 78.8 / 76.4
2020 $\rightarrow$ 2021 |  66.0 / 66.3 | 60.9 / 62.4 |  79.1 / 76.4
TimeLM |  2019 | 2020 | 64.3 / 65.4 | 59.3 / 61.1 | 77.9 / 76.6
2021 | 63.2 / 61.9 | 56.7 / 56.1 | 75.7 / 73.0
2020 + 2021 | 65.7 / 65.5 | 61.0 / 61.2 |  78.9 / 76.4
2020 $\rightarrow$ 2021 |  65.9 / 64.8 |  61.1 / 60.6 | 78.4 / 75.5
2020 | 2020 | 62.9 / 64.4 | 58.3 / 60.3 | 76.5 / 75.7
2021 | 64.0 / 63.1 | 58.9 / 58.5 | 77.9 / 75.3
2020 + 2021 | 65.3 / 65.4 |  60.7 / 61.4 |  78.7 / 75.9
2020 $\rightarrow$ 2021 |  65.5 / 65.3 | 60.6 / 61.3 | 78.0 / 75.9
2021 | 2020 | 64.2 / 65.4 | 59.5 / 61.1 | 77.4 / 76.4
2021 | 63.5 / 62.3 | 58.7 / 57.9 | 77.5 / 74.1
2020 + 2021 | 64.5 / 65.8 | 59.8 / 61.9 | 77.9 / 76.5
2020 $\rightarrow$ 2021 |  65.1 / 64.9 |  60.0 / 60.7 |  78.1 / 75.8

Table 7:  Results of different strategies to ingest the training set of the
2021-set in TweetNER7 for different language models ($\rightarrow$: continuous
fine-tuning; +: concatenation of datasets). The best results in each model of
the 2021-set / 2020-set are highlighted in bold character / underline in each
metric.

###  § 4.2 Continuous vs. Joint Fine-Tuning

In the previous experiments we have shown the differences between training and
testing on the time period or not. Instead, this analysis comes under the
assumption that a labeled 2021 training set is available. Thus, the main aim
of this analysis is to explore different strategies to improve the original
model. In addition to fine-tuning LMs on the combined set of the 2020-set and
2021-set as in Subsection 4.1, we employed a continuous fine-tuning scheme,
where we first fine-tune LMs on the 2020-set and then continue fine-tuning on
the 2021-set. Table 7 shows the results of all strategies for different
language models. As can be observed, continuous fine-tuning provides the best
results in terms of micro F1 and type-ignored F1 in the 2021 test sets in most
cases, although the differences with respect to the concatenation of sets are
not substantial.

###  § 4.3 Self-Labeling

In both Subsections 4.1 and 4.2, we compared different strategies when a
human-annotated training dataset from the test period was considered, namely
the training and the validation sets from the 2021-set. This shows that
improvements can be obtained when the time between training and test data is
reduced. However, in many cases and real-world applications this is not
practical as it requires a large amount of human resources to annotate newer
tweets whenever. Thus, we consider an alternative approach to rely on
distantly annotated tweets by the already fine-tuned model. This solution was
explored by Agarwal and Nenkova (2022) in a similar setting, with promising
results. In this paper, we reproduced their experiments in our TweetNER7
dataset focusing on short-term temporal shift.

Training Set | Micro F1 | Macro F1 | Type-ig. F1
---|---|---|---
2021 / 2020 | 2021 / 2020 | 2021 / 2020
2020 |  64.8 / 65.7 |  60.0 / 61.9 | 78.4 / 76.1
2020-extra | 64.6 / 65.5 | 59.3 / 61.4 | 78.6 / 76.2
2020 + 2020-extra | 64.7 / 65.2 | 59.6 / 61.0 |  78.7 / 76.8
2020 $\rightarrow$ 2020-extra | 64.6 / 65.5 | 59.5 / 61.5 | 78.6 / 76.4
2021-extra | 64.2 / 65.7 | 59.3 / 61.8 | 78.2 / 76.9
2020 + 2021-extra | 64.3 / 65.6 | 59.3 / 61.7 | 78.4 / 76.9
2020 $\rightarrow$ 2021-extra | 64.5 / 65.5 | 59.5 / 61.4 | 78.6 / 76.3

Table 8:  Results of the self-labeling experiment with different strategies
for RoBERTaLARGE model ($\rightarrow$: continuous fine-tuning; +:
concatenation of datasets) where micro and macro F1 score as well as type-
ignored F1 score on the test set of 2021-set / 2020-set are reported. The best
results in each of the 2021-set / 2020-set are highlighted in bold character /
underline in each metric.

####  4.3.1 Evaluation

Experimental Setting. For our experiments we focused on the best model in our
previous experiments, which is RoBERTaLARGE. We collected extra (unlabeled)
tweets following the same procedure described in Antypas et al. (2022), that
results in 93,594 and 878,80 tweets from the period of 2020-set and 2021-set,
respectively. Over those extra tweets, we use the RoBERTaLARGE NER model fine-
tuned on the 2020-set to predict labels.

Results. Table 8 shows the result of self-labeling, where we report three
patterns of model fine-tuning: (i) fine-tuning only on the pseudo dataset
(e.g., 2020-extra); (ii) fine-tuning on the joint dataset of the training set
of the 2020-set and the pseudo dataset (e.g., 2020 + 2020-extra); and (iii)
continuous fine-tuning of the 2020-set fine-tuned model on the pseudo dataset
(e.g., 2020 $\rightarrow$ 2020-extra). In general, we can not find any major
improvement by self-labeling, regardless of the strategy. In a way, this
contradicts the self-labeling experiment on the TTC dataset performed by
Agarwal and Nenkova (2022).555While in our setting we extract a larger number
of tweets, this trend does not change with less self-labeled training data.
This may suggest that the temporal-shift of TweetNER7 is more challenging to
mitigate than TTC, and self-labeling is not enough in itself to overcome the
temporal shift.

####  4.3.2 Contextual Prediction Analysis

To explore the reason why self-labeling does not help to mitigate temporal-
shift in TweetNER7, we conducted an analysis over the self-labeled tweets.
Inspired by recent semi-parametric approach in information retrieval Lewis et
al. (2021), we considered a retrieval module that fetches relevant tweets
given a target entity from the self-labeled corpus and see the portion of
retrieved tweets containing the true prediction. To be precise, we first ran
the NER model prediction on target tweets, and for each of the predicted
entities. Then, we queried tweets from the extra tweet corpus used in
Subsection 4.3 to compute the ratio of correct predictions within the
retrieved predictions, which we call contextualized predictions. Since we are
interested in the error of the original prediction, we focus only on the
entities where the original prediction is incorrect.

Figure 3 describes the whole pipeline and we use Whoosh
library666https://pypi.org/project/Whoosh/ for search engine where the query
is always the entity name, constraining the search result by the number of
days from the query tweet.777Setting days as 7 means the search results should
be in the range of 7 days before/after was made. Similarly to the analysis in
§ 4.3, we used the RoBERTaLARGE fine-tuned on the 2020-set of TweetNER7 and
evaluated the contextualized predictions on the 2021 test set.

Figure 3: Overview of the pipeline to retrieve contextualized prediction.

Figure 4 shows the ratio of positive and negative predictions in the
contextualized tweets. These are further broken into two error types whether
it is the same prediction as the original prediction or not, along with the
days we set as a search constraint. Most frequent predictions are usually the
same as the original predictions, which means that the original language model
tends to output similar predictions for the same entities, irrespective of the
context. As far as the time variable is concerned, the ratio is almost
consistent over time, which suggests that the possible original bias of the
model does not change over time. Nonetheless, the second most frequent
predictions are on average the correct ones, with a large gap with respect to
other types of error. This implies there may still be a useful signal to
improve the original prediction in the self-labeled corpus.

Figure 4:  Ratio of positive and negative predictions in the contextualized
tweets, split into two error types: same prediction as the original prediction
or not. The X-axis represents the days from the original tweet (0=same date as
the original tweet) and results are broken on 20-day chunks..

##  § 5 Conclusion

In this paper, we have constructed TweetNER7, a new NER dataset for Twitter,
in which we annotated 11,382 tweets with seven entity types. The collected
tweets are distributed uniformly over time from September 2019 to August 2021,
which facilitates temporal analysis in NER for social media. The dataset is
diverse topic-wise, as we leveraged weekly trending topics to query tweets and
near-duplicated and irrelevant tweets were dropped. To establish baselines on
TweetNER7, we fine-tuned standard LMs including a few Twitter-specific LMs.
Moreover, we performed a few targeted temporal-related analyses in order to
better understand the short-term temporal effect. Finally, we show that self-
labeling is not enough to mitigate the temporal-shift and had no noticeable
improvement over the baseline vanilla fine-tuning, which further highlights
the challenging nature of the dataset.

##  § 6 Limitations and Future Work

The TweetNER7 dataset was constructed on English tweets so it is limited to
English, as most of the existing NER datasets for social media Derczynski et
al. (2016). In the future we are planning to apply a similar methodology to
extend it to languages other than English. Given the dynamic nature of social
media, TweetNER7 is designed to study short-term temporal-shift (e.g.,
monthly) but would not be suitable for analysing longer temporal shifts (e.g.,
yearly) Rijhwani and Preotiuc-Pietro (2020). We selected Twitter as the data
source but temporal-shift is a common problem in social media generally. As a
future work, we are planning to add more data from other social media
platforms as in WNUT17 (Derczynski et al., 2017) to give us more general
insights to understand temporal shift phenomena in social media more
generally.

## Acknowledgements

Jose Camacho-Collados is supported by a UKRI Future Leaders Fellowship.

## References

  * Agarwal and Nenkova (2022) Oshin Agarwal and Ani Nenkova. 2022.  Temporal effects on pre-trained models for language processing tasks.  _Transactions of the Association for Computational Linguistics_ , 10:904–921.
  * Antypas et al. (2022) Dimosthenis Antypas, Asahi Ushio, Jose Camacho-Collados, Leonardo Neves, Vitor Silva, and Francesco Barbieri. 2022.  Twitter Topic Classification.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Camacho-Collados et al. (2022) Jose Camacho-Collados, Kiamehr Rezaee, Talayeh Riahi, Asahi Ushio, Daniel Loureiro, Dimosthenis Antypas, Joanne Boisson, Luis Espinosa-Anke, Fangyu Liu, Eugenio Martínez-Cámara, et al. 2022.  Tweetnlp: Cutting-edge natural language processing for social media.  _arXiv preprint arXiv:2206.14774_.
  * Collier and Kim (2004) Nigel Collier and Jin-Dong Kim. 2004.  Introduction to the bio-entity recognition task at JNLPBA.  In _Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP)_ , pages 73–78, Geneva, Switzerland. COLING.
  * Del Tredici et al. (2019) Marco Del Tredici, Raquel Fernández, and Gemma Boleda. 2019.  Short-term meaning shift: A distributional exploration.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2069–2075, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Derczynski et al. (2016) Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016.  Broad Twitter corpus: A diverse named entity recognition resource.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 1169–1179, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Derczynski et al. (2017) Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017.  Results of the WNUT2017 shared task on novel and emerging entity recognition.  In _Proceedings of the 3rd Workshop on Noisy User-generated Text_ , pages 140–147, Copenhagen, Denmark. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hovy et al. (2006) Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006.  OntoNotes: The 90% solution.  In _Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers_ , pages 57–60, New York City, USA. Association for Computational Linguistics.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  * Jiang et al. (2022) Hang Jiang, Yining Hua, Doug Beeferman, and Deb Roy. 2022.  Annotating the tweebank corpus on named entity recognition and building nlp models for social media analysis.  In _Proceedings of the Language Resources and Evaluation Conference_ , pages 7199–7208, Marseille, France. European Language Resources Association.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, et al. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  _Advances in Neural Information Processing Systems_ , 34:29348–29363.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Lewis et al. (2021) Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021.  PAQ: 65 million probably-asked questions and what you can do with them.  _Transactions of the Association for Computational Linguistics_ , 9:1098–1115.
  * Li et al. (2016) Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016.  Biocreative v cdr task corpus: a resource for chemical disease relation extraction.  _Database_ , 2016.
  * Li et al. (2020) Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020.  Dice loss for data-imbalanced NLP tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 465–476, Online. Association for Computational Linguistics.
  * Liu et al. (2018) Yijia Liu, Yi Zhu, Wanxiang Che, Bing Qin, Nathan Schneider, and Noah A. Smith. 2018\.  Parsing tweets into Universal Dependencies.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 965–975, New Orleans, Louisiana. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Loureiro et al. (2022) Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-collados. 2022.  TimeLMs: Diachronic language models from Twitter.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 251–260, Dublin, Ireland. Association for Computational Linguistics.
  * Nguyen et al. (2020) Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020.  BERTweet: A pre-trained language model for English tweets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 9–14, Online. Association for Computational Linguistics.
  * Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.  Cross-lingual name tagging and linking for 282 languages.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  _Technical report, OpenAI_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rijhwani and Preotiuc-Pietro (2020) Shruti Rijhwani and Daniel Preotiuc-Pietro. 2020.  Temporally-informed analysis of named entity recognition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7605–7617, Online. Association for Computational Linguistics.
  * Salinas Alvarado et al. (2015) Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015.  Domain adaption of named entity recognition to support credit risk assessment.  In _Proceedings of the Australasian Language Technology Association Workshop 2015_ , pages 84–90, Parramatta, Australia.
  * Tedeschi and Navigli (2022) Simone Tedeschi and Roberto Navigli. 2022.  MultiNERD: A multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation).  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 801–812, Seattle, United States. Association for Computational Linguistics.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Ushio and Camacho-Collados (2021) Asahi Ushio and Jose Camacho-Collados. 2021.  T-NER: An all-round python library for transformer-based named entity recognition.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , pages 53–62, Online. Association for Computational Linguistics.
  * Wei et al. (2015) Chih-Hsuan Wei, Yifan Peng, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Jiao Li, Thomas C Wiegers, and Zhiyong Lu. 2015.  Overview of the biocreative v chemical disease relation (cdr) task.  In _Proceedings of the fifth BioCreative challenge evaluation workshop_ , volume 14.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020\.  LUKE: Deep contextualized entity representations with entity-aware self-attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6442–6454, Online. Association for Computational Linguistics.


# Federated Learning for Zero-Day Attack Detection in 5G and Beyond V2X
Networks

1st Abdelaziz Amara korba  L3I, University of La Rochelle, France
LRS, Badji Mokhtar Annaba University, Algeria
2nd Abdelwahab Boualouache  FSTM, University of Luxembourg
Luxembourg
3rd Bouziane Brik  DRIVE, University of Bourgogne
Franche Comte, France
4rd Rabah Rahal  LRS, Badji Mokhtar Annaba University, Algeria
5rd Yacine Ghamri-Doudane  L3I, University of La Rochelle,France
6rd Sidi Mohammed Senouci  DRIVE, University of Bourgogne
Franche Comte, France

###### Abstract

Deploying Connected and Automated Vehicles (CAVs) on top of 5G and Beyond
networks (5GB) makes them vulnerable to increasing vectors of security and
privacy attacks. In this context, a wide range of advanced machine/deep
learning-based solutions have been designed to accurately detect security
attacks. Specifically, supervised learning techniques have been widely applied
to train attack detection models. However, the main limitation of such
solutions is their inability to detect attacks different from those seen
during the training phase, or new attacks, also called zero-day attacks.
Moreover, training the detection model requires significant data collection
and labeling, which increases the communication overhead, and raises privacy
concerns. To address the aforementioned limits, we propose in this paper a
novel detection mechanism that leverages the ability of the deep auto-encoder
method to detect attacks relying only on the benign network traffic pattern.
Using federated learning, the proposed intrusion detection system can be
trained with large and diverse benign network traffic, while preserving the
CAVs’ privacy, and minimizing the communication overhead. The in-depth
experiment on a recent network traffic dataset shows that the proposed system
achieved a high detection rate while minimizing the false positive rate, and
the detection delay.

###### Index Terms:

5GB, Connected and Automated Vehicles, Security, Zero-day attacks, Federated
Learning

##  I Introduction

Fifth generation (5G) and beyond (5GB) networks promise to revolutionize the
transportation industry by enabling ultra-reliability with ultra-low latency
and high bandwidth communications [1]. These advances will significantly
empower many verticals, such as smart agriculture, health, and Intelligent
Transportation Systems (ITS). As part of ITS, Connected and Automated Vehicles
(CAVs) have been taken significant and careful considerations in 3GPP 5G
standards [2]. Specifically, integrating V2X communications into the 5G
ecosystem has enabled innovative use cases and applications, such as advanced
driving, vulnerable road user protection, and vehicle platooning [3]. Yet this
progress is expected to be extended with 5GB, contributing thus to reducing
traffic accidents and dramatically saving road users’ lives. However, CAVs at
all automation levels will face a massive vector of cyberattacks coming from
5GB technologies and leading to hazardous situations for road users. For
example, Distributed and Denial of Service (DDoS) attacks have already been
demonstrated to break 5G services [4]. But the impact of these attacks are
likely to be more expansive with the integration of CAVs. More than this,
cyberattacks are working continuously to develop novel tactics for breaching
and breaking such systems. Facing all these challenges, Machine Learning (ML)
appears as a key cybersecurity enabler to protect 5GB-enabled CAVs [5].
Various Machine Learning (ML)/ Deep Learning (DL) based Intrusion Detection
Systems (ML/DL-based IDSs) have been proposed to protect vehicular networks
against attacks. Most of them rely on supervised and centralized learning [6].
Centrally training the detection model requires significant data collection
and labeling, which increases the communication overhead, and may raise
privacy concerns. To mitigate centralized learning limitations, collaborative
ML [7] has been used, enabling thus continuous accuracy evolution and
flexibility. Nevertheless, several limitations exist in early collaborative
ML-based IDSs [8, 9, 10]. Specifically, they generate a significant
communication overhead during ML model updates and may violate data privacy,
since learning nodes might share private information. To cope with the
aforementioned issues, recent research [11, 12, 13, 14] leveraged the
potential of federated leaning (FL) paradigm, which has shown promising
results in many applications. FL is a distributed ML paradigm allowing several
nodes to train a global model cooperatively without sharing their datasets,
avoiding thus overhead and mitigating privacy risks [7]. Interestingly, all
existing FL-based IDSs for 5GB-enabled CAVs rely on supervised learning
techniques. One important limitation of using such techniques is their
inability to detect attacks different from those seen during the training
phase (unseen attacks), and zero-day attacks. Another challenging issue is
data imbalance, i.e., the numbers of benign and malicious traffic samples are
not in the same range. Benign network traffic samples are easily available. On
the other hand, malicious samples are scarce or unavailable. The lack of a
thorough dataset of attack samples limits the usage of supervised techniques.
Finally, most existing detection approaches assume that FL clients maintain
labeled datasets that may use at each round. This assumption may not be
realistic, as CAVs cannot label the network flow on every turn.

As a CAV runs a set of well-known applications (safety, convenience,
commercial, etc), their communication pattern should present a high degree of
regularity so long as they are not under attack or faulty. Similarly, an
attack must alter its communication pattern. Therefore, we believe it is
possible to use anomaly detection techniques to model the CAV’s benign (or
expected) communication pattern and detect attacks as anomalous occurrences.
To overcome the aforementioned limitations of the existing IDSs, in this
paper, we propose an unsupervised federated learning based IDS that leverages
a deep auto-encoder model to train the detection model, relying only on benign
network traffic. Thanks to federated learning, the proposed IDS can be trained
with large and diverse benign network traffic, while preserving the CAVs’
privacy. The proposed IDS aggregates the detection model updates within the
Multi-access Edge Computing (MEC) server to enhance the learning efficiency
and minimize latency. The in-depth experiment on a recent network traffic
dataset shows that the proposed system achieves a high detection rate while
minimizing the false positive rate, and the detection delay.

The remainder of this paper is organized as follows. Section II describes
related work. The design of our scheme is presented in Section III. Section IV
depicts the performance evaluation results, and finally, Section V concludes
the paper.

##  II Related Work

Several distributed ML-based IDSs have been proposed for detecting attacks in
5GB-enabled CAVs. Negi et al. [8] presented a DL-based IDS to detect anomalies
in ITS based on Long Short-Term Memory (LSTM). In this work, time series data
are collected by CAVs and sent to the cloud to enable the training and
retraining of a global model using a cluster of servers instead of one server.
Shu et al. [9] proposed a collaborative IDS based on supervised DL, Generative
Adversarial Networks (GANs), and Software Defined Networking (SDN). The
proposed system enables distributed SDN controllers managing sub-networks of
CAVs to train a global model for the whole network without directly exchanging
their sub-network flows. However, both [8] and [9] raise privacy issues since
datasets are shared between learning nodes. Zhang et al. [10] proposed a
distributed ML-based IDS that enables CAVs to directly communicate to train a
global model based on supervised learning without sharing their datasets.
However, peer-to-peer distributed learning generates a large overhead,
degrading communication performance. Uprety et al. [11] proposed a FL-based
privacy-preserving collaborative IDS for CAVs. This work enables CAVs (FL
clients) to train DL models on a locally labeled dataset and share their
parameters with the central FL server to build a global model. Boualouache et
al. [12] proposed a FL-based privacy-preserving collaborative IDS based on
supervised learning that leverages a set of FL servers to train the global
model. Liu et al. [13] proposed FL for collaborative IDS. This work suggests
offloading the training to distributed vehicular edge nodes. Specifically,
CAVs act as FL clients for building models based on their locally labeled
datasets and Roadside Units (RSUs) for aggregating global models. Hbaieb et
al. [14] proposed SDN-FL-based IDS for CAVs. In this work, SDN controllers
train local models based on labeled datasets built using data collected from
CAVs, while the aggregation of global models is performed on the cloud.

Overall, existing FL-based IDSs for 5GB-enabled CAVs [11, 12, 13, 14] have
specifically two main limitations: (i) they are based on supervised learning
which limits their effectiveness against unseen and zero-day attacks, and (ii)
they assume that FL clients have labeled datasets, which in practice might be
unrealistic. Considering these gaps, we propose a novel network-based IDS
trained using a deep auto-encoder model. Relying only on benign network
traffic, our system can detect unseen or zero-day attacks so long as they
alter the benign communication pattern of the CAV. Additionally, our solution
does not compromise the CAV’s privacy since it is built through federated
learning.

##  III Proposed Solution

Collaborative learning allows training the model with a large amount of
network traffic from diverse CAVs, while preserving data privacy, and
minimizing the communication overhead. The proposed MEC-enabled learning
scheme trains the deep Auto-Encoder (AE) model in a federated way, as
illustrated in Figure 1. First, the raw captured packets are converted to
flows. Then, for each flow, a set of pertinent features are calculated. Next,
the local dataset of flows is fed to the AE model initially distributed by the
MEC server. The training rounds are orchestrated by the MEC server and
executed by the AE on the participating CAV’s local dataset.

Figure 1: MEC-Enabled Federated learning architecture

###  III-A Flow extraction & features engineering

First, to identify a traffic flow, we use a combination of five properties
from the packet header, including the network and the transport layer headers
of the TCP/IP protocol stack. These are as follows: source IP address,
destination IP address, source port number, destination port number, and
protocol. For each flow extracted, a set of features are calculated according
to a given time window (ex. 100 seconds). Flow features include mainly packet
header characteristics and statistics computed from the aggregating network
and transport layers header information of the packets in a flow. The network
features are related to: time, packets, bytes, and flag groups. The list of
features as well as their description are given in table I.

TABLE I: List of network features

Feature names Description numHdrs Number of headers (depth) in hdrDesc l4Proto
Layer 4 protocol (TCP, UDP, HOPOPT,…) dstPortClass Port based classification
of the destination port name (HTTPS, Telnet, NTP, ..) numPktsSnt, numByteSnt
Number of transmitted packets/ byte per second minPktSz, maxPktSz, avePktSize,
stdPktSize Minimum/Maximum/Average/ Standard deviation of layer 3 packet size
pktAsm, bytAsm Packet/ Byte stream asymmetry duration Duration of the flow in
seconds minIAT, maxIAT, aveIAT, stdIAT Minimum/ Maximum/ Average/ Standard
deviation of inter-arrival time pktps/ bytps Number of sent packets/ byte per
second ipTTLChg, ipTOS, ipFlags IP TTL change count, IP Type of Service, IP
aggregated flags tcpISeqN TCP Initial Sequence Number tcpSeqFaultCnt TCP
sequence number fault count tcpPAckCnt TCP packet ack count
tcpFlwLssAckRcvdBytes TCP flawless ack received bytes tcpInitWinSz,
tcpAveWinSz, tcpMinWinSz, tcpMaxWinSz, TCP initial/ average / minimum/ Maximum
effective window size tcpWinSzDwnCnt, tcpWinSzUpCnt TCP effective window size
change down/up count tcpWinSzThRt TCP packet count ratio below window size
WINMIN threshold tcpFlags TCP aggregated protocol flags (cwr, ecn, urgent,
ack, push, reset, syn, fin) tcpAnomaly TCP aggregated header anomaly flags
tcpOptions TCP aggregated options tcpMSS TCP maximum segment size tcpEcI TCP
estimated counter increment tcpUtm, tcpBtm TCP estimated up/boot time
tcpSSASAATrip TCP trip time SYN, SYN-ACK Destination — SYN-ACK, ACK Source
tcpRTTAckTripMin, tcpRTTAckTripMax, tcpRTTAckTripAve TCP ACK minimum/ maximum/
average trip time tcpRTTAckTripJitAve, tcpRTTAckJitAve TCP ACK trip/ round
trip time average jitter tcpRTTSseqAA TCP round trip time {SYN, SYN-ACK, ACK}
and {ACK-ACK}

###  III-B Modeling benign network traffic pattern

The Auto-Encoder (AE) model [15] is an unsupervised model that compresses
input vectors as code vectors using a set of recognition weights and then
converts back to $m$ $(m<d)$ number of neurons reconstructed input vectors
using a set of generative weights. There are two major parts in an AE
architecture: the encoder and the decoder. The encoder reduces the dimension
of the input vectors ($x_{i}\in R^{d}$) to numbers of neurons that form the
hidden layer. The activation of the neuron $i$ in the hidden layer is given
by:

$h_{i}=f^{{}_{\theta}}(x)=s(\sum_{j=1}^{n}W_{ij}^{input}x_{j}+b_{i}^{input})$
(1)

where $x$ is the input vector, $\theta$ is the parameters
$\begin{Bmatrix}W^{input},b^{input}\end{Bmatrix}$, $W$ is an encoder weight
matrix of dimension $m\times d$, while $b$ is a bias vector of dimension $m$.
Thus, the input vector is encoded to a vector with fewer dimensions. The
decoder maps the low-dimensional hidden representation $h_{i}$ to the original
input space $R^{d}$ by the same transformation as the encoder. The function of
mapping is as follows:

$x_{{}^{\prime}}^{i}=g_{\theta^{\prime}}(h)=s(\sum_{j=1}^{n}W_{ij}^{hidden}h_{j}+b_{i}^{hidden})$
(2)

The set of decoder parameters is
$\theta^{\prime}(W^{hidden}h_{j}+b^{hidden})$. The objective of an autoencoder
is to minimize the reconstruction error relative to $\theta$ and
$\theta^{\prime}$ :

$\displaystyle\theta^{*},\theta^{{}^{\prime}*}=arg_{\theta,\theta^{{}^{\prime}}}min\frac{1}{n}\sum_{i=1}^{n}\varepsilon(x_{i},x_{i}^{{}^{\prime}})$
(3)
$\displaystyle=arg_{\theta,\theta^{{}^{\prime}}}min\frac{1}{n}\sum_{i=1}^{n}\varepsilon(x_{i},g_{\theta^{{}^{\prime}}}(f_{\theta}(x_{i})))$
(4)

The reconstruction error is utilized as the anomaly score. Network flows with
significant reconstruction errors are regarded as malicious flows (anomalies).
Only benign flows are used to train the AE model. After training, the AE model
will reconstruct benign flows exceptionally well, but not malicious flows that
it has never seen. Algorithm 1 illustrates the anomaly detection process using
the reconstruction errors of the AE model. The threshold $\alpha$ is the sum
of the sample’s mean squared error (MSE) median and the sample’s five times
the MSE median absolute deviation (MAD) over the validation set. MAD uses the
deviation from the median, which is less likely to be skewed by outlier
values.

1

2 BEGIN

3 PHASE 1: Flow extraction & Preprocessing

4 INPUT: $pkts$: raw packets, $TW$: Time window (s)

5 Extract flow from $pkts$ according to the $TW$

6 $Data\leftarrow$ Calculate $l$ features vectors (see table I for features
list)

7 $X_{Tr},X_{V},X_{Te}\leftarrow$ Splitting $Data$

8 ## PHASE 2: Training the AE Model

9 INPUT: $X_{Tr}$ : Train dataset, $X_{V}$: Validation dataset

10 $\phi,\theta\leftarrow$ train the AE on $X_{Tr}$

11 for  _$i\in\\{1,...,N\\}$_ do

12 $RE_{V}[i]=\left\|x_{V}^{(i)}-g_{\theta}(f_{\phi}(x_{V}^{(i)}))\right\|$

13 end for

14$\alpha=\widetilde{MSE}_{RE_{V}}+5\times median(RE_{V}[i]-median_{RE_{V}})$

15

16## PHASE 3: Testing

17 INPUT: $X_{Te}$: Test dataset, $\alpha$: Threshold

18

19for  _$i\in\\{1,...,N\\}$_ do

20 $RE[i]=\left\|x_{Te}^{(i)}-g_{\theta}(f_{\phi}(x_{Te}^{(i)}))\right\|$

21 if  _$RE[i] >\alpha$_ then

22 $x_{Te}^{(i)}$ is a malicious flow

23 else

24 $x_{Te}^{(i)}$ is a benign flow

25 end if

26

27 end for

END

Algorithm 1 AE-based Anomaly Detection

###  III-C MEC-enabled federated training process

First, the MEC server initializes the learning parameters of the shared model
in terms of neural network configuration (number of layers, number of neurons,
activation functions, etc.), batch size, learning rate, number of epochs, etc.
It then shares such parameters with the participant CAVs. Each CAV computes
local updates on top of both the AE model and its local data. Once done, each
CAV sends its local model’s parameters (weights) to the MEC server. The latter
aggregates the received local models to generate a global learning model,
before sending it back to the involved CAVs, in order to initiate a new
training round.

In our study, the federated learning problem across multiple CAVs is
formulated as a federated optimization problem and resolved using the FedAvg
algorithm [16]. Indeed, using its local data, each CAV calculates the average
gradient on top of the model $w$ for a corresponding training round $r$.
Thereafter, each CAV performs a local gradient descent on the currently used
model with its own data. On the other hand, the MEC server aggregates these
local updates and transfers back the global model to the CAV collaborators.
This process is repeated during a number of rounds, defined initially by the
MEC server. Algorithm 2 illustrates the main steps performed by both the
central MEC server and each participating CAV (Client).

1

2

3

4BEGIN

5 Variables: K: index of clients, B: local batch size, E: number of local
epochs, $\eta$ : learning rate

6

7ClientUpdate: #

8 $\beta\leftarrow$ (split $\mathit{P_{k}}$ into batch of size $\mathit{B}$ )

9 for  _each local epoch $i\in\\{1,...,E\\}$_ do

10 for  _batch $b\in\textit{B}$_ do

11
$w\leftarrow\textit{w}-\eta\bigtriangledown\mathit{\mathfrak{\mathit{l}}}(w;\mathit{b})$

12

13 end for

14 return $w$to server

15 end for

16

17Server executes:

18 Initialize ${w}_{0}$

19

20for  _each round $t\in\\{1,...,N\\}$_ do

21 $m\leftarrow$ max(C. K, 1)

22 $S_{t}\leftarrow$ (random set of m clients)

23 for  _each client $k\in S_{t}$ in parallel_ do

24 $w_{t+1}^{k}\leftarrow ClientUpdate(k,w_{t})$

25 $w_{t+1}\leftarrow\sum_{k=1}^{k}\frac{n_{k}}{n}w_{t}^{k}+1$

26

27 end for

28

29 end for

30

END

Algorithm 2 Federated Averaging Algorithm

##  IV Performance Evaluation

In this section, we first briefly describe the dataset [17] used in this
research. Then, we present and discuss in detail the detection performances of
the proposed system. Finally, we compare our approach with supervised and
centralized approaches.

###  IV-A Dataset preprocessing & features enginnering

To the best of our knowledge, VDoS [17] is the only publicly available dataset
that includes benign and malicious traffic generated based on a realistic
testbed. The network traffic was gathered in three different settings: urban,
rural, and highway. The experimental environment included two vehicles, 3
physical machines, 4 virtual machines, 2 access points, a 4G modem, and two
Cisco antennas. Common user applications such as Google Maps, YouTube, social
networks, and other real-time applications (video/audio calls) have been run
to generate benign network traffic. To generate malicious traffic, three Kali-
Linux machines run three scenarios of DoS attack: UDP Flood, SYN Flood, and
Slowloris packets alternately. In this research, we do not consider the third
scenario, because we believe it is quite unusual that a CAV may hosts a web
server. For further details about the testbed and the dataset generation
please refer to [17].

TABLE II: Dataset samples distribution | Highway | Rural | Urban
---|---|---|---
Normal | 48303 | 32303 | 65742
SynFlood | 65790 | 65790 | 67043
UdpFlood | 65790 | 65790 | 65790

To extract flows and calculate features from raw traffic (PCAP files), we
developed some scripts using Tranalyzer flow traffic exporter [18]. We tried
several time windows (TW) to sample the network traffic. Having achieved
similar performance, and taking into account that a smaller TW allows faster
detection, we fixed the TW to 1s. Table II presents the number of benign and
malicious flows extracted for each type of environment. Pearson correlation
filter is then used to discard highly correlated features ( $>95\%$).

###  IV-B Experimental results

We trained and tested the proposed system in the Google Colab cloud
environment. We used the Pytorch package to implement the local and federated
learning models. We implemented a deep auto-encoder with three hidden layers
(50% dimension decrease from one layer to another). The binary Mean Squared
Error (MSE) loss function was used. Table III illustrates the hyperparameter
setup used for local and federated learning. To evaluate the detection
performance of our solution, in addition to the false positive rate (FPR), we
considered the following metrics :

$\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$ (5)

$\text{F1-Score}=\frac{2TP}{2TP+FP+FN}$ (6)

TABLE III: Hyperparameters values

Hyperparameter Value Description lr 0.012 Learning rate for the deep AE
Nb_clients 10 Number of clients Nb_selected 3-6 Number of clients we choose
for train Batch_size 128 Defines the dataset size in each training iteration
R_samp_sz 1000 Choose some data from the train set to retrain the data from
trained model Nb_rounds 20 Total number of communication rounds for the global
model to train Epochs 15 For train client model Nb_retrain_epochs 5 Total
number for retrian the global server after receiving the model weights
Nb_local_epochs 50 Only for the local deep AE training

Overall, the proposed solution presented a high detection rate with a low
false positive rate. The detection model was able to discriminate between the
reference benign traffic profile and malicious network traffic. Table IV shows
how the average accuracy, F1-score, and false positive rate may change by
varying the number of participating CAVs. Using just three CAVs allowed for
nearly the same detection performances while cutting the training time by
about 30%. We can see from figure 2, that there is no further improvement in
terms of loss function score beyond the $11^{th}$ round. This demonstrates
that the detection model did not require a large number of communication
rounds to converge.

As can be observed from figure 3, the proposed system performed better against
the SynFlood attack. This can be explained by the relatively large number of
TCP-related features (compared to UDP) included in the features vector.
Although 99.99% F1-Score and 0.01% FPR rate have been achieved using
supervised learning, specifically, decision tree in [17], our system shows
comparable detection performances using only benign traffic. To get a good
idea of how well the proposed model worked, we compared it to a centralized
model that was trained with the whole dataset. The results comparison depicted
in figure 4 shows that the federated model yielded remarkably similar
detection performances as the centralized model.

##  V Conclusion

New attack vectors have emerged from the integration of V2X communication in
the 5G ecosystem, which may lead to hazardous situations for road users. Most
of the existing IDSs in 5G-V2X are either unable to detect emerging zero-day
attacks because they rely on supervised learning, or do not meet privacy
requirements due to data collection required for centralized learning. To
tackle these limits, we proposed in this paper a new IDS based on a deep auto-
encoder model, which leverages the predictability of benign network traffic to
detect attacks. Relying on federated training orchestrated by the MEC server,
the proposed IDS did not require any data collection or labeling. Through in-
depth experiments on a recent dataset, we have shown that the proposed IDS
provides high performance even with few communication rounds and a short TW
sampling, which allows fast training and low detection delay. In future work,
we plan to evaluate the proposed IDS on other non-Identical Independent
Distribution (non-IID) datasets including more sophisticated and recent
attacks.

TABLE IV: Evaluation of detection performances Nb. Clients | Accuracy | F1-Score | FPR | Time (mn)
---|---|---|---|---
10 | 87.94% | 91.21% | 6.95% | 14.80
8 | 87.94% | 91.22% | 6.98% | 11.53
6 | 87.95% | 91.23% | 7.06% | 9.32
3 | 87.94% | 91.21% | 6.92% | 4.43
Figure 2: Loss Vs Nb. rounds
SynFloodUdpFlood$0$$20$$40$$60$$80$$100$$89.13$$88.94$$90.05$$89.9$$6.59$$7.2$AccuracyF1-ScoreFPR
Figure 3: Predictive performances per attack scenario
CentralizedFederated$0$$20$$40$$60$$80$$100$$83.99$$87.94$$90.34$$91.21$$6.92$$6.92$AccuracyF1-ScoreFPR
Figure 4: Performances comparison between centralized and federated learning

## Acknowledgment

This work was supported by the 5G-INSIGHT bilateral project (ID: 14891397) /
(ANR-20-CE25-0015-16), funded by the Luxembourg National Research Fund (FNR),
and by the French National Research Agency (ANR).

## References

  * [1] Ijaz Ahmad, Shahriar Shahabuddin, Tanesh Kumar, Jude Okwuibe, Andrei Gurtov, and Mika Ylianttila.  Security for 5g and beyond.  IEEE Communications Surveys & Tutorials, 21(4):3682–3722, 2019\.
  * [2] Mario H. Castañeda Garcia, Alejandro Molina-Galan, Mate Boban, Javier Gozalvez, Baldomero Coll-Perales, Taylan Şahin, and Apostolos Kousaridas.  A Tutorial on 5G NR V2X Communications.  IEEE Communications Surveys Tutorials, 23(3):1972–2026, 2021.
  * [3] 3GPP TS 23.287.  Architecture enhancements for 5G System (5GS) to support Vehicle-to-Everything (V2X) services, Jul 2020.
  * [4] Noble Arden Elorm Kuadey, Gerald Tietaa Maale, Thomas Kwantwi, Guolin Sun, and Guisong Liu.  DeepSecure: Detection of Distributed Denial of Service Attacks on 5G Network Slicing-Deep Learning Approach.  IEEE Wireless Communications Letters, 2021.
  * [5] Abdelwahab Boualouache and Thomas Engel.  A Survey on Machine Learning-based Misbehavior Detection Systems for 5G and Beyond Vehicular Networks.  arXiv preprint arXiv:2201.10500, 2022.
  * [6] Rabah Rahal, Abdelaziz Amara Korba, Nacira Ghoualmi-Zine, Yacine Challal, and Mohamed Yacine Ghamri-Doudane.  Antibotv: A multilevel behaviour-based framework for botnets detection in vehicular networks.  Journal of Network and Systems Management, 30(1):1–40, 2022.
  * [7] Wided Hammedi, Bouziane Brik, and Sidi Mohammed Senouci.  Federated deep learning-based framework to avoid collisions between inland ships.  In 2022 International Wireless Communications and Mobile Computing (IWCMC), pages 967–972, 2022.
  * [8] Naman Negi, Ons Jelassi, Hakima Chaouchi, and Stephan Clemençon.  Distributed online Data Anomaly Detection for connected vehicles.  In 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pages 494–500. IEEE, 2020.
  * [9] Jiangang Shu, Lei Zhou, Weizhe Zhang, Xiaojiang Du, and Mohsen Guizani.  Collaborative intrusion detection for VANETs: a deep learning-based distributed SDN approach.  IEEE Transactions on Intelligent Transportation Systems, 2020.
  * [10] Tao Zhang and Quanyan Zhu.  Distributed privacy-preserving collaborative intrusion detection systems for VANETs.  IEEE Transactions on Signal and Information Processing over Networks, 4(1):148–161, 2018.
  * [11] Aashma Uprety, Danda B Rawat, and Jiang Li.  Privacy Preserving Misbehavior Detection in IoV using Federated Machine Learning.  In 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC), pages 1–6. IEEE, 2021.
  * [12] Abdelwahab Boualouache and Thomas Engel.  Federated learning-based scheme for detecting passive mobile attackers in 5G vehicular edge computing.  Annals of Telecommunications, 77(3):201–220, 2022.
  * [13] Hong Liu, Shuaipeng Zhang, Pengfei Zhang, Xinqiang Zhou, Xuebin Shao, Geguang Pu, and Yan Zhang.  Blockchain and Federated Learning for Collaborative Intrusion Detection in Vehicular Edge Computing.  IEEE Transactions on Vehicular Technology, 2021.
  * [14] Amal Hbaieb, Samiha Ayed, and Lamia Chaari.  Federated learning based IDS approach for the IoV.  In Proceedings of the 17th International Conference on Availability, Reliability and Security, pages 1–6, 2022.
  * [15] Geoffrey E Hinton and Richard Zemel.  Autoencoders, minimum description length and helmholtz free energy.  Advances in neural information processing systems, 6, 1993.
  * [16] H. B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In AISTATS, 2017.
  * [17] Rabah Rahal, Abdelaziz Amara Korba, and Nacira Ghoualmi-Zine.  Towards the development of realistic dos dataset for intelligent transportation systems.  Wireless Personal Communications, 115(2):1415–1444, 2020.
  * [18] S. Burschka and B. Dupasquier.  Tranalyzer: Versatile high performance network traffic analyser.  In 2016 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1–8, 2016.