# Who Wrote it and Why?
Prompting Large-Language Models for Authorship Verification

Chia-Yu Hung, Zhiqiang Hu, Yujia Hu, Roy Ka-Wei Lee
Singapore University of Technology and Design
{chiayu_hung<EMAIL_ADDRESS>
{yujia_hu<EMAIL_ADDRESS>
###### Abstract

Authorship verification (AV) is a fundamental task in natural language
processing (NLP) and computational linguistics, with applications in forensic
analysis, plagiarism detection, and identification of deceptive content.
Existing AV techniques, including traditional stylometric and deep learning
approaches, face limitations in terms of data requirements and lack of
explainability. To address these limitations, this paper proposes PromptAV, a
novel technique that leverages Large-Language Models (LLMs) for AV by
providing step-by-step stylometric explanation prompts. PromptAV outperforms
state-of-the-art baselines, operates effectively with limited training data,
and enhances interpretability through intuitive explanations, showcasing its
potential as an effective and interpretable solution for the AV task.

##  1 Introduction

Motivation. Authorship verification (AV) is a fundamental task in the field of
natural language processing (NLP). It aims to determine the authorship of a
given text by analyzing the stylistic characteristics and patterns exhibited
in the writing Stamatatos (2016). The importance of AV lies in its wide range
of applications, including forensic analysis Iqbal et al. (2010), plagiarism
detection Stein et al. (2011), and the detection of deceptive or fraudulent
content Claxton (2005). The rise of Large-Language Models (LLMs) Chowdhery et
al. (2022); Brown et al. (2020), has facilitated the generation of human-like
text where one would now need to factor in the possibility of machine-
generated text Uchendu et al. (2023). This development give rise to a new
layer of complexity in distinguishing between machine-generated and human-
written text, subsequently amplifying the significance of the AV problem.

Over the years, various techniques have been proposed to address the AV task.
Traditional approaches relied on stylometric features, such as n-gram
frequencies, vocabulary richness, and syntactic structures, to capture the
unique writing style of individual authors Ding et al. (2017). Machine
learning techniques, such as support vector machines (SVMs) and random
forests, have been employed to model the relationship between these
stylometric features and authorship Brocardo et al. (2013). More recently,
deep learning approaches have shown promising AV results by learning intricate
patterns directly from raw text data Bagnall (2015); Ruder et al. (2016);
Fabien et al. (2020).

Despite these advancements, existing AV techniques have certain limitations.
Firstly, most existing AV methods require a large amount of labeled training
data, which can be costly to acquire, especially for scenarios with limited
available data. Secondly, there is a lack of explainability in the predictions
made by these techniques. In many practical applications, it is crucial to
understand why a particular decision was reached, which is particularly
relevant in legal contexts or situations where interpretability is paramount.

Research Objectives. To address these limitations, We propose PromptAV, a
novel technique that utilizes LLMs for authorship attribution by employing
step-by-step stylometric explanation prompts. These prompts guide the LLMs to
analyze and evaluate the textual features that contribute to authorship,
enabling the model to provide explanations for its predictions. PromptAV
demonstrates effectiveness in both zero-shot and few-shot settings, offering
improved performance compared to existing baselines, especially in scenarios
with limited training data. Additionally, detailed case studies showcase
PromptAV’s ability to provide intuitive explanations for its predictions,
shedding light on the factors contributing to authorship attribution and
enhancing the interpretability of the AV process.

Figure 1: Prompts used by (a) CoT, (b) PS+ prompting, (c) PromptAV for a AV
task. The trigger intructions of the various techniques are highlighted in the
prompt.

Contribution. This paper introduces PromptAV, a prompt-based learning
technique that harnesses the linguistics capabilities of LLMs for AV. By
providing step-by-step stylometric explanation prompts, PromptAV achieves
improved performance, offers explainability in its predictions, and operates
effectively in both zero-shot and few-shot settings. The experimental results
demonstrate the potential of PromptAV as an effective and interpretable
solution for the AV task.

##  2 Related Work

AV is a well-established research topic. Classical approaches to AV have
predominantly centered on stylometric features encompassing n-gram
frequencies, lexical diversity, and syntactic structures to discern the unique
writing styles intrinsic to individual authors Stamatatos (2016); Lagutina et
al. (2019). Classical machine learning algorithms were also used to model the
associations between these stylometric attributes and authorship Brocardo et
al. (2013); Hu et al. (2023). Recently, deep learning models such as RNNs and
CNNs being employed to extract more complex patterns from textual data Bagnall
(2015); Ruder et al. (2016); Benzebouchi et al. (2018). These deep learning
architectures have exhibited promising advancements in AV tasks. Furthermore,
the advent of the Transformer architecture Vaswani et al. (2017) has prompted
the development of Transformer-based AV models Ordoñez et al. (2020); Najafi
and Tavan (2022); Tyo et al. (2021).

##  3 Methodology

In an recent work, Wei et al. (2023) have proposed Chain of thought (CoT)
prompting, which employs a series of intermediate reasoning steps to
significantly improve the ability of LLMs in performing complex reasoning.
Wang et al. (2023) further extended the concept to propose PS+ prompting, a
zero-shot prompting methodology that instructs LLMs to formulate a strategic
plan prior to problem-solving.

Inspired by these works, we introduce PromptAV, a prompting strategy that
incorporates key linguistic features—identified in Boenninghoff et al. (2019)
as the crucial variables for LLMs to evaluate. These linguistic features serve
as rich, often subtle, markers of an author’s distinct writing style and are
hence of paramount importance for the AV task. For instance, the consistent
use or avoidance of certain punctuation marks, or the preference for
particular special characters, can be idiosyncratic to an author. Similarly,
elements like tone and expression, though harder to quantify, can offer
significant insights into an author’s unique ‘voice’. The integration of these
linguistic attributes within the PromptAV framework does not simply add layers
of complexity to the model, it fundamentally enhances the model’s ability to
capture the author-specific nuances within textual data. This capability, in
turn, improves the model’s accuracy and reliability when attributing
authorship to unidentified texts, making PromptAV a powerful tool in the field
of AV. Figure 1 provides an overview that compares the prompts used in CoT,
PS+ prompting, and PromptAV.

We also note that conventional prompting techniques that instruct the LLM to
respond strictly with binary “yes” or “no” answers frequently result in the
LLM responding ‘no” for the majority of instances within an AV task. To
mitigate this problem, PromptAV instructs the LLM to generate a confidence
score ranging from 0 to 1, rather than a binary response. To calibrate these
generated confidence scores, we augment the prompt with the additional
directive “Your answer should reflect a moderate level of strictness in
scoring”.

###  3.1 Few Shot Prompting

In the conventional setup of few-shot CoT prompting, each example necessitates
the manual crafting of reasoning steps, an approach that is impractical for
the AV task. This is primarily due to the challenges in ensuring the
consistent quality of hand-crafted reasoning across diverse prompting
templates. Hence, we resort to leveraging the capacity of LLMs as zero-shot
reasoners to generate the required reasoning steps Kojima et al. (2023).

To construct the few-shot PromptAV and CoT examples, we introduce an
additional directive to the original prompt: “It is given that after following
the instruction, the confidence score obtained is [X]. Show the step-by-step
execution of the instruction so that this score is achieved.” Here, [X] is
either 0.9 or 0.1, contingent upon whether the pair of texts were written by
the same author. Subsequently, we feed this modified prompt to GPT-4
111https://openai.com/research/gpt-4 to generate intermediate reasoning steps.

Zhao et al. Zhao et al. (2021) has demonstrated that both the selection of
training examples and their sequencing can significantly influence
accuracy.Therefore, to ensure fairness, we utilized the same examples for all
our 2-shot and 4-shot experiments. These examples were randomly sampled from
the training set and presented to the LLM in the same order.

Setting | Model | Accuracy
---|---|---
0-Shot | PromptAV | 0.587
| PS+ prompting | 0.536
| CoT | 0.524
2-Shot | PromptAV | 0.667
| CoT | 0.595
4-shot | PromptAV | 0.635
| CoT | 0.510
Table 1: Performance of PromptAV and the baselines on k-shots settings.
Highest accuracy are bolded. Text 1: " We’re all sinners , or about to be . "
I picked up this flick as part of a double film DVD offer along with " The
Hanged Man " for just a buck , and truth be told , both movies were a whole
lot better than either had a right to be for the price . The title on this one
was " The Gun and The Cross " , close enough to the original to get the
message across . The title character wields both , as Marjoe Gortner is
revealed to be gunfighter Ernie Parsons , masquerading as a preacher after he
finds a real one who was bushwhacked . Donning the clothes of the dead
minister , he arrives at the town of Castle Walk just in time to help the
local folk deal with nasty town boss Ross ( David Huddleston ) … [Truncated
due to length restriction] Text 2: " This is a mad dream Antinea , but I shall
put an end to the madness . " The title " Hercules and the Captive Women " is
a bit of a misnomer , there was only one woman captive , and along with her
mother , Queen Antinea of Atlantis , were the only two women in the entire
film . The International English title mentions the ’ Conquest of Atlantis ’ ,
and that probably makes more sense , but in the end , it’s the destruction of
Atlantis that takes center stage . There’s an entire mythology offered by
Zantas , the last high priest of Uranus , that explains what’s going on with
Antinea and her cohort of Grand Vizier clones . … [Truncated due to length
restriction]
---
PromptType |  Generated Explanation
0-shot PromptAV |  Based on the provided texts, here is a general assessment of the likelihood that Text 1 and Text 2 were written by the same author: 1\. Punctuation style: Both texts use similar punctuation styles, such as commas, periods, and quotation marks. However, Text 1 uses more parentheses and dashes compared to Text 2. Confidence: 0.7 2\. Special characters style, capitalization style: Both texts have similar capitalization styles, with proper nouns and the beginning of sentences capitalized. However, Text 1 has more continuous capitalization (e.g., "The Gun and The Cross") compared to Text 2. Confidence: 0.6 3\. Acronyms and abbreviations: Both texts use acronyms and abbreviations, but there are no significant differences in their usage. Confidence: 0.8 4\. Writing style: Both texts have a similar writing style, with a mix of descriptive and narrative elements. However, Text 1 focuses more on reviewing a movie, while Text 2 provides a summary of a film’s plot. Confidence: 0.7 5\. Vocabulary usage: Both texts use a variety of vocabulary, but there are no significant differences in their usage. Confidence: 0.8 Overall, based on the assessment of the relevant variables, there is a moderate likelihood (confidence score: 0.7) that Text 1 and Text 2 were written by the same author.
0-shot CoT |  Based on a moderate level of strictness in scoring, the likelihood that Text 1 and Text 2 were written by the same author is low. The two texts have different subject matters and writing styles, with Text 1 discussing a western film and Text 2 discussing a Hercules film. Additionally, the tone and language used in each text are distinct. Therefore, it is unlikely that the same author wrote both texts. Based on these factors, I would assess the likelihood that Text 1 and Text 2 were written by the same author as low, with a confidence level of around 0.3 on the scale of 0 to 1.
0-shot PS+ prompting |  Based on a moderate level of strictness in scoring, I would assess the likelihood that Text 1 and Text 2 were written by the same author as low. The two texts have different themes, writing styles, and subject matters. Text 1 discusses a western film, while Text 2 discusses a Hercules film. The language and tone used in each text are also distinct. Therefore, I would assign a confidence score of 0.2, indicating a low likelihood of the same authorship.
Table 2: A text pair example with the generated responses from PromptAV, CoT and PS+ Prompting Model | Accuracy
---|---
PromptAV | 0.587
PromptAV_Obfuscated | 0.580
PS+ prompting | 0.536
CoT | 0.524
Table 3: Performance of PromptAV against a series of authorship obfuscation
methods. Highest accuracy are bolded

##  4 Experiments

###  4.1 Experimental Settings

Dataset. We evaluate PromptAV and the baselines using IMDb62 Seroussi et al.
(2014), which is commonly used dataset for AV task. Each record in dataset
contains a pair of texts and the ground truth labels, i.e., a “positive” pair
of texts belong to the same author, and “negative” for otherwise. As we are
interested in performing the AV task in zero-shot and few-shot settings, we
focus on sampling a manageable test set of 1,000 text pairs from the original
IMDb62 dataset. The resulting test set comprises 503 positive and 497 negative
text pairs.

Implementation. All of our experiments were conducted on the gpt-3.5-turbo
model, with the temperature parameter set to 0. We benchmarked PromptAV
against PS+ prompting in zero-shot settings and against CoT prompting in zero,
two, and four-shot settings. Given that all three methods generate a
confidence score in the range of 0 to 1, we reported the accuracy
corresponding to the optimal threshold that yields the highest accuracy. We
call this the Optimal Threshold Methodology and we will dicuss more about it
in Appendix A.3.

###  4.2 Experimental Results

Table 1 shows the performance of PromptAV and the baselines on $k$-shot
settings. We observe that PromptAV consistently achieved higher accuracy
across varying shot settings. This demonstrate PromptAV’s efficacy in
performing AV task with little or no training data. Furthermore, as the number
of shots increases from 0 to 4, PromptAV maintains its edge over the CoT. This
suggests that PromptAV effectively leverages the additional information
provided in the few-shot scenarios, optimizing its decision-making process.

The superior performance of PromptAV over PS+ prompting and CoT in the zero-
shot setting underscores its capacity to make effective use ofkey linguistic
features for AV even in the absence of training examples. This is indicative
of the fact that PromptAV is adept at understanding intrinsic linguistic
characteristics, which are critical in differentiating writing styles of
various authors.

###  4.3 Explainability Case Studies

While achieving explainability in the AV task is notoriously difficult,
PromptAV rises to the challenge by providing detailed, interpretable
solutions. This is accomplished through a meticulous analysis of the
linguistic features. A sample of the generated responses from PromptAV, CoT,
and PS+ prompting is illustrated in Table 2. In contrast to CoT prompting or
PS+ prompting, PromptAV offers a comprehensive comparison of linguistic
features, such as punctuation style and vocabulary usage. For example,
PromptAV detects that both texts have similar capitalization style and writing
style, with a mix of descriptive and narrative elements. On the other hand,
CoT and PS+ prompting systems deliver a more superficial level of analysis.
While they can recognize discrepancies in vocabulary selection across texts,
they lack the capacity to provide an exhaustive explanation or a deeper
analysis of these differences.

###  4.4 Authorship obfuscation

In our pursuit to understand PromptAV’s potential robustness, we took a
focused approach by testing it against an obfuscation dataset. We selected the
suggested Mutant-X algorithm Mahmood et al. (2020), a recognized method in the
authorship obfuscation domain, and applied it to the IMDb62 dataset that was
referenced in our paper. We apply PromptAV in zero-shot setting on the
obfuscated dataset and benchmark against its performance on the original
dataset. The experimental results are presented in Table 3. The results reveal
a negligible decline in performance when confronted with obfuscated text.
Interestingly, the accuracy achieved, even with obfuscation, surpasses the
results of zero-shot CoT and PS+ prompting on non-obfuscated text. This
indicates a promising level of resilience in PromptAV against authorship
obfuscation methods. While these preliminary results are encouraging, we
acknowledge the need for more comprehensive testing against a broader range of
obfuscation methods to assert PromptAV’s robustness conclusively.

###  4.5 Stylometric Feature Impact

The field of stylometry is vast, with a plethora of features available for
authorship analysis. The central aim of PromptAV was to showcase the
flexibility and adaptability of the framework when working with different sets
of stylometric features. We conducted a series of experiments using varying
sets of stylometric features with PromptAV in zero-shot setting. We report the
result in Table 4. The result clearly showcases that the performance varies
based on the chosen feature set. The peak accuracy achieved with a 9-feature
set emphasizes that PromptAV’s success is contingent on the quality and
appropriateness of the selected features. We genuinely believe that the field
holds significant undiscovered potential, with combinations of features that
could further elevate the performance of PromptAV. Our results only scratch
the surface, and our intention is to spur the community into exploring these
myriad possibilities.

##  5 Conclusion

In this paper, we proposed PromptAV, a prompt-based learning technique that
harnesses the linguistics capabilities of LLMs for AV. Our experimental
results demonstrated PromptAV’s superior performance over the state-of-the-art
baselines in zero-shot and few-shot settings. Case studies were also conducted
to illustrate PromptAV’s effectiveness and explainability for the AV task. For
future works, we will conduct experiments on more AV datasets and explore
other linguistic features.

##  6 Limitations

This study, while substantive, acknowledges certain limitations. A significant
constraint is the absence of a ground truth for the generated explanation,
which renders the evaluation of the efficacy of PromptAV’s generated
explanation somewhat challenging. This suggests a potential avenue for tapping
into the expertise of forensic linguistic experts to assess and curate some of
the reasoning generated by PromptAV. Additionally, we have observed instances
where PromptAV generates illusory explanations by identifying the usage of
specific vocabulary within the text, even when such vocabulary may not have
been actually used.

## References

  * Bagnall (2015) Douglas Bagnall. 2015.  Author identification using multi-headed recurrent neural networks.  _arXiv preprint arXiv:1506.04891_.
  * Benzebouchi et al. (2018) Nacer Eddine Benzebouchi, Nabiha Azizi, Monther Aldwairi, and Nadir Farah. 2018\.  Multi-classifier system for authorship verification task using word embeddings.  In _2018 2nd International Conference on Natural Language and Speech Processing (ICNLSP)_ , pages 1–6. IEEE.
  * Boenninghoff et al. (2019) Benedikt Boenninghoff, Steffen Hessler, Dorothea Kolossa, and Robert M Nickel. 2019\.  Explainable authorship verification in social media via attention-based similarity learning.  In _2019 IEEE International Conference on Big Data (Big Data)_ , pages 36–45. IEEE.
  * Brocardo et al. (2013) Marcelo Luiz Brocardo, Issa Traore, Sherif Saad, and Isaac Woungang. 2013.  Authorship verification for short messages using stylometry.  In _2013 International Conference on Computer, Information and Telecommunication Systems (CITS)_ , pages 1–6. IEEE.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.
  * Claxton (2005) Larry D Claxton. 2005.  Scientific authorship: Part 1. a window into scientific fraud?  _Mutation Research/Reviews in Mutation Research_ , 589(1):17–30.
  * Ding et al. (2017) Steven HH Ding, Benjamin CM Fung, Farkhund Iqbal, and William K Cheung. 2017.  Learning stylometric representations for authorship analysis.  _IEEE transactions on cybernetics_ , 49(1):107–121.
  * Fabien et al. (2020) Maël Fabien, Esau Villatoro-Tello, Petr Motlicek, and Shantipriya Parida. 2020\.  BertAA : BERT fine-tuning for authorship attribution.  In _Proceedings of the 17th International Conference on Natural Language Processing (ICON)_ , pages 127–137, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NLPAI).
  * Hu et al. (2023) Xinyu Hu, Weihan Ou, Sudipta Acharya, Steven H Ding, Ryan D’Gama, and Hanbo Yu. 2023.  Tdrlm: Stylometric learning for authorship verification by topic-debiasing.  _Expert Systems with Applications_ , page 120745.
  * Iqbal et al. (2010) Farkhund Iqbal, Liaquat A Khan, Benjamin CM Fung, and Mourad Debbabi. 2010.  E-mail authorship verification for forensic investigation.  In _Proceedings of the 2010 ACM Symposium on Applied computing_ , pages 1591–1598.
  * Kojima et al. (2023) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023.  Large language models are zero-shot reasoners.
  * Lagutina et al. (2019) Ksenia Lagutina, Nadezhda Lagutina, Elena Boychuk, Inna Vorontsova, Elena Shliakhtina, Olga Belyaeva, Ilya Paramonov, and PG Demidov. 2019.  A survey on stylometric text features.  In _2019 25th Conference of Open Innovations Association (FRUCT)_ , pages 184–195. IEEE.
  * Mahmood et al. (2020) Asad Mahmood, Zubair Shafiq, and Padmini Srinivasan. 2020.  A girl has a name: Detecting authorship obfuscation.
  * Najafi and Tavan (2022) Maryam Najafi and Ehsan Tavan. 2022.  Text-to-text transformer in authorship verification via stylistic and semantical analysis.  In _Proceedings of the CLEF_.
  * Ordoñez et al. (2020) Juanita Ordoñez, Rafael Rivera Soto, and Barry Y Chen. 2020.  Will longformers pan out for authorship verification.  _Working Notes of CLEF_.
  * Ruder et al. (2016) Sebastian Ruder, Parsa Ghaffari, and John G Breslin. 2016.  Character-level and multi-channel convolutional neural networks for large-scale authorship attribution.  _arXiv preprint arXiv:1609.06686_.
  * Seroussi et al. (2014) Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. 2014.  Authorship attribution with topic models.  _Computational Linguistics_ , 40(2):269–310.
  * Stamatatos (2016) Efstathios Stamatatos. 2016.  Authorship verification: a review of recent advances.  _Research in Computing Science_ , 123:9–25.
  * Stein et al. (2011) Benno Stein, Nedim Lipka, and Peter Prettenhofer. 2011.  Intrinsic plagiarism analysis.  _Language Resources and Evaluation_ , 45:63–82.
  * Tyo et al. (2021) Jacob Tyo, Bhuwan Dhingra, and Zachary C Lipton. 2021.  Siamese bert for authorship verification.  In _CLEF (Working Notes)_ , pages 2169–2177.
  * Uchendu et al. (2023) Adaku Uchendu, Thai Le, and Dongwon Lee. 2023.  Attribution and obfuscation of neural text authorship: A data mining perspective.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2023) Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023.  Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-thought prompting elicits reasoning in large language models.
  * Zhao et al. (2021) Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.

##  Appendix A Appendix

###  A.1 Computation and Time comsumption of PromptAV

In our experiments, we applied PromptAV using GPT-3.5 via the OpenAI API.
Specifically, evaluating 1,000 examples took approximately 8 hours. The
expense associated with using PromptAV via the OpenAI API is indeed calculated
at a rate of $0.004 per 1,000 tokens. For a dataset of 1,000 samples, the
estimated cost came to around $10. This cost, we believe, is justifiable given
the unique interpretability insights PromptAV provides, especially in critical
applications like forensic analysis where the implications of the results are
profound. It is possible to batch-process texts through sending multiple
requests using multiple API keys. With adequate budget, we could scale-up the
AV operations.

###  A.2 Stylometric Feature Impact

Number of features | Accuracy
---|---
8 | 0.587
9 | 0.602
10 | 0.585
11 | 0.564
12 | 0.572
Table 4: Performance of PromptAV in 0 shot setting with varying sets of
stylometric features.

###  A.3 Optimal Threshold Methodology

The method of reporting using an optimal threshold was our initial approach to
compare the results of different prompting methods. The intention was to
establish a baseline comparison across varying methods in a controlled
environment. In real-world applications, the threshold would likely require
adjustment based on specific use-cases and available data.

Practical Implications of Threshold: It is indeed a challenge to select an
optimal threshold that would be universally valid. In our experiment applying
PromptAV on the IMDb62 dataset, we found that the optimal threshold lies
between 0.2 - 0.3. This range suggests that the model tends to generate a low
confidence score, which provides significant insights into its functioning and
decision-making. The low confidence score may be indicative of the model’s
cautious approach to attributing authorship, ensuring a reduced number of
false positives in practical scenarios. This threshold may varied, depending
on the complexity of the dataset, and the amount of writing sample
observations the model made for each author.

Selection of the Confidence Score: The confidence score selection was an
empirical decision based on our preliminary experiments, intending to optimize
the balance between false positives and false negatives. We have conducted an
experiments to show the tradeoff on performance when selecting different
threshold for the confident score. Note that experiments are conducted using
PromptAV with zero-shot demonstration setting.

Confidence Score | PromptAV | PS+ | CoT
---|---|---|---
0.1 | 0.506 | 0.501 | 0.524
0.2 | 0.587 | 0.536 | 0.522
0.3 | 0.579 | 0.529 | 0.505
0.4 | 0.578 | 0.529 | 0.497
0.5 | 0.569 | 0.527 | 0.497
0.6 | 0.55 | 0.484 | 0.498
0.7 | 0.53 | 0.493 | 0.498
0.8 | 0.519 | 0.497 | 0.497
0.9 | 0.5 | 0.497 | 0.497
Table 5: Performance of PromptAV and the baselines in 0 shot setting with
varying confidence score threshold. Highest accuracy are bolded

###  A.4 Zero-shot vs supervised methods

Our work with PromptAV centers around its capability to handle the AV task in
a zero-shot setting. Acquiring labeled data, especially for tasks like AV, can
be quite challenging due to privacy concerns, the need for expert annotators,
and other complexities associated with the domain. We recognize the importance
and relevance of benchmarking against state-of-the-art AV methods. Their
dependence on copious amounts of labeled data inherently places them in a
different operating paradigm. For instance, BERT is able to achieve a high
accuracy of 0.702 if it is trained on IMDb62 full training dataset with 74,400
samples. However, BERT will require large amount of training data to achieve
the superior performance.


# Local Planner Bench: Benchmarking for Local Motion Planning

Max Spahn∗, Chadi Salmi∗, Javier Alonso-Mora∗ *This research was supported by
Ahold Delhaize and ERC. All content represents the opinion of the authors,
which is not necessarily shared or endorsed by their respective employers
and/or sponsors.The authors are with the Department of Cognitive Robotics,
Delft University of Technology, 2628 CD, Delft, The Netherlands {m.spahn,
<EMAIL_ADDRESS>
###### Abstract

Local motion planning is a heavily researched topic in the field of robotics
with many promising algorithms being published every year. However, it is
difficult and time-consuming to compare different methods in the field. In
this paper, we present localPlannerBench, a new benchmarking suite that allows
quick and seamless comparison between local motion planning algorithms. The
key focus of the project lies in the extensibility of the environment and the
simulation cases. Out-of-the-box, localPlannerBench already supports many
simulation cases ranging from a simple 2D point mass to full-fledged 3D 7DoF
manipulators, and it’s straightforward to add your own custom robot using a
URDF file. A post-processor is built-in that can be extended with custom
metrics and plots. To integrate your own motion planner, simply create a
wrapper that derives from the provided base class. Ultimately we aim to
improve the reproducibility of local motion planning algorithms and encourage
standardized open-source comparison.

Local Motion Planning, Benchmark, Collision Avoidance
Code: github.com/tud-amr/localPlannerBench 111version 1.0.0:
https://github.com/tud-amr/localPlannerBench/tree/v.1.0.0

##  I INTRODUCTION

Robot motion planning in dynamic environments is fundamentally different from
motion generation in static environments because initial plans must be
constantly updated in the presence of unforeseen events. Algorithms that aim
to adapt global plans according to dynamic changes in the environment and
transfer them into actions at runtime fall into the category of local motion
planning or reactive motion planning. As the number of applications (e.g.
mobile robots, robotic arms, mobile manipulators) is quite diverse, different
methods have been presented in the last years showing varying performance
depending on the scenarios. These methods can be roughly divided into four
categories: (a) geometric approaches, such as the potential field method [1],
reciprocal collision avoidance with velocity obstacles [2], Riemannian Motion
Policies [3] or optimization fabrics [4, 5], (b) optimization-based
approaches, such as STOMP [6] or model predicitive control [7, 8], (c)
compositon of motion primitives [9, 10], and (d) learning-based approaches
[11, 12]. Due to a steady increase in number of methods and a tendency to
focus on one of the above approaches, it becomes increasingly difficult and
time-consuming to make proper comparisons among methods. Additionally, new
methods in the field are often not publicly available so comparisons require
re-implementation of these methods. Together with a general bias toward
optimizing parameters of a proposed method, fair comparisons become more
challenging. To address these issues, we offer localPlannerBench, a bench-
marking tool to allow quick and seamless comparison between different local
motion planning algorithms. Additionally, we promote extendibility to allow
users to modify localPlannerBench according to their setups.

First, we review briefly other benchmarking software with a similar purpose
while laying out the differences to our framework (Section II). Then, we state
the scope of our framework and introduce the approach. (Section III). We lay
out the general structure of localPlannerBench and explain individual
components in Section III-E. Lastly, we briefly present a use-case where two
different planners are compared (Section IV).

##  II RELATED WORK

As we propose localPlannerBench for benchmarking local motion planning
algorithms, we briefly revise existing benchmarks in robotics to highlight
differences. Sampling-based motion planning for global motion planning was
first standardized in the Open Motion Planning Library (OMPL) [13]. There,
many algorithms are implemented and accessible to the user. Today, it is used
as the backend planning library for many robotics software [14, 15]. While the
benefit of assembling different algorithms in a single library is, still
today, highlighted by OMPL, it is not a benchmark suite on its own. Robowflex
is a wrapper around various motion planning libraries that can evaluate
different global motion planning algorithms in isolation [16]. It also
includes standardized motion planning problems to promote comparisons. The
goal behind localPlannerBench is to provide a similar framework but instead of
focusing on global motion planning methods, which are most valuable in static
environments, we focus on dynamic environments and thus on local motion
planning methods. Some benchmarking frameworks focus purely on mobile robots
and/or autonomous driving and provide benchmarks for local motion planning
methods in cluttered and human-shared environments [17, 18]. With
localPlannerBench, we generalize this approach beyond mobile robots by
including robotic arms and mobile manipulators. To make motion planning
methods more reproducible and comparable, all of the mentioned works play a
central role and localPlannerBench takes this role for local motion planning.

(a)

(b)

(c)

Figure 1: Examples of environments (pybullet in Figs. 1(a) and 1(b) and simple
ODE in Fig. 1(c)) available in localPlannerBench. Goal states are depicted as
green circles, and obstacles as either red or black depending on the
environment.

##  III APPROACH

###  III-A Scope

We define local motion planning as the problem of computing actions for the
robot at runtime, such that the sequence of actions leads to progress in the
fulfillment of the goal while avoiding collisions with its environment. From
this definition, it follows that interactive motion planning, where the robot
is allowed to manipulate objects, is not considered. Moreover, we consider
collision avoidance the only safe way of navigating through the environment.
While the initial state can be specified by a robot configuration, the goal is
a composition of geometric constraints. Some examples that can be stated as
geometric constraints are, that the end-effector has to be at a defined
position and/or orientation, some joints need to be in a specific position,
some links must align or the robot has to follow a certain trajectory.
Although not supported in this version, the latter could be generated with
global planner. In other words, the meaningful behavior of the robot in the
environment is assumed to be parameterizable by these geometric goals.

Furthermore, perception pipelines are outside the scope of this work. At this
stage, we assume a working perception pipeline that can detect all obstacles
as they move through the environment. In the future, we want to integrate more
abstract perception pipelines, e.g. generating occupancy grids based on the
obstacles positions and shapes. Low-level control of the joints of the robot
is also out of the scope of this work. That means, we provide environments
with different control modalities, but the control signal is directly passed
to the environment, so a perfect low-level controller is assumed.

###  III-B Definitions

An environment refers to the physics simulator (or real world) that produces
robot states and obstacle poses and applies actions to the robot. The problem
formulation is defined by the geometric parameterization of the goal, the
robot, the specifications of the obstacles and additional global environment
variables. A planner is a local motion planning method that computes an action
based on the current state of the environment. We define a metric as a
quantitative measure for the performance evaluation of a planner. A single run
of the environment with corresponding problem formulation and planner is
referred to as a trial. A study, within localPlannerBench, refers to a series
of trials.

###  III-C Rational

Our guidelines for developing and maintaining localPlannerBench are the
following:

  * •

Simplicity It should be as simple as possible to run a trial/study.

  * •

Modularity It should be straightforward to add a planner, metric or problem
formulation.

  * •

Specificity It should focus on local motion planning and local motion planning
only.

In the following, we derive design choices based on those motivations.

###  III-D Environment

The environment in localPlannerBench is not specific to a single physics
simulator and should even be compatible with the real world through e.g. a ROS
bridge (modularity). However, at the time of writing, two physics simulators
are officially supported (see Fig. 1), planar environments are based on a
simple ODE-simulation and pybullet [19] is used for more complex 3D
environments. In the pybullet environments we use the unified robot
description format (URDF) that allows to import new robots simply by this
robot description. To achieve a clean separation between the simulation
environment and planning algorithm, we opted to design an Open-AI gym [20]
environment wrapper for the different robot types and control modalities that
we support (Simplicity, Modularity). Pybullet is not considered the most
accurate physics engine, but it is much easier to install compared to
competing simulators like ISAAC-Gym [21] or MuJoCo [22] (Simplicity). Besides,
as low-level control and interactive motion planning are outside of our scope,
the accuracy of PyBullet is considered sufficient (Specificity).

Figure 2: Code structure of localPlannerBench. Blue components are provided,
while the orange components should be passed along at run-time. A specific
planner (green) could be implemented by the user, and the outputs are shown as
yellow blocks.

###  III-E Code structure

The code in localPlannerBench is split up into components as can be seen in
Fig. 2. Each component has a unique and clearly defined responsibility
(Modularity). The parameters specifying specific configurations of these
components are as much as possible defined outside of the code itself using
yaml-files. When possible, the code structure reflects the definitions
proposed in Section III-B:

  * •

The Problem class corresponds to the problem formulation and requires its own
configuration file to read in the requested problem formulation. Additionally,
the component supports randomization of the initial configuration, goal state,
and obstacle states. The limits of the randomization should also be defined
within the configuration file.

  * •

The AbstractPlanner is a blueprint for a planner class. When a user creates a
new planner, she should derive from this provided base class, so that the
planner will automatically be registered as a valid planner to the rest of the
package. A planner created using this component is parameterized by the
corresponding configuration file.

  * •

The runner executable is the entry point when running a trial or study i.e. it
is the executable that the user interacts with. The component contains the
code for the actual execution loops over the Open-AI gym simulation
environments and saves the results afterward.

  * •

The post-processor executable processes the results based on user specified
metrics and plot characteristics. Like the runner, it is an executable that
the user interacts with.

###  III-F Usage

There are currently two general possible usages of localPlannerBench: (a)
access an open implementation of a local motion planner and its tuned
parameters by an expert or (b) compare your own local motion planner against
baselines that are already implemented in localPlannerBench

###  III-G Running a study

localPlannerBench provides a minimal command-line interface (CLI) that
processes the configuration files described in the previous section. The CLI
calls the runner that runs a study with the given parameters. Specifically,
the user specifies the location of the configurations files for the problem
formulation and the planner. Additionally, flags and arguments for the study
can be speficied through the CLI. For example, the user could specify the
location of the output files, whether the simulation should be rendered or
not, how many trials should be run, etc. Finally, all relevant information to
reproduce a study is saved in a time-stamped results folder. An overview of
arguments can be found in Table I.

Argument |  | Explanation
---|---|---
caseSetup | : | Path to problem formulation file.
planners | : | List of paths to planner configuration files.
res-folder | : | Path to store the result data.
render | : | Display GUI.
numberRuns | : | How many trials to run.
random-goal | : | Should the goal be random?
random-init | : | Should the initial configuration be random?
random-obst | : | Should the obstacles be random?
TABLE I: List of CLI arguments for the runner executable

###  III-H Postprocessing a study

To promote better reproducibility, running, and post-processing a study is
completely separated. Specifically, the output of a study is the raw data that
was directly accessible during the execution. That is: the joint position, the
joint velocities, the taken actions, the positions of the links of the robot,
the position of obstacles, and the goal at each time step. During post-
processing, the raw data is evaluated according to user metrics and is
potentially plotted. Several performance metrics are implemented, see the
repository for a full list. However, additional metrics could be added using a
similar approach as for the planner. An overview of post-processor arguments
can be found in Table II.

Argument |  | Explanation
---|---|---
expFolder | : | Path to the results of a study.
kpis | : | List of all kpis/metrics to be evaluated.
plot | : | Auto-generate plot from results.
series | : | Are you evaluating a single trial or multiple?
compare | : | Are you comparing planners?
TABLE II: CLI arguments for the post_processor executable

##  IV EXAMPLE: Optimization fabrics vs MPC

An example of how localPlannerBench can be used to compare the two methods,
Static Fabrics and Model Predictive Control (MPC), is found in [4]. The goal
of the comparison was to investigate if the significant reduction in solver
time of Static Fabrics impacts other metrics like path length, clearance or
time to goal. Using the runner, a study was run for a hundred trials,
including both planners. Afterwards, the plots in Figure 3 were generated
using the post-processor.

(a)

(b)

Figure 3:  Exemplary comparison between two different local motion planning
algorithms in localPlannerBench.

##  V CONCLUSION

As the field of local motion planning matures, the need for standardized
reproducible benchmarks increases. To make the life of a researcher a little
easier, we’ve created localPlannerBench. This framework is capable of
simulating local motion planning problems, for many different robots,
modalities, and environments. Furthermore, localPlannerBench is fully open
source and designed with simplicity, modularity, and specificacy in mind,
enabling users to easily modify localPlannerBench to fit their unique use
cases as long as they fit within the scope as what we define as local motion
planning. In the future, we hope to extend localPlannerBench in several ways:
integrating basic global planning methods to provide reference trajectories
for local planning, adding artificial uncertainty to the states and
observations to help make the studies more representative of real scenarios,
adding support for observations in the form of occupancy maps, semantic maps,
or even direct camera or lidar data to accommodate new directions in local
motion planning. localPlannerBench has already been proven to be useful for
academic comparison and evaluations [4], and we hope it will gain futher
attention in this field of local motion planning.

## References

  * [1] K. B. Kaldestad, S. Haddadin _et al._ , “Collision avoidance with potential fields based on parallel processing of 3d-point cloud data on the gpu,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2014, pp. 3250–3257.
  * [2] J. Van Den Berg, J. Snape _et al._ , “Reciprocal collision avoidance with acceleration-velocity obstacles,” in _2011 IEEE International Conference on Robotics and Automation_. IEEE, 2011, pp. 3475–3482.
  * [3] N. D. Ratliff, J. Issac _et al._ , “Riemannian motion policies,” _arXiv preprint arXiv:1801.02854_ , 2018.
  * [4] N. D. Ratliff, K. Van Wyk _et al._ , “Optimization fabrics,” _arXiv preprint arXiv:2008.02399_ , 2020.
  * [5] M. Spahn, M. Wisse, and J. Alonso-Mora, “Dynamic optimization fabrics for motion generation,” _arXiv preprint arXiv:2205.08454_ , 2022.
  * [6] M. Kalakrishnan, S. Chitta _et al._ , “Stomp: Stochastic trajectory optimization for motion planning,” in _2011 IEEE international conference on robotics and automation_. IEEE, 2011, pp. 4569–4574.
  * [7] L. Hewing, K. P. Wabersich _et al._ , “Learning-based model predictive control: Toward safe learning in control,” _Annual Review of Control, Robotics, and Autonomous Systems_ , vol. 3, pp. 269–296, 2020.
  * [8] B. Brito, M. Everett _et al._ , “Where to go next: learning a subgoal recommendation policy for navigation in dynamic environments,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4616–4623, 2021.
  * [9] M. Dharmadhikari, T. Dang _et al._ , “Motion primitives-based path planning for fast and agile exploration using aerial robots,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 179–185.
  * [10] K. Hauser, T. Bretl _et al._ , “Using motion primitives in probabilistic sample-based planning for humanoid robots,” in _Algorithmic foundation of robotics VII_. Springer, 2008, pp. 507–522.
  * [11] C. Wang, Q. Zhang _et al._ , “Learning Mobile Manipulation through Deep Reinforcement Learning,” _Sensors_ , vol. 20, no. 3, p. 939, Feb. 2020. [Online]. Available: https://www.mdpi.com/1424-8220/20/3/939
  * [12] T. Jurgenson and A. Tamar, “Harnessing Reinforcement Learning for Neural Motion Planning,” in _Robotics: Science and Systems XV_. Robotics: Science and Systems Foundation, Jun. 2019. [Online]. Available: http://www.roboticsproceedings.org/rss15/p26.pdf
  * [13] I. A. Șucan, M. Moll, and L. E. Kavraki, “The Open Motion Planning Library,” _IEEE Robotics and Automation Magazine_ , vol. 19, no. 4, pp. 72–82, 12 2012.
  * [14] D. Coleman, I. Sucan _et al._ , “Reducing the barrier to entry of complex robotic software: a moveit! case study,” _arXiv preprint arXiv:1404.3785_ , 2014.
  * [15] R. Diankov and J. J. Kuffner, “Openrave: A planning architecture for autonomous robotics,” 2008.
  * [16] C. Chamzas, C. Quintero-Pena _et al._ , “Motionbenchmaker: A tool to generate and benchmark motion planning datasets,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 882–889, 2021.
  * [17] L. Kästner, T. Bhuiyan _et al._ , “Arena-bench: A benchmarking suite for obstacle avoidance approaches in highly dynamic environments,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 4, pp. 9477–9484, 2022\.
  * [18] E. Heiden, L. Palmieri _et al._ , “Bench-mr: A motion planning benchmark for wheeled mobile robots,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4536–4543, 2021.
  * [19] E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016–2021.
  * [20] G. Brockman, V. Cheung _et al._ , “Openai gym,” 2016.
  * [21] V. Makoviychuk, L. Wawrzyniak _et al._ , “Isaac gym: High performance gpu-based physics simulation for robot learning,” 2021. [Online]. Available: https://arxiv.org/abs/2108.10470
  * [22] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2012, pp. 5026–5033.


# Watching Smartly from the Bottom: Intrusion Detection revamped through
Programmable Networks and Artificial Intelligence

Sergio Armando Gutiérrez, John Willian Branch, Luciano Paschoal Gaspary, Juan
Felipe Botero S.A. Gutiérrez is Researcher and Lecturer at UNAULA, Medellín,
Colombia.
E-mail<EMAIL_ADDRESS>J.W. Branch is Full Professor at
UNAL, Medellín, Colombia.
E-mail<EMAIL_ADDRESS>L. Gaspary is Associate Professor at UFRGS, Porto
Alegre, Brazil.
E-mail<EMAIL_ADDRESS>J.F.Botero is Associate Professor at UDEA, Medellín,
Colombia.
E-mail<EMAIL_ADDRESS>Manuscript received October 1, 2020; revised
October 1, 2020.

###### Abstract

A recent research line has explored the possibility of leveraging
functionalities of Programmable Data Planes to offload part of Machine
Learning algorithms to the data plane, which might contribute to increase
their accuracy and responsiveness by having a more detailed visibility of the
traffic. This approach introduces a significant opportunity for evolution in
the critical field of Intrusion Detection. In this paper, we discuss how
Programmable Data Planes might complement different stages of an Intrusion
Detection System based on Machine Learning. We present two use cases that make
evident the feasibility of this approach and highlight aspects that must be
considered when addressing the non straightforward task of deploying solutions
leveraging data-plane functionalities.

##  I Introduction

The emergence of the Programmable Data Planes (PDPs) concept has represented
an outstanding advance in the complete realization of the Software Defined
Networking (SDN) Paradigm. The capacity of influencing Forwarding Device (FD)
behavior via abstractions such as programming languages is an idea that has
been explored through different initiatives proposed by industry and academia.
An important milestone in the evolution of network programmability was the
development of OpenFlow as an implementation of SDN. However in traditional
SDN, the network operation is still constrained to the set of protocols and
headers supported by the hardware of the FDs. Therefore, the definition of
custom functions for packet processing becomes very difficult.

Recently, the concept of PDPs has emerged to overcome these constraints by
allowing going beyond the definition of flow rules. PDPs enable a complete
control of the network behavior, from the applications to the packet
processing within the FDs, including the definition and parsing/deparsing of
custom headers. It provides an unprecedented opportunity to develop new
features within the FDs and revisiting existing functions for network
management [1]. P4 is currently the de-facto standard language for describing
how network packets should be processed.

One of the areas that can take advantage of PDP/P4 is network security.
Particularly, Intrusion Detection Systems (IDSs) can be revamped by
implementing them as efficient functions deployed at the data plane, capable
to react promptly to network anomalies that might represent threats. IDSs
typically depend on the collection of traffic features which are later fed
into sophisticated systems mostly based on Machine Learning (ML) algorithms.
ML has been successfully used in network security due to its capacity to
detect and discover patterns and behaviors previously unseen in the network
traffic. Most of the approaches for security developed in the context of SDN
and based on ML have been implemented at the control plane despite the issues
associated to accuracy and high overhead they might introduce [2].

The features introduced by PDPs make possible to consider a new landscape for
security solutions based on ML, by conceiving offloading part of the
algorithms to the FDs. Thus, more accurate and responsive solutions can be
deployed. Delegating part of the functions to the data plane might also help
to reduce the overhead at the control plane. Nevertheless, the decision
regarding how much of the functions should be offloaded to the data plane is
not trivial [3] because the computation capabilities of FDs are limited, and
an excessive offloading of functionalities might hinder the maximum throughput
in packet forwarding.

In this paper, our main contribution is presenting a critical discussion on
the challenges introduced when intersecting Programmable Data Planes and
Artificial Intelligence (AI) for intrusion detection. We focus on devising how
to leverage the functionalities offered by PDPs in the implementation of AI
and specially ML algorithms. We specially try to address the question
regarding how much of the operations of the algorithms is feasible to be
offloaded to the FDs.

##  II Machine Learning at the Programmable Data Plane for Improved Intrusion
Detection

Figure 1: Stages of a ML system applied to Network Management domains (e.g.
Intrusion Detection) [4]

ML has become an important milestone in several kinds of cyber security
solutions due to its ability to extract, in a timely fashion, anomalies and
patterns that might be symptom of internal or external attacks against the
infrastructure. These solutions are usually integrated by network and host
security components, minimally including firewalls, anti viruses and Intrusion
Detection Systems (IDSs) [4].

IDSs are being actively revisited in the context of programmable networks.
Most of the solutions developed for implementing IDSs have been deployed as
applications running in the control plane. However, this approach for the
implementation of IDSs has two main drawbacks. First, the set of traffic
features derived either from standard counters (e.g., those available in
current versions of OpenFlow) or, in extreme situations, via the PACKET_IN
events, are insufficient to get reasonable accuracy in ML algorithms. Second,
ML algorithms are typically compute-intensive tasks, which, if not properly
designed/deployed, might introduce control plane overhead and disrupt the
correct operation of the network [5].

As we previously introduced, the advent of PDP makes possible the offloading
of some functionalities to the data plane. This is an strategy that might help
to overcome the limitations associated to scarce information as input while
reducing the overhead of the control plane. Next, we present an overview of
the stages of an ML-based IDS, and we outline how PDPs can be leveraged to
improve some of these stages through custom packet processing and the
offloading of certain operations [4]. Figure 1 presents the sequential view
and relationships of these typical stages.

Data Collection. PDPs extend the possibilities for data collection beyond the
standard statistics available in the FDs. Custom statistics can be introduced,
and some stateful processing can be included within FDs, which might even
indicate events with high value for intrusion detection tasks [6]. Despite the
limitations inherent to the computing power and programming primitives
available in programmable FDs, there are two functionalities that can be
leveraged to implement efficient data collection for ML algorithms: Custom
packet parsing and data aggregation. Custom Packet parsing allows processing
headers that can be used to store personalized information and statistics.
These headers enable differentiated processing and information gathering
within FDs, or applying custom hash functions in order to distinguish
specifics sets of packets [7]. On the other hand, data aggregation helps to
reduce the amount of data that needs to be transmitted in the network from FDs
to Control Plane, for the execution of complex operations there [8].

Data Processing. Recent proposals introduce the concept of In-Network
processing as a service, where the implementation of the mentioned operations
in data plane becomes available to be used by high level ML or data-analytics
frameworks [9, 10]. Other proposals introduce the notion of queries that
trigger the joint collection and preliminary analysis of data in order to
produce statistics that can be later delivered to higher level processing
mechanisms [7]. The main notion behind these approaches is the fact that
features of PDPs such as programmable parsing, state storing, custom hashing
and the flexibility of match/action tables allow not only measurements and
counts, but also performing analysis in parallel to the data collection.

Model Construction. The literature presents several approaches to leverage the
features available in PDP for the implementation of different algorithms
considering the computational constraints of programmable FDs [3]. These
proposals include the use of switch registers to implement bit-based value
storage and arithmetic which might be even useful to implement Neural Networks
[11] and match action and long prefix match tables with comparison operators
for the implementation of algorithms such as decision trees, support vector
machines and Naive Bayes classifiers. This approach reduces the amount
information that needs to be forwarded towards control plane (e.g. PACKET_IN
events) which contributes to diminish the control channel overhead [12] while
increases the accuracy and responsiveness [13, 3]. In addition to the direct
implementation within the FDs, another approach to be followed is cooperating
in the training of large scale models via local analysis of metrics. This
approach is called Federated Learning, and can be leveraged for the training
of complex models such as Deep Neural Networks [11].

Model Validation. Validation is a high level task which involves extensive
analysis and feedback provided by human experts. Hence, PDPs do not have a
direct intervention in the tasks associated to this stage. However,
functionalities such as Complex Event Processing [6] and query-driven
telemetry [7] are helpful to provide insights to debug situations of low
accuracy and poor performance of the ML algorithms.

Deployment. This operation must consider the particularities involved in the
development of the algorithms. For example, the availability in a given
hardware architecture of the type of tables required, or the amount of
registers that may be used to store packet state are aspects that must be
validated [13, 11]. For a detailed discussion on the issues associated to the
deployment of ML algorithms within FDs, please refer to [13] and [3].

Result Analysis. Functionalities such as In-Band Network Telemetry, which
relies on PDP features [7], and Complex Event Processing [6] might provide
important insights for this stage. Also, the definition of thresholds for
specific features and time intervals [14], and even the support for query-
driven information collection might contribute to assess the effectiveness of
the algorithms while allowing some degree of data analysis.

Table I summarizes our insights regarding where to implement the particular
functions, either in the control or the data plane and the Machine Learning
Algorithm(s) that might be involved.

TABLE I: Place of deployment for functionalities associated to an ML-Based IDS Stage | Functionality | Where to implement | ML algorithm | Related Work
---|---|---|---|---
Data Collection | Custom statistics and metrics | Data Plane | CUSUM | [6]
Data Collection | Header hashing | Data Plane | Sketches | [7]
Data Collection | Custom packet parsing | Data Plane | KNN | [9, 10]
Data Processing | Partial state storing | Data Plane | Sketches | [13]
Data Processing | Threshold comparison | Data Plane | CUSUM,SVM | [14]
Data Processing | In-network processing as a Service | Control Plane | SVM,KNN,PCA | [9]
Model Construction | Complex Arithmetic | Control Plane | NN,SVM,RNN | [8, 15, 11]
Model Validation | Complex Event Processing | Control Plane | RNN | [6]
Model Validation | Query-Driven Telemetry | Control Plane | Sketches,RNN | [7]

The processing capabilities available nowadays in programmable devices are
mainly useful for data collection and data processing. Currently, most of the
functionalities associated to ML algorithms such as Cumulative Summation Chart
Control (CUSUM), Support Vector Machines (SVM) K-Nearest Neighbors (KNN),
Principal Component Analysis (PCA), Basic Neural Networks (NN) or Recurrent
Neural Networks (RNN) tend to be implemented within the control plane.
However, the implementation of ML algorithms within the forwarding devices can
be expected in a near future, by taking advantage of more robust hardware
platforms for these devices.

##  III Use Cases

In the previous section, we presented an overview about how PDPs can
contribute in the implementation of some of the stages of a system based on ML
algorithms applied to the domain of IDS. In this section, we present two use
cases which exemplify how to specifically leverage functionalities of PDPs in
the context of security solutions based on these ML algorithms.

###  III-A Offloading Data Collection and Processing in DDoS attack Detection
to Programmable Data Planes

DDoS attack detection in traditional SDN environments introduces two design
challenges: 1) the control plane is overloaded due to the continuous polling
of network traffic information as well as the calculation and classification
of the feature set; and 2) OpenFlow cannot obtain specific features
fundamental for DDoS detection (e.g. per-flow packet inter-arrival times and
packet payload sizes). These two challenges can be overcome by offloading some
of the ML-based detection stages from the control plane to P4-based PDP.

Fig. 2 shows a DDoS detection network architecture where some ML stages are
offloaded from the control plane to the programmable data plane that
implements a time window-based scheme. During each window, it simultaneously:
i) collects new flow information; i.e. it performs the Data Collection stage
mentioned in Section II by extracting specific per-flow information needed to
build the feature set for the attack detection, and ii) reports the
information collected in the previous window to the control plane; i.e. it
carries out the Data Processing stage by pre-processing the needed per-flow
features and forwarding them to the control plane that, in turn, easily
generates the feature set to feed the ML-based detection algorithm.

When a new packet arrives to the FD, the statistics collector extracts the
packet information building specific flow descriptors using just the
arithmetic operations available in P4. Then, these descriptors are sent to the
control plane for the generation of the feature set (detailed information of
the descriptors is provided in [12]). Once the time window expires, the
pipeline handler checks if the statistics collector contains new flow
information to report and, if this is the case, the information is forwarded
to the information reporter which generates a report packet with the specific
descriptors of the flows that changed during the time window. Finally, the
report packet is forwarded to the control plane using the P4Runtime interface.

Figure 2: ML-based DDoS attack detection architecture coordinating control and
programmable data planes [12]

The control plane is composed of an SDN controller connected to a ML-based
traffic classification application. The feature set composition component
parses the custom header of the packet that arrived from the data plane and
builds a tuple with the resulting per-flow feature set. Finally, this set is
received by the classifier application that executes an already loaded ML
model which performs binary classification (i.e. determines if each received
tuple belongs either to DDoS class or benign class).

In this approach, we implemented two ML classifiers for different time window
duration: Random Forest (RF) and K-Nearest Neighbor (KNN). The experimental
evaluation consisted on reproducing a DDoS attack contained on a state-of-the-
art dataset. The experimental environment was built on mininet, using the
reference P4 Switch implementation (BMV2) [1]. The comparison of the two
classifiers used showed that KNN achieved better performance than RF. Obtained
results led to an accuracy from 93% to 96% for the different time windows,
which is within the upper range seen in the literature.

Besides reaching high accuracy, the offloading of the data collection and
processing stages to the PDP provides two main advantages. 1) A configurable
time window forwarding just one report packet to the control plane removes the
need of duplicating packets and send them to external IDS. It also prevents
the continuous transfer of OpenFlow statistics from the FD to the control
plane, therefore reducing traffic overload in the control channel. 2) Data
plane programmability allows to extract just the feature descriptors needed by
the control plane to build the feature set in order to perform the ML-based
DDoS attack detection. In contrast, traditional SDN-OpenFlow collects
statistics that are useless to detect DDoS attacks. Besides, OpenFlow is
unable to collect some features that are paramount to classify DDoS attacks
with ML-based algorithms (please see [12] for further details).

###  III-B Offloading Real-Time DDoS Attack Detection to Programmable Data
Planes

There are situations (e.g., in service provider core networks) in which the
fast, real-time detection of an intrusion is a primary requirement, even if
sacrificing (a little of the) accuracy. Next, we briefly present a mechanism
in which we devise a lightweight entropy-based anomaly detection technique and
offload it entirely to a programmable forwarding device based on P4 [14].

Figure 3 illustrates the main architectural components of the mechanism. It
implements a processing pipeline to calculate the entropies of source and
destination IP addresses of packets arriving at an FD. They are estimated for
consecutive packets of the incoming stream, named observation windows. Upon
completing each window, the traffic characterization components (for source
and destination IPs) consume the entropy values to produce a legitimate
traffic model. The anomaly detection component calculates detection thresholds
as functions of this model, issuing an attack alarm when the last entropy
estimates exceed them.

Figure 3: In-network mechanism for real-time DDoS attack detection [14]

Given the restrictions of the primitive set of operations provided by P4, a
simplified design was compulsory in several circumstances. For example, we
employed customized count sketches to approximate the frequencies of different
IP addresses. To solve compute-intensive arithmetic functions, we used
longest-prefix match (LPM) lookup tables. The lack of procedures to iterate
over data structures required the design of incremental, per-packet
computation methods. Such adjustments played a vital role in enabling the
offload of several ML system stages to the data plane (see Figure 1), such as
data collection and processing (entropy values), model construction
(legitimate traffic), and analysis (threshold-based alarming).

A set of experiments for assessing the efficacy and efficiency of our proposal
was performed on a software-based P4 infrastructure. These experiments used
datasets from CAIDA representing normal and DDoS traffic. The DDoS traffic
data set is built in such a way that it contains only attack-related traces,
and has been used in several high impact papers. Regarding the use of a
software-based infrastructure, we consider that it does not limit our
assessment, since accuracy and resource utilization are expected to be
equivalent regardless the actual hardware target.

For low-intensity attacks, e.g., occupying a fraction of 4% of the whole
traffic, the detection accuracy exceeds 90%. Our approach requires a few
hundred milliseconds to detect attacks since it perform per-packet analysis
(as opposed to sampling). We refer the reader to [14] for a detailed
description of this work.

##  IV Discussion and insights

The implementations described above are by themselves evidence on the
feasibility of achieving the effective combination of ML and PDPs for network
security solutions. However, from the analysis of these possibilities, the
aforementioned use cases, and considering the examples of solutions available
in the literature, there are several points that we would like to highlight
since they introduce important and relevant insights leading to further
research lines.

Executing Complex Operations on the Data Plane. The set of mathematical
operations supported by programmable switches is limited. While implementing
addition or logical bit shift operations is easy, general mathematical
operations such as those commonly involved in ML algorithms (e.g., square
roots, divisions and in general floating point arithmetic) can not be
implemented. An approach proposed in the literature to overcome this situation
is using the tables within the switch to store values computed externally,
corresponding to hyperparameters of the ML algorithms upon their training.
Thus, the part of the algorithms running at the FDs consists of matching,
conditional and logic operators, which are naturally available within the
devices [13, 14].

Lightweight ML Algorithms in PDPs. Due to resource limitation, the algorithms
to be implemented within the FDs need to be lightweight. An strategy to
simplify the algorithms is the one previously mentioned of using look up
tables. The tables would contain the results of complex calculations that are
performed outside the FDs (e.g., at the control plane). This could be combined
with additional strategies such as the simplification of data representation
for the ML algorithms. Binarized Neural Networks are an example, in which the
weights associated to the Neural Network are replaced by binary bits stored
within registers in the FDs [11].

Centralized vs. Distributed PDP-based ML Systems. A centralized operation
would be the trivial scenario by taking advantage of the local traffic
visibility. However, for complex ML algorithms (e.g., Deep Neural Networks),
this would introduce an important overhead in operations such as feature
extraction. An approach to overcome this overhead is Federated Learning. In
this approach, FDs have a copy of the trained ML model, and improve them
according to their local observation. The adjusted model is sent back to the
control plane, where it is combined with the information provided by other
FDs, and it is further distributed to all the devices [11].

Scope of decisions according to the amount of information. PDPs enable data
collection in local contexts via the observation of traffic crossing the
forwarding device. This capability allows quick reaction upon the detection of
anomalies or shifts in pattern traffics which might be symptom of threats or
ongoing attacks. However, due to the locality of this observation, the
collected information can not be leveraged for strategic decisions by itself.
This information needs to be collected and somehow integrated in more
comprehensive analysis in order to make strategic decisions based on a global
view of the acquired information [8, 11].

What to Execute at the Data and Control Planes? The literature has shown two
main different approaches where the functionalities of programmable FDs make
feasible the implementation of simple and even complex ML algorithms. The
first approach mainly consists in implementing simple tasks such as counting,
classification, aggregation and grouping. Depending on the complexity of the
primitive operations involved (e.g. arithmetic), the implementation might be
complete or partially supported with the external execution of those
operations not available within the FD [13, 11, 12, 14]. The second approach
consists in offering the operations available in the FD as services which can
be invoked from the Control Plane. Hence, these operations can be leveraged
from complex algorithms, offloading the low level and simpler tasks to the FD
while leaving the more complex operations to be performed at control plane.
This approach reduces the overhead by avoiding excessive information exchange
between control and data plane [7, 9].

##  V Conclusions

In this paper, we have discussed the contributions that PDPs bring to the
development of IDS solutions based on ML algorithms. In the presented use
cases we showed the implementation of traditional ML algorithms such as KNN.
However, more complex and state-of-the-art solutions such as Deep Neural
Networks can also leverage PDPs. Initial efforts on this trend can be observed
on the interest of this topic evident in different high quality research
venues such as the NETAI Workshop which is part of the SIGCOMM conference, the
ACM HotNETS workshop, and IEEE IM Conference, among others. Although data-
plane functionalities provide more sophisticated possibilities, implementing
functions at forwarding devices is a non-trivial task that needs to be
carefully performed to explore the advantages while preserving high
performance in packet forwarding.

## Acknowledgment

This paper has been supported by the project “Red Temática CYTED 519RT0580”
funded by the Ibero-American Science and Technology Program CYTED.

## References

  * [1] W. L. da Costa Cordeiro, J. A. Marques, and L. P. Gaspary, “Data plane programmability beyond openflow: Opportunities and challenges for network and service operations and management,” _Journal of Network and Systems Management_ , vol. 25, no. 4, pp. 784–818, 2017.
  * [2] J. Xie, F. R. Yu, T. Huang, R. Xie, J. Liu, C. Wang, and Y. Liu, “A survey of machine learning techniques applied to software defined networking (sdn): Research issues and challenges,” _IEEE Communications Surveys & Tutorials_, vol. 21, no. 1, pp. 393–430, 2018.
  * [3] D. R. Ports and J. Nelson, “When should the network be the computer?” in _Proceedings of the Workshop on Hot Topics in Operating Systems_ , 2019, pp. 209–215.
  * [4] D. C. Le and N. Zincir-Heywood, “A frontier: Dependable, reliable and secure machine learning for network/system management,” _Journal of Network and Systems Management_ , pp. 1–23, 2020.
  * [5] A. Binbusayyis and T. Vaiyapuri, “Identifying and benchmarking key features for cyber intrusion detection: An ensemble approach,” _IEEE Access_ , vol. 7, pp. 106 495–106 513, 2019.
  * [6] T. Kohler, R. Mayer, F. Dürr, M. Maaß, S. Bhowmik, and K. Rothermel, “P4cep: Towards in-network complex event processing,” in _Proceedings of the 2018 Morning Workshop on In-Network Computing_ , 2018, pp. 33–38.
  * [7] A. Gupta, R. Harrison, M. Canini, N. Feamster, J. Rexford, and W. Willinger, “Sonata: Query-driven streaming network telemetry,” in _Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication_ , 2018, pp. 357–371.
  * [8] A. Sapio, M. Canini, C.-Y. Ho, J. Nelson, P. Kalnis, C. Kim, A. Krishnamurthy, M. Moshref, D. R. Ports, and P. Richtárik, “Scaling distributed machine learning with in-network aggregation,” _arXiv preprint arXiv:1903.06701_ , 2019.
  * [9] C. Mustard, F. Ruffy, A. Gakhokidze, I. Beschastnikh, and A. Fedorova, “Jumpgate: In-network processing as a service for data analytics,” in _11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19)_ , 2019\.
  * [10] A. Sapio, I. Abdelaziz, A. Aldilaijan, M. Canini, and P. Kalnis, “In-network computation is a dumb idea whose time has come,” in _Proceedings of the 16th ACM Workshop on Hot Topics in Networks_ , 2017, pp. 150–156.
  * [11] Q. Qin, K. Poularakis, K. K. Leung, and L. Tassiulas, “Line-speed and scalable intrusion detection at the network edge via federated learning,” in _2020 IFIP Networking Conference (Networking)_. IEEE, 2020, pp. 352–360.
  * [12] S. G. Macías, L. P. Gaspary, and J. F. Botero, “Oracle: Collaboration of data and control planes to detect ddos attacks,” 2020, [arXiv preprint arXiv:2009.10798, September 2020].
  * [13] Z. Xiong and N. Zilberman, “Do switches dream of machine learning? toward in-network classification,” in _Proceedings of the 18th ACM Workshop on Hot Topics in Networks_ , 2019, pp. 25–33.
  * [14] Â. C. Lapolli, J. A. Marques, and L. P. Gaspary, “Offloading real-time ddos attack detection to programmable data planes,” in _2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM)_. IEEE, 2019, pp. 19–27.
  * [15] B. Lewis, M. Broadbent, and N. Race, “P4id: P4 enhanced intrusion detection,” in _2019 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN)_. IEEE, 2019, pp. 1–4.

Sergio Armando Gutiérrez Received his PhD in Computer Science from Universidad
Nacional de Colombia, Medellín in 2018. He is currently full time lecturer at
Universidad Autónoma Latinoamericana. His research interests include computer
networks, security in computer networks, , programmable devices and
application of machine learning to computer networks. ORCID:
0000-0003-2880-4601
---
Luciano Paschoal Gaspary received his Ph.D. in Computer Science from
Universidade Federal do Rio Grande do Sul, Brazil in 2002. He is currently
Deputy Dean and Associate Professor at the Institute of Informatics at UFRGS.
His research interests include computer networks, network management, and
computer system security. ORCID: 0000-0002-7561-5582
---
Juan Felipe Botero received his Ph.D. degree in Telematics engineering from
Universidad Técnica de Cataluña, Spain, in 2013. He is an Associated Professor
with the Electronics and Telecommunications Engineering Department at
Universidad de Antioquia, Colombia. His main research interests include
quality of service, software defined networking, network virtualization, and
resource allocation. ORCID: 0000-0002-7072-8924
---
John Willian Branch received his PhD in Systems Engineering in 2007 from
Universidad Nacional de Colombia, Medellín. He is a full professor at the
Computing and Decision Sciences Department, in the Facultad de Minas,
Universidad Nacional de Colombia, Medellin, Colombia. His research interests
include: automation, computer vision, digital image processing and
computational intelligence techniques applied in different domains ORCID:
0000-0002-0378-028
---