# Detection of Fake Generated Scientific Abstracts

Panagiotis C. Theocharopoulos1, Panagiotis Anagnostou1, Anastasia Tsoukala1,
Spiros V. Georgakopoulos2, Sotiris K. Tasoulis1 and Vassilis P. Plagianakos1
1Department of Computer Science and Biomedical Informatics
University of Thessaly, Greece
Email: {ptheochar, panagno, antsoukala, stasoulis<EMAIL_ADDRESS>2Department of
Mathematics
University of Thessaly, Greece
Email<EMAIL_ADDRESS>
###### Abstract

The widespread adoption of Large Language Models and publicly available
ChatGPT has marked a significant turning point in the integration of
Artificial Intelligence into people’s everyday lives. The academic community
has taken notice of these technological advancements and has expressed
concerns regarding the difficulty of discriminating between what is real and
what is artificially generated. Thus, researchers have been working on
developing effective systems to identify machine-generated text. In this
study, we utilize the GPT-3 model to generate scientific paper abstracts
through Artificial Intelligence and explore various text representation
methods when combined with Machine Learning models with the aim of identifying
machine-written text. We analyze the models’ performance and address several
research questions that rise during the analysis of the results. By conducting
this research, we shed light on the capabilities and limitations of Artificial
Intelligence generated text.

###### Index Terms:

GPT-3, ChatGPT, COVID-19, Deep Learning, Large Language models

##  I Introduction

The development of the transformer architecture along with the attention
mechanism took the lead on Natural Language Processing (NLP) tasks, due to its
ability to process an entire input of sequence at once, unlike the Recurrent
Neural Networks (RNNs). The wider use of the transformers architecture leads
to Large Language models (LLMs). The LLMs are language models trained on a
tremendous amount of data with multiple parameters. Furthermore, the
development of pre-trained models allowed language models to learn generic
language patterns and structures from large and diverse data before being
fine-tuned for a specific task [12].

After the conversational Artificial Intelligence (AI) tool ChatGPT became
publicly available in December 2022, the discipline of computer science, among
other fields, was shaken to its core. Developed by OpenAI, ChatGPT is a
language model representing a refined and evolved version of Generative Pre-
trained Transformer (GPT) models. The tool’s remarkable potential for
executing a vast array of tasks with high precision has attracted widespread
attention.

The introduction of ChatGPT, and similar language models, has been met with
excitement and concern from scientists and researchers. Despite the impressive
development of language models in AI and NLP, there are valid concerns
regarding their misuse [11]. These models may be used for more sophisticated
phishing or social engineering schemes and for impersonating individuals or
organizations. It is crucial to cautiously protect potentially affected areas
by malicious intent and consider the ethical implications with further
research [15]. Journalism, cybersecurity, intellectual property, customer
service, and education are most prominently affected by the proliferation of
generated text applications. The area of concern of our study is education, as
the ability of language models to generate academic writing could lead to a
rise in plagiarism and academic dishonesty. Recently, some examples of
generated scientific and academic texts have already ended up as conference
papers [3]. The first scientific paper generator was the SCIgen program,
created in 2005 by three MIT graduate students. SCIgen is a computer program,
created as a prank, which generates random computer science research papers
[24]. Detecting synthetic text is getting extremely challenging because
sophisticated text generation has been pursued for many years, culminating in
the impressive capabilities of GPT-3, which can even deceive human readers
[10]. Many text generation models followed either for severe or humorous
causes. However, the necessity for a distinction between generated and actual
text became even more substantial [10, 3]. Currently, GPT-3 has approximately
570 GB of text data as a training dataset, combining the 175 billion
parameters [1]. Although ChatGPT and GPT-3 differ in terms of training
methodology and fine-tuning, the latter is provided via its platform by OpenAI
primarily for research purposes [21, 1].

The scope of this study is to investigate a way that distinguishes text
generated by LLMs, in the particular scientific text produced by GPT-3.
Additionally, the study tries to understand the results by answering logical
questions, based on the false outcomes of the best-performed model. The rest
of this paper is structured into different sections. Section II provides the
related literature about the topic. Section III describes the methodology
applied in the study. Section V, discusses our thoughts on the findings. We
conclude the paper in Section VI and suggest ideas for future research
directions.

##  II Related Work

Although the concept of AI text recognition based on Large Language Models
(LLMs) is relatively new, there is already related research in the field. A
tool called Giant Language Model Test Room (GLTR) [7] is implemented to detect
whether a text has been generated by a machine or written by a human. The work
is based on several statistical methods to detect AI-generated text from GPT-2
and BERT, including computing the model density of generated output and
comparing it to human-generated text, as well as using the probability of a
word and the distribution over four buckets of absolute ranks of predictions.
On the same page, in order to discriminate machine-generated text, the study
[27] proposes the FActual Structure of Text (FAST) method, a graph-based model
that utilizes the factual structure of a document for deep fake detection of
text. The graph nodes are extracted by Named Entity Recognition (NER).
Furthermore, the sentence representations were constructed via document-level
aggregation for the final prediction, where the consistency and coherence of
continuous sentences are sequentially modeled. For the evaluation of FAST, the
authors used AI-generated text data generated from GROVER and GPT-2. The
approach outperformed transformer-based models [27]. Based on the text
generated by GPT-2 model, OpenAI created a detector of whether a text is
artificially generated. The detector has been based on a fine-tuning version
of the RoBERTa base. The model has been trained using $510$ tokens and using
$5,000$ text samples from the WebText dataset $5,000$ texts generated by a
GPT-2 model. The model achieved $95\%$ of accuracy [23]. Furthermore, in [14]
is presented a coherence-based contrastive learning model (CoCo) for the
detection of text generated from machines. To achieve that, the authors
modeled the text coherence with entity consistency and sentence interaction.
The CoCo model outperformed in terms of accuracy and F1 score similar models
such as GPT-2, RoBERTa, and XLNet but also the GROVER and FAST models.
Additionally, the authors of [18] proposed a tool, called DetectGPT, that
detects machine-generated text from LLMs. DetectGPT is a more effective
approach for detecting machine-generated text than existing zero-shot methods.
The authors found that DetectGPT significantly improved the detection of fake
news articles generated by $20$B parameter GPT-NeoX, achieving an AUROC of
$0.95$ compared to the strongest zero-shot baseline’s AUROC of $0.81$.
Additionally, the authors found that DetectGPT does not require training a
separate classifier or collecting a dataset of a real or generated corpus,
making it a more efficient and practical approach for detecting machine-
generated text [18].

The [16] has focused on the involvement of AI writing in scientific writing.
The authors analyze the similarities and differences between the two types of
content, including writing style, consistency, coherence, language redundancy,
and factual errors. The results suggest that while AI has the potential to
generate accurate scientific content, there is still a gap in terms of depth
and overall quality. The study utilizes the logistic regression model as a
detector along with OpenAI’s RoBERTa detector. The results suggest that there
are significant differences in the distribution of text between human-written
and AI-generated scientific work and that AI-generated abstracts tend to be
more generic than human-written ones. Additionally, the logistic regression
model achieved a higher F1-score for detecting AI-generated text [16].

Finally, as the extensive use of GPT-3 and ChatGPT continues to grow, OpenAI
published on January 31, 2023, an update of the aforementioned model. The
revised version of the AI text classifier fine-tuned pre-trained models in
order to distinguish the AI-generated text. The model used data from three
origins of the human-written text, a dataset from Wikipedia, the 2019 WebText
dataset, and a group of human examples gathered during the instruction of
InstructGPT. Although, the model needs $1,000$ tokens and more in order to
produce reliable results. The revised edition of the model has shown better
results than the previous version, which was based on GPT-2. In the validation
set, the AUC score increased from $0.95$ to $0.97$, and in the challenge set,
it increased from $0.43$ to $0.66$. Furthermore, the classifier accurately
identifies $26\%$ of text generated by AI as possibly AI-written (true
positives) but mistakenly identifies $9\%$ of human-written text as AI-
generated (false positives) [21].

##  III Research Methodology

The purpose of this study is to suggest a strategy for distinguishing between
scientific literature produced by humans and AI-generated. We examine various
techniques for text representation and Machine Learning models which are
applied to both human-written and AI-generated scientific abstracts. We
utilized the GPT-3 model to create a database of AI-generated abstracts, which
is publicly available111https://github.com/panagiotisanagnostou/AI-GA. We
examine well-established NLP methods for text representation as well as
embedded representation methods in order to train classic ML and Deep Learning
models examining their performances and trying to answer three research
questions:

  1. 1.

“What words dominate the misclassified AI generated texts compared to the
correctly classified ones?”

  2. 2.

“What is the reason behind the misclassified texts?”

  3. 3.

“Does the size of the title we provided the GPT-3 model affect the
classification error?”

The strategy that produces the most accurate outcomes will be recommended for
addressing this particular issue. A workflow diagram of the adopted strategy
for creating the AI-generated dataset is presented in Figure 1.

Figure 1: Schematic overview of the study. Dataset Generation: From the
CORD-19 data has been collected the titles and the abstracts of the academic
literature. The titles of the selected work have been prompted to GPT-3 model,
via its API for the AI-generated abstract based on its title (Left). Data
Analysis: The study involved text cleaning and data representations using
various methods, as well as the models’ results evaluation (Right).

###  III-A AI-Generated Dataset

For this study, we used the publicly available COVID-19 Open Research Dataset
(CORD-19) [26]. The CORD-19 is an aggregation of published papers and pre-
prints from multiple sources, which has been collected in order to promote
research on COVID-19 and related coronaviruses. The dataset contains tens of
thousands of scholarly articles, some of them with full text. The primary
objective of CORD-19 is to provide the global research community with a wealth
of information, to support the fight against COVID-19. This has been achieved
by creating a freely available dataset, which can be used in conjunction with
advanced Natural Language Processing (NLP) and other Artificial Intelligence
(AI) techniques to extract new insights. As there is a large number of
academic papers regarding coronaviruses, there is a pressing need for text
mining and information retrieval systems that can help medical researchers
keep up. We randomly selected a subset of academic works from the CORD-19
dataset, consisting of $14,331$ English-language papers with both titles and
abstracts.

In order to produce the AI-generated abstracts, we used one of the most
advanced LLMs available, the GPT-3. Specifically, we used the GPT-3 model also
known as the Davinci model, which is one of the most capable GPT-3 models and
produces higher quality results [20]. Although GPT-3.5 models would have been
preferable for this task, they were not publicly available via an API at the
time of this experiment. To create the AI abstracts, we developed an
appropriate prompt with the following form: “Create an abstract for a
scientific journal with a formal tone, academic language, and a background
story of the topic in a unique paragraph with the title: $\mathbf{\hat{t}}$”,
where $\mathbf{\hat{t}}$ is the human-written original title of the academic
paper. In order to make the generated text more creative and novel, several
adjustments were made to the requested parameters. These adjustments included
increasing the freedom to create novelty by selecting the most likely tokens
through changes to the randomness of the generated text and penalizing the
frequency of repeated words or phrases to generate more diverse responses.
Additionally, the model was set to avoid using words or phrases already
present in the prompt or previously generated text in order to generate more
unique responses. The titles provided for the prompt were the same as the
titles selected from the CORD-19 dataset. The final dataset contains $28,662$
entries, involving $14,331$ human-written abstracts and $14,331$ AI-generated
abstracts, corresponding to the same titles. In order to ensure that the
analysis is based on high-quality data, we had to perform a text-cleaning
procedure. The cleaning procedure involved:

  * •

Removing special characters such as HTML tags or special characters e.g. next
line symbol,

  * •

Removing whitespace or line breaks in the text,

  * •

Removing stop words, such as “the”, “and”, “of”, etc. because they do not
convey meaningful information,

  * •

Removing any non-alphabetic characters or numbers that may be present in the
text,

  * •

Normalizing text involves converting all text to a consistent format, such as
lowercase, to make it easier to process and analyze,

  * •

Removing the two most frequent words, namely “paper” and “study” from both
original and generated texts due to their high frequency on the generated
text.

###  III-B Text representation

Text representation is an essential step in NLP. The main purpose is to
transform unstructured text data into a structured and machine-readable
format. The purpose of text representation is to enable text algorithms to
perform various NLP tasks such as text classification, sentiment analysis, and
machine translation. In the following section, we describe some of the methods
used as we carried out our tests.

####  III-B1 Term Frequency-Inverse Document Frequency (TF-IDF)

TF-IDF is a method employed in NLP and information retrieval to evaluate the
significance of a term in a document or corpus. It is created by combining two
distinct components: Term Frequency (TF) and Inverse Document Frequency (IDF).
The TF component quantifies the number of occurrences of a term in a document,
while the IDF component assigns less weight to commonly used words and more
weight to less frequent words. When a word appears more frequently in
documents, it will have a higher term frequency, while less frequent
occurrences of a word will result in greater importance (IDF) for that word
when it is searched in a specific document. TF-IDF is the product of
multiplying TF and IDF components [22]. During our study, we removed words
that exhibited less than 1% or more than 99% of their values to filter out
unwanted noise in the data from sparse and dense variables.

####  III-B2 Named Entity Recognition

Named Entity Recognition (NER) is a fundamental task in the field of NLP,
aimed at identifying and classifying named entities from a given text. The
task of NER involves analyzing text data and identifying named entities based
on a range of linguistic features, including part-of-speech tags, syntactic
structures, and context. NER algorithms employ various machine learning
techniques such as statistical models, neural networks, and rule-based methods
to automatically detect and extract named entities from unstructured text
data. The output of a NER system is typically a structured representation of
the identified entities, which may include labels indicating their types [9,
19]. As we conducted our experiments, we utilized a bioinformatics-focused
approach due to the main topic of the abstracts being about COVID-19,
employing NER via the SciSpacy python package’s “en_core_sci_lg” model to
identify relevant bioinformatics terminology [19]. The extracted n-grams, up
to a size of 9, were then used as input for our classification models.

####  III-B3 Word2Vec

Word2Vec is a popular method for generating word embeddings, which are vector
representations of words that capture their semantic and syntactic meaning
[17]. Word2Vec is a widely used natural language processing technique that is
predicated on the premise that words that co-occur in similar linguistic
contexts tend to have similar semantic meanings. It comprises two neural
network-based models, namely the Continuous Bag of Words (CBOW) and Skip-gram
models, which generate word embeddings from large text corpora. These
embeddings encode the semantic and syntactic relationships between words,
thereby facilitating a range of natural language processing applications[17,
8]. In this experiment, we used the pre-trained model google-news-300, an
unsupervised natural language processing model developed by Google. It is
based on the Word2Vec technique and is designed to generate high-quality word
embeddings from large corpora of news articles. The model consists of
300-dimensional vectors, each of which represents a unique word in the
vocabulary.

####  III-B4 Contextualized Representations

Contextualized representations refer to the capability of language models to
produce unique and dynamic representations of words and sentences based on the
surrounding context. These representations not only capture the inherent
meaning of individual words but also the subtleties and nuances of their usage
within a specific context. The use of contextualized representations enhances
the accuracy and naturalness of language output and is increasingly vital in
NLP tasks such as sentiment analysis, question answering, and machine
translation. Contextualized representations are achieved through deep neural
network architectures that are trained on extensive text data. These models
usually rely on recurrent or transformer-based architectures that can capture
long-range dependencies between words and phrases in a text [5].

An example of such a model is the Bidirectional Encoder Representations from
Transformers (BERT), which has achieved state-of-the-art results on a variety
of natural language processing tasks [4]. BERT is a type of machine learning
model that uses bidirectional representation, which allows it to comprehend
the entire sequence of words in the context of the sentence. The model is
trained using a Masked Language Model (MLM) objective, which involves masking
out $15\%$ of the input words during training. These masked tokens are
replaced with either a “MASK” token, a random word, or the same word, with
frequencies of $80\%$, $10\%$, and $10\%$, respectively [2]. BERT comes in two
models: the base model with 12 encoders and the large model with 64 encoders.
Additionally, the BERT model has the ability to perform Next Sentence
Prediction, where it is pre-trained on pairs of text to learn the
relationships between sentences and determine if a given sentence follows the
previous sentence or not. BERT’s input includes Token Embeddings, Segment
Embeddings, and Positional Embeddings. The word tokenization process is
performed by the BERT tokenizer, which uses the concept of word-piece
tokenizer to break down certain words into sub-words or word pieces if the
word can be represented by multiple tokens [4, 25, 2]. During our research
experiments, we used the BERT base pre-trained model. On top of BERT’s
architecture, we added a fully connected layer, where each neuron is connected
to every neuron in the previous layer, and each connection has an associated
weight parameter that is learned during training. In addition, a
regularization technique randomly drops out the $50\%$ of the neurons in the
layer during each training iteration, to prevent the model from overfitting
over the training data.

In our experiments, we adopted an alternative methodology that involved the
use of a pre-trained NLP model designed for general usage. The objective
behind this approach was to map our data onto the embedding space established
by the pre-existing model. Specifically, we employed the “English Wikipedia
Dump of November 2021” model to achieve this goal [13]. The Wikipedia dataset
is comprised of filtered articles from all languages. The construction of
these datasets entails the extraction of data from the Wikipedia dump, with
each language having a distinct file. Each record within the dataset
corresponds to a complete Wikipedia article, which has been processed to
exclude superfluous sections, such as markdown, references, and other unwanted
content [6]. By adopting this technique, we succeeded in creating a data
matrix of $300$ variables.

##  IV Experimental Results

This section presents the results of how each method of word representation
technique, combined with the machine learning methods, performed and
distinguish the original abstracts from the AI-generated ones. To evaluate the
effectiveness of the methods, we conducted a comparison using a range of text
representation techniques as mentioned in Section III-B with various Machine
Learning (ML) methods such as Logistic Regression (LR), Multinomial Naive
Bayes (MNB), Support Vector Machine (SVM), Long Short-Term Memory networks
(LSTM), and BERT.

Our analysis involved training and testing these algorithms on a large corpus
of original and AI-generated abstracts. The training of the methods has
involved the $80\%$ of the corpus, containing $22,930$ entries, and the
evaluation of the remaining $20\%$ of $5,732$ entries. Furthermore, the
training and the evaluation entries contain both the real and the generated
abstract from each of the paper titles we used in the abstract generation
process. For the evaluation of the performance of each algorithm, we used the
metrics such as accuracy, precision, recall, and F1 score.

The combination of text representations with the ML models can be summarized
as follows:

  * •

LR with TF-IDF

  * •

MNB with TF-IDF

  * •

SVM with TF-IDF

  * •

LR with NER

  * •

MNB with NER

  * •

SVM with NER

  * •

LR with EWD

  * •

SVM with EWD

  * •

BERT with keeping the initial weights fixed

  * •

BERT with fine-tuning the model by updating the initial weights (BERT Fine-
Tuning)

  * •

LSTM with BERT

  * •

LSTM with Word2Vec

Table I presents the average performance of each method over 100 independent
iterations. To mitigate the risk of overfitting, we employed an early stopping
strategy by continuously monitoring the train and validation loss,
particularly for the LSTM and BERT models. Specifically, once the validation
loss reached its minimum value and began to increase, we terminated the
training process and saved the current model. Our analysis reveals that the
LSTM model combined with Word2Vec representation achieved the highest accuracy
rate of $98.7\%$. Although the BERT model performed equally well in terms of
accuracy and better in F1 score, a composite measure of precision and recall,
it exhibited lower precision and recall scores. The LSTM model incorporating
Word2Vec achieved an AUC score of 0.987, which is higher than the AUC scores
of the original and updated OpenAI models. The OpenAI models were developed to
distinguish AI-generated text from human-written text, and their AUC scores
were 0.95 and 0.97, respectively.

TABLE I: Model results Models | Metrics
---|---
Accuracy | F1 | Precision | Recall
LSTM + BERT | 0.697 | 0.700 | 0.692 | 0.708
MNB + NER | 0.797 | 0.800 | 0.811 | 0.789
BERT | 0.834 | 0.830 | 0.830 | 0.830
SVM + EWD | 0.865 | 0.857 | 0.909 | 0.811
MNB + TF-IDF | 0.868 | 0.863 | 0.897 | 0.831
LR + EWD | 0.874 | 0.870 | 0.901 | 0.841
LR + NER | 0.906 | 0.905 | 0.900 | 0.912
SVM + NER | 0.918 | 0.918 | 0.912 | 0.924
LR + TF-IDF | 0.975 | 0.975 | 0.971 | 0.980
SVM + TF-IDF | 0.980 | 0.980 | 0.978 | 0.982
BERT (Fine-Tuning) | 0.987 | 0.984 | 0.986 | 0.982
LSTM - w2v | 0.987 | 0.987 | 0.987 | 0.986

Our findings suggest that Word2Vec embeddings outperformed BERT embeddings in
this particular task, likely due to the specialized language domain of
scientific abstracts. Specifically, Word2Vec was better able to capture word
context within sentences, while BERT focused on predicting masked tokens or
generating subsequent sentences in the document. Additionally, while Word2Vec
generated fixed-size embeddings for each word, BERT’s embeddings were
contextualized and varied based on the word’s usage in the document.

Figure 2: Most frequent word appearances in both the human-created texts
(left) and the AI-generated texts (right).

##  V Discussion

The experimental results show that it can be detected the human created and
AI-generated scientific abstracts highly accurate. The best model, LSTM with
the Word2Vec representation) misclassified 103 samples from the total test
samples, where 43 were human-created abstracts while the rest were AI-
generated. It is obvious that the portion of misclassified texts is
significantly low. Having set up the research question in Section III we
continue with an explanatory analysis of the wrong classified abstracts, to
understand the particularities of the AI-generated abstract that lead the
model to classify them as human text. The crucial element of abstract
construction is the title of the scientific papers, so we investigate the
attributes of the titles.

Figure 3: Most frequent word appearances in all the titles (left) and in the
titles of the misclassified AI-generated titles (right). Figure 4: Most
frequent word appearances in all the AI-generated abstracts (left) and in the
misclassified AI-generated abstracts (right).

The first question we raised is, “Does the size of the title we provided the
GPT-3 model affect the classification error?” The answer to this question is
straightforward negative if we consider that the average size of the titles
that constitute our dataset is 13.64 words, and the length of the
misclassified generated text has an average title size of 13.68 words. At this
point, there is no to pose the same question in the human-created texts.

The next question that needs answering is, “What words dominate the
misclassified AI-generated texts compared to the correctly classified ones?”
Before we answer this question, we need to understand the fundamental
difference between human-created and AI-generated texts. To that end, Figure 2
presents the most frequently appeared words in both the human created and the
AI-generated texts. It becomes immediately apparent from this figure that
there is a difference between those two sets. In the human-created texts, we
can see that they contain not-so-common words (such as influenza or treatment)
with high frequency. On the contrary, AI-generated texts contain more widely
used words. With this knowledge, we can now see Figure 4 and Figure 3. The
combination of those figures shows us that both the titles and the abstract of
the misclassified AI-generated texts contain, with high frequency, not-so-
common words, and in the case of the titles, they contain significantly more
uncommon words than the rest of the titles of the dataset. Only from this fact
can we conclude that the more information, in terms of content, contained in
the titles given to the GPT-3 model, the better text generation it will
conduct.

On the other hand, the human-created text that was misclassified had a poorer
vocabulary than the main bulk of the human-created texts, so much so that the
LSTM with Word2Vec methodology misclassified it as AI-generated text. The word
analysis can be seen in Figure 4.

Figure 5: Most frequent word appearances in all the human-created abstracts
(left) and in the misclassified human-created abstracts (right).

##  VI Conclusion

In this work, we presented a method of distrusting between text generated by
Artificial Intelligence and text created by humans. To that end, we presented
several NLP classification methodologies, from simple ones, such as the LR
with TF-IDF method, or more complicated ones, such as LSTM with BERT. From the
results of the classification task, we can conclude that the problem at this
point of the development of the GPT-3 model is seemingly a problem that can be
tackled efficiently.

Moving forward, we intend to produce a higher volume dataset using the updated
state-of-the-art Large Language Model, with the aim of further evaluating and
improving the efficacy of our proposed method. The larger dataset will enable
us to investigate the generalizability and scalability of our approach across
different domains, languages, and text genres. Moreover, it will allow us to
explore the potential of our method for real-world applications. By advancing
our understanding of the capabilities and limitations of AI-generated text, we
can pave the way for more responsible and ethical use of this technology in
the future.

## References

  * [1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [2] Nalini Chintalapudi, Gopi Battineni, and Francesco Amenta.  Sentimental analysis of covid-19 tweets using deep learning models.  Infectious Disease Reports, 13(2):329–339, 2021.
  * [3] Evan Crothers, Nathalie Japkowicz, and Herna Viktor.  Machine generated text: A comprehensive survey of threat models and detection methods.  arXiv preprint arXiv:2210.07321, 2022.
  * [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [5] Kawin Ethayarajh.  How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings.  arXiv preprint arXiv:1909.00512, 2019.
  * [6] Wikimedia Foundation.  Wikimedia downloads.
  * [7] Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush.  Gltr: Statistical detection and visualization of generated text, 2019\.
  * [8] Yoav Goldberg and Omer Levy.  word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method.  arXiv preprint arXiv:1402.3722, 2014.
  * [9] Archana Goyal, Vishal Gupta, and Manish Kumar.  Recent named entity recognition and classification techniques: a systematic review.  Computer Science Review, 29:21–43, 2018.
  * [10] Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck.  Automatic detection of generated text is easiest when humans are fooled.  arXiv preprint arXiv:1911.00650, 2019.
  * [11] Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan.  Automatic detection of machine generated text: A critical survey.  arXiv preprint arXiv:2011.01314, 2020.
  * [12] Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stepha Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci.  Chatgpt for good? on opportunities and challenges of large language models for education.  Learning and Individual Differences, 103:102274, 2023.
  * [13] Andrei Kutuzov, Murhaf Fares, Stephan Oepen, and Erik Velldal.  Word vectors, reuse, and replicability: Towards a community repository of large-text resources.  In Proceedings of the 58th Conference on Simulation and Modelling, pages 271–276. Linköping University Electronic Press, 2017.
  * [14] Xiaoming Liu, Zhaohan Zhang, Yichen Wang, Yu Lan, and Chao Shen.  Coco: Coherence-enhanced machine-generated text detection under data limitation with contrastive learning, 2022.
  * [15] Yongqiang Ma, Jiawei Liu, and Fan Yi.  Is this abstract generated by ai? a research for the gap between ai-generated scientific text and human-written scientific text.  arXiv preprint arXiv:2301.10416, 2023.
  * [16] Yongqiang Ma, Jiawei Liu, Fan Yi, Qikai Cheng, Yong Huang, Wei Lu, and Xiaozhong Liu.  Ai vs. human – differentiation analysis of scientific content generation, 2023.
  * [17] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.
  * [18] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn.  Detectgpt: Zero-shot machine-generated text detection using probability curvature.  arXiv preprint arXiv:2301.11305, 2023.
  * [19] Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar.  ScispaCy: Fast and robust models for biomedical natural language processing.  In Proceedings of the 18th BioNLP Workshop and Shared Task. Association for Computational Linguistics, 2019.
  * [20] OpenAI.  Openai api.  Computer software, 2021.
  * [21] OpenAI.  New ai classifier for indicating ai-written text.  Computer software, 2023.
  * [22] Shahzad Qaiser and Ramsha Ali.  Text mining: use of tf-idf to examine the relevance of words to documents.  International Journal of Computer Applications, 181(1):25–29, 2018\.
  * [23] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al.  Release strategies and the social impacts of language models.  arXiv preprint arXiv:1908.09203, 2019.
  * [24] Jeremy Stribling, Max Krohn, and Dan Aguayo.  Scigen-an automatic cs paper generator, 2005.
  * [25] Panagiotis C Theocharopoulos, Anastasia Tsoukala, Spiros V Georgakopoulos, Sotiris K Tasoulis, and Vassilis P Plagianakos.  Text analysis of covid-19 tweets.  In International Conference on Engineering Applications of Neural Networks, pages 517–528. Springer, 2022.
  * [26] Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Nancy Xin Ru Wang, Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier.  CORD-19: The COVID-19 open research dataset.  In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Online, July 2020. Association for Computational Linguistics.
  * [27] Wanjun Zhong, Duyu Tang, Zenan Xu, Ruize Wang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin.  Neural deepfake detection with factual structure of text.  arXiv preprint arXiv:2010.07475, 2020.


# Modified Supervised Contrastive Learning
for Detecting Anomalous Driving Behaviours

Shehroz S. Khan1,2, Ziting Shen2, Haoying Sun2, Ax Patel2, and Ali Abedi1

###### Abstract

One of the major reasons for vehicle accidents is distracted driving.
Therefore, detecting distracted driving behaviours is of paramount importance
to reduce millions of deaths and injuries occurring worldwide. It is given
that majority of the time, drivers are driving normally. Distracted or
anomalous driving behaviours are deviations from the ’normal’ driving that
need to be identified correctly to alert the driver. However, these driving
behaviours do not comprise of one specific type of driving style and their
distribution can be different during training and testing phases of a
classifier. We formulate this problem as a supervised contrastive learning
approach to learn a visual representation to detect normal, and seen and
unseen anomalous driving behaviours. We made a change to the standard
contrastive loss function to adjust the similarity of negative pairs to aid
the optimization. Normally, in a (self) supervised contrastive framework, the
architecture contains encoding layers followed by a projection head. The
projection head layers are omitted during testing phase as the encoding layers
are considered to contain general visual representative information. However,
we assert that for supervised contrastive learning task, including projection
head will be beneficial. We showed our results on a Driver Anomaly Detection
dataset that contains $783$ minutes of video recordings of normal and
anomalous driving behaviours of $31$ drivers from various from top and front
cameras (both depth and infrared). We also performed an extra step of fine
tuning the labels in this dataset. Out of $9$ video modalities combinations,
our modified contrastive approach improved the ROC AUC on $7$ in comparison to
the baseline models (from $3.12\%$ to $8.91\%$ for different modalities); the
remaining two models also had manual labelling. We performed statistical tests
that showed evidence that our modifications perform better than the baseline
contrastive learning setup. Finally, the results showed that the fusion of
depth and infrared modalities from top and front view achieved the best AUC
ROC of $0.9738$ and AUC PR of $0.9772$.

## Introduction

According to World Health Organization, approximately $1.3$ million people die
each year due to road traffic accidents (Pietrasik 2021). Distracted driving
is one of the key risk factors, along with speeding, use of substances, and
unsafe vehicles (Pietrasik 2021). In many jurisdictions, such as in Ontario
(Canada), distracted driving laws prohibit drivers to use phones or other
hand-held wireless devices, to view display screens unrelated to driving, or
to manually program GPS devices during driving (of Ontario 2021). In reality,
distracted driving extends from such legal definitions to various day-to-day
actions, such as drinking and eating, talking with passengers, combing hair,
taking hands off the wheel and adjusting radio while driving. These actions
increase the risk of distracted driving related collisions.

Driver monitoring system targets to identify distracted or dangerous driving
behaviours that could cause potential traffic accidents. It can assess a
driver’s alertness, attentiveness, and focus and generates warning signals if
a distracted driving behaviour is observed. However, distracted driving does
not comprise of a unique set of behaviours behaviours and could result due to
negligence, carelessness, fatigue, or other unknown reasons. Most of the time
a driver may be driving safely, yet a minor distraction could be fatal.
Therefore, due to the diversity and less occurrence of these distracted
behaviours, we term them as ‘anomalous’ driving behaviours. This should not be
confused with traditional ‘anomaly detection’ problems where the anomalous
events are not available during training phase. In our problem formulation,
some annotated anomalous driving behaviours are available during training
phase. From a machine learning perspective, this represents an imbalanced
class classification problem with lots of normal or safe driving data and few
anomalous driving samples. Moreover, the distribution of anomalous driving
behaviours could be different during training and testing phases due to
variations in people exhibiting different types of anomalous driving
behaviours. Hence, new anomalous driving behaviours may occur during testing
that were not observed in the training phase.

If the data is captured through videos, then supervised 3D Convolutional
Neural Network (or its variants) with weighted classes could be the default
choice. However, more recently Contrastive learning (CL) has shown state-of-
the-art results in both self-supervised (Chen et al. 2020) and supervised
classification problems (Khosla et al. 2020). CL (Chen et al. 2020) enables to
learn effective visual representations by contrastive positive and negative
pairs (w.r.t. an anchor); thus, enabling them to work in imbalanced dataset
situations. CL architecture is generally realized through an encoder (2D or 3D
CNN or its variants), followed by projection head (feed forward layers).
Supervised CL (Khosla et al. 2020) leverages the class labels present in the
data to achieve state-of-the-art results in classification problems. In the
denominator of a typical CL loss function the exponential of dot product of
negative pairs is summed. If the labelled negative pairs are closer to the
positive pairs, due to negative samples being near to anchor or mislabelling,
then this can lead to difficulty in the overall optimization of the loss
function. This problem could be mitigated by scaling the sum term such that
the summed similarity of negative pairs remain far to align with the loss
function. We proposed to average the sum of negative pairs instead. The
projection head is generally discarded in self-supervised (Chen et al. 2020)
and supervised CL approaches (Khosla et al. 2020) with the assumption that the
representation learned at the encoder stage captures a generic visual
representation. However, we assert that for the supervised classification
task, the visual representation learned at the projection head stage should be
more helpful for the classification task.

We validated the above mentioned modifications on a driver anomaly detection
(DAD) dataset (Kopuklu et al. 2021). This dataset was collected from a driving
simulator comprising of a training set ($25$ participants, $650$ minutes) and
test set ($6$ participants, $133$ minutes) with different normal and anomalous
driving behaviours captured through top and front fitted depth and IR cameras.
The majority of the anomalous behaviours in the test set are not present in
the training set. We further manually annotated the videos to fine tune the
labels in this dataset. Our modified contrastive loss function, together with
projection head and manual labelling improved results on $7$ out of $9$ camera
modalities by $3.12\%$ to $8.91\%$.

## Related Work

Anomaly detection problems mostly refer to situations where the data for the
anomalous class (or positive class) is not available during the training phase
either due to its rarity, difficulty in collection, and health or safety
hazards (Khan and Madden 2014). However, in many situations, some labelled
data may be available for the positive class, albeit with a large skewed data
distribution. Such problems can be handled in a supervised machine learning
setup using different strategies, such as weighted loss functions and data
augmentation. In the context of video-based anomaly detection, there exists
many approaches based on autoencoders (Nogas, Khan, and Mihailidis 2020; Vu et
al. 2019; Borghesi et al. 2019) and adversarial learning (Khan, Nogas, and
Mihailidis 2021; Cai et al. 2021; Mehta et al. 2021). Supervised approaches
for video classification, includes based on 3D Convolutional Neural Networks
(3D-CNN) (Ji et al. 2012) and/or combined with sequential models (Wu et al.
2015; Girdhar et al. 2019). However, these models may be difficult to train on
highly skewed data, especially in cases where the distribution of positive or
anomalous class differs across training and test set.

Contrastive learning (Hadsell, Chopra, and LeCun 2006) approaches have been
evolving around the idea of contrasting positive pairs against negative pairs,
maximizing the ‘similarity’ among positive pairs while minimizing the
‘similarity’ between positive-and-negative pairs. The contrastive loss
equation is also closely connected to N-pair loss, triplet loss, as the latter
being a special case of generalized contrastive loss where the number of
positives and negatives are each one (Khosla et al. 2020).

Chen et al. (Chen et al. 2020) added a projection head consisting of several
feed forwarded layers in front of an encoder that improved quality of learned
representation. They showed that models with unsupervised pre-training with CL
achieved better than their supervised counterparts. Khosla et al. (Khosla et
al. 2020) extended this idea for supervised CL to leverage labels present in
the data. Their results achieved better results with Resnet-200 on ImageNet
dataset. They also showed that supervised CL outperformed over cross-entropy
on other datasets and two ResNet variants. This aprroach was shown to be
robust to data corruptions, and stable to the choice of optimizers and data
augmentations. CL has been used in anomaly detection with success. Zheng et
al. (Zheng et al. 2021) applied self-supervised CL on graph neural networks
for anomaly detection and outperformed state-of-the-art methods on several
datasets. Wang et al. (Wang, Zou, and Zhang 2020) developed a cluster
attention contrast approach to learn distinct subcategories of normality to
distinguish it from anomalous events.

Kopuklu et al. introduced the driver anomaly dataset containing $783$ minutes
of overall video data collected from $31$ individual, using top and down
fitted depth and IR cameras. They applied a supervised CL approach (the neural
network architecture was the same as Chen et al. (Chen et al. 2020)) that
achieved better results than cross-entropy loss on detecting anomalous driving
behaviours. The work presented in this paper presents modifications on Kopuklu
et al. (Kopuklu et al. 2021) in terms of loss function, including projection
head and leveraging manual labelling to obtain improved results on $7$ out of
$9$ camera modalities and support them with statistical testing.

Anomalous Actions in Training Set | Anomalous Actions in Test Set
---|---
Talking on the phone-left | Talking on the phone-left | Adjusting side mirror | Wearing glasses
Talking on the phone-right | Talking on the phone-right | Adjusting clothes | Taking off glasses
Messaging left | Messaging left | Adjusting glasses | Picking up something
Messaging right | Messaging right | Adjusting rear-view mirror | Wiping sweat
Talking with passengers | Talking with passengers | Adjusting sunroof | Touching face/hair
Reaching behind | Reaching behind | Wiping nose | Sneezing
Adjusting radio | Adjusting radio | Head dropping (dozing off) | Coughing
Drinking | Drinking | Eating | Reading
Table 1: List of anomalous actions in the training and test sets. The last two
columns in the test set contain actions not seen in the training set.

## Driver Anomaly Detection Datasets

Anomalous diving behaviours involve various types of distraction, such as
looking away from the road, moving away head, setting the radio, and texting
or calling on a cell phone, to name a few. To capture different types of
driving distractions, several driving datasets are available for analysis. For
example, datasets CVRR-HANDS 3D (Ohn-Bar, Martin, and Trivedi 2013), VIVA-
Hands (Das, Ohn-Bar, and Trivedi 2015) and DriverMHG (Köpüklü et al. 2020) are
hand focused datasets that are correlated with a driver’s ability to drive. In
addition, datasets such as DrivFace (Diaz-Chito, Hernández-Sabaté, and López
2016), DriveAHead (Schwarz et al. 2017) and DD-Pose (Roth and Gavrila 2019)
provides information on drivers face for paying attention and head pose
annotations of yaw, pitch and roll angles. Drive&Act (Martin et al. 2019)
provides distraction related anomalous driving data for $5$ near-IR cameras.
There are also many other driver monitoring datasets that provide face, head,
hands, and body actions to detect normal and distracted driving behaviours,
such as DMD (Ortega et al. 2020), and Pandora (Borghi et al. 2017). A major
limitation of these datasets is that all the normal and anomalous driving
samples are present in the train and test set; thus indirectly simplifying the
complexity of the problem. Anomalous behaviours vary across people and the
same set of actions available during training may not be available during
testing. The generalization capabilities of these datasets in real-world
driving situations remain unknown.

Kopuklu et al. (Kopuklu et al. 2021) presented the Driver Anomaly Detection
(DAD) that overcomes the limitations of previous datasets. This dataset is
collected from a driver simulator containing a real BMW car cockpit, while the
participants drive in a computer game that is projected in front of the car.
Two Infineon CamBoard pico flexx cameras (depth and infrared (IR)) were placed
on top and in front of the driver. The front camera recorded the drivers’
head, body and visible part of the hands, while the top camera recorded the
driver’s hands movements. Therefore, this dataset is both multi-view and
multi-modal. The dataset includes a training set comprising of $650$ minutes
of video data collected from $25$ participants, and a test set comprising of
$133$ minutes of video data from $6$ participants.

The multi-view and multi-modal video recordings were synchronized with the
resolution of $224\times 171$ pixels and frame rate of $45$ fps. Within the
training set, each participants performed $6$ normal driving and $8$ anomalous
driving video recordings with the same time duration. In total, there are
around $550$ minutes recording for normal driving and $100$ minutes recording
of anomalous driving. Within the test set, each participant recorded $6$ video
recordings that last around $3.5$ minutes. Anomalous actions were contained
randomly within those test video clips. In total, the test set contains around
$88$ minutes of normal driving and $45$ minutes of anomalous driving. In
addition to samples from $8$ anomalous driving behaviours in the training set,
the testing set contained $16$ new distracted actions (see Table 1). The
anomalous actions in the DAD dataset include multiple criteria, such as head
and body movements (reaching behind, talking with passengers), and hand
interactions (messaging, drinking). By collecting a large amount of videos
containing a wider amount of anomalous driving actions, the DAD dataset
represents the real-life situation more comprehensively and is suitable for
learning deep learning models. Figures 1 and 2 show sample normal and
anomalous driving actions captured from the depth and IR cameras from top and
front views.

(a)

(b)

(c)

(d)

Figure 1: Normal driving behaviour frame captured from different views and
modalities. (a) Front depth, (b) Front IR, (c) Top depth, (d) Top IR.

(a)

(b)

(c)

(d)

Figure 2: Anomalous driving behaviour frame (adjusting radio) captured from
different views and modalities. (a) Front depth, (b) Front IR, (c) Top depth,
(d) Top IR.

## Modified Supervised Contrastive Learning

The network architecture for supervised CL framework consists of the following
three components:

  * •

Encoding layers, $f_{\theta}$(.) – A 3D-CNN that extracts vector
representations ($h_{i}$) of input video clips ($x_{i}$),
$h_{i}=f_{\theta}(x_{i})$.

  * •

Projection layers, $g_{\beta}$(.) – It comprises of fully connected layers to
transform latent representation from the encoding layers to another latent
representation, s.t. $v_{i}=g_{\beta}(h_{i})$. The contrastive loss is defined
on the l2-normalized versions of this representations.

  * •

Contrastive Loss – It is calculated from the similarity between vector
representations of anchor and normal video clips (normal pair) and between
anchor and anomalous video clips (anomalous pair). The anchor is taken as the
clip from normal driving class.

In consecutive training iterations, each mini-batch contains $K$ normal and
$M$ anomalous video clips. The encoding and projection layers create visual
representations, $v_{ni}$, and $v_{ai}$, $i\in\\{1,\ldots,K+M\\}$ from normal,
and anomalous training clips, respectively. Each mini-batch contains $K(K-1)$
normal pairs and $KM$ anomalous pairs, which are input to the contrastive loss
function. The contrastive loss calculates the similarity in normal and
anomalous pairs by calculating the dot product between the latent
representations $(v_{ni}^{T}v_{nj}$ and $v_{ni}^{T}v_{am}$, scaled by a
temperature parameter $\tau$ and exponential function). The numerator
comprises of the similarity of normal pair and the denominator contains the
similarity in normal pairs and an aggregate of the similarities off all the
anomalous pairs. The minimization of the above objective function results in
joint training of the encoding and projection layers s.t. maximize the
similarities between normal samples and minimize the similarities between
normal and anomalous samples. However, the summation of similarities in
anomalous pairs could be problematic in cases where anomalous samples are
similar to normal samples, either due to nature of the data, mislabelling or
noise. In such cases, the overall sum of the negative pairs could become
larger, which is undesirable for the task and makes optimization harder. In
order to overcome this issue, we propose to scale the sum of anomalous pairs
s.t. the overall similarity of aggregated anomalous pairs remains smaller
(modification #1). Scaling the summation of anomalous pairs with a very small
number could lead to learning trivial representation. Therefore, we propose to
use the average of summation of negative pairs (see Equation 1). Other
possibilities for scaling could be explored by treating it as a hyper
parameter.

$\mathcal{L}_{ij}=-log\frac{exp(v_{ni}^{T}v_{nj}/\tau)}{exp(v_{ni}^{T}v_{nj}/\tau)+\frac{1}{M}\sum_{m=1}^{M}exp(v_{ni}^{T}v_{am}/\tau)}$
(1)

${L}=\frac{1}{K(K-1)}\sum_{i=1}^{K}\sum_{j=1}^{K}\mathbbm{1}_{j\neq
i}\mathcal{L}_{ij}$ (2)

where $L$ is the loss function for the minibatch.

After training with contrastive loss, projection layers are normally discarded
for testing purposes (Chen et al. 2020; Kopuklu et al. 2021; Khosla et al.
2020). The rationale for this choice comes from the work of Chen et al. in the
context of self-supervised contrastive learning (Chen et al. 2020) that the
latent representation, $h_{i}$, learned at the encoding stage will be more
general and act as a pre-trained model for further classification task.
However, we argue that for the supervised learning task, the representations
learned at the projection layers ($v_{i}$) can be more useful by virtue of the
contrastive loss learnt at the projection layers. The representation learnt at
the projection layers aligns with the supervised learning task of keeping
normal and anomalous classes far apart (modification #2).

In the DAD dataset, entire anomalous clips were given that label. However,
after a closer inspection, we observed that anomalous driving clips may not
contain anomalous actions for the entire duration of the clip. Thus, two
members of the research team manually re-labelled only the ‘anomalous’ frames
in those clips and discarded rest of the frames (modification #3). While
manually re-labelling, we categorize a frame as normal driving behaviour if
the participant has both the hands on the wheel and if the angle between the
front facing direction and their current gaze is approximately smaller than
$60$ degrees, as determined by visual inspection. Any frame that does not meet
the above criteria is considered as anomalous driving behaviour. The normal
driving clips were not re-labelled to keep variations of typical driving
behaviours in those clips. This additional step is indeed specific to this
dataset; however, the intent was to improve the quality of the training data.
We expect manual labelling to perform better because there will be less
overlap with the normal driving in anomalous clips; thus preventing the model
from learning conflicting information.

During testing phase, the jointly trained network is used for obtaining latent
visual representation for each test sample. To calculate a score, a linear
classifier can be used on top of the frozen encoder and projection head
layers. However, this step is not necessary and the learned representations
can be directly used at the testing time for calculating a score. A template
for the normal driving, ($v_{n}$) (Kopuklu et al. 2021), can be calculated
from latent representations obtained at the projection head of all the normal
clips:

$v_{n}=\frac{1}{N}\sum_{i=1}^{N}\frac{g_{\beta}(x_{i})}{\lVert
g_{\beta}(x_{i})\rVert_{2}}$ (3)

For a test clip, $t_{i}$, cosine similarity can be calculated between its
latent representation at the projection head and the normal template as
follows:

$s_{i}=v_{n}^{T}\frac{g_{\beta}(t_{i})}{\lVert g_{\beta}(t_{i})\rVert_{2}}$
(4)

where $s_{i}$ is the similarity score that can be used for further analysis.
The area under the curve (AUC) of the Receiver Operating Characteristic (ROC)
and Precision-Recall (PR) curve are used as performance metric. Due to the
imbalance between normal and anomalous driving samples, AUC PR is a useful
metric to highlight the relevance of a positive result given the baseline
probabilities of a problem.

## Experimentation Details

### Data Processing

As the DAD dataset contains depth and IR modalities, video frames in each
modality were converted to grayscale (single channel) images and normalized to
the $[0,1]$ range. In the spatial domain, video frames were resized and center
cropped to $112\times 112$ pixels. In the temporal domain, non-overlapping
windows of length $32$ were extracted and then downsampled to $16$ frames. To
ensure a full window can be formed as needed, the last window generated from a
video gets padded with the final available frame.

During training, data augmentation in the form of random cropping, random
rotation, salt and pepper noise, and random horizontal flipping (top view
only) were applied to consecutive frames of training samples (Kopuklu et al.
2021). With the original labelling (before modification #3), $46950$ normal
and $8600$ anomalous training windows are available in the DAD dataset. After
modification #3, the number of normal training windows remains the same, but
$7878$ anomalous training windows. Thus the imbalance ratio in this dataset
with original and manual labelling is $5.45$ and $5.95$, respectively.

During testing, Kopuklu et al. (Kopuklu et al. 2021), created overlapping
windows of length $16$ on the original frame rate per second (fps), without
downsampling from $32$. As a result, their training and testing models will
not be comparable. We identified this mistake in their work, created non-
overlapping windows and downsampled from $32$fps to $16$fps. In total, we
generated $11268$ testing windows.

### Training Phase

The encoder is a ResNet-18 whose output consists of two 512-D tensors, one
unnormalized and the other one L2-normalized. The input to the projection
layers consists of unnormalized $512$-D tensors, and its output is normalized
$128$-D tensors ($512$-$256$-$128$). The models were trained for $250$ epochs
on a server with $64$ GB of RAM and an NVIDIA Tesla P100 PCIe $12$ GB GPU.
Stochastic gradient descent was used with an initial learning rate of $0.01$
that was reduced by a factor of $10$ every $100$ epochs. The temperature
parameter, $\tau$, was set to $0.1$. Each mini-batch contained $160$ windows
in total, with $10$ normal driving windows and $150$ anomalous driving
windows. All these settings were kept as same as Kopuklu et al. (Kopuklu et
al. 2021). The training samples in the DAD dataset were divided into two sets
of validation set ($20\%$ of samples) and training set ($80\%$ of samples)
with the same ratios of normal and anomalous samples. After every $10$
training epochs, the performance of the training models (the base encoder and
projection head) was validated on the validation set in terms of AUC ROC, and
the models (the base encoder and the projection head) achieving the highest
AUC were saved as the best models (or replaced the best models so far). The
saved best models after $250$ training epochs were considered as the final
trained models and were used for testing. It is to be noted that previous work
used the latent representation at $f_{\theta}$ during testing phase and
discard $g_{\beta}$. We present the results on using both the representations
learned at $f_{\theta}$ and $g_{\beta}$ and show that latter works better in
the supervised classification task.

### Testing Phase

As discussed earlier, Kopuklu et al. (Kopuklu et al. 2021) tested their models
on the original $32$ fps, whereas the models were trained on downsampled $16$
fps video data. They mentioned a scoring method that computes score from a
$16$ length window and slides it by one frame to give score to each middle
frame. Since they gave score to each frame, they operated in the original fps
and ignored that the training and testing models were incompatible and non-
comparable. We identified this problem, created non-overlapping windows,
downsampled the testing data from $32$ fps to $16$ fps, and computed cosine
similarity score for each of the non-overlapping windows w.r.t. to the normal
template (see Equation 4) and calculated AUC ROC and PR from them. For the
testing phase, each batch contains $25$ windows, with each window containing
$16$ frames.

Different modalities and views are also combined to verify if their fusion
works better for detecting anomalous driving behaviours. In these cases,
$f_{\theta}$ and $g_{\beta}$ representations are learned for each modality.
Then during testing, corresponding windows are passed through $f_{\theta}$ and
$g_{\beta}$ (from trained models) and individual similarity scores are
obtained. The average of these individual modality scores is taken as a score
for calculating AUC ROC and PR.

One separate model was trained for each modality and view, i.e. each model for
top and front view, IR and depth modality (four models). These models were
tested for nine modality combinations – four individual modalities and views
(Top(D), Top(IR), Front(D), Front(IR)), two combinations for top and front for
depth+IR (Top(DIR) and Front(DIR)), two combinations for depth and IR for
top+front (Fusion(D) and Fusion(IR)), and one for combining all four top,
front, IR and depth (Fusion(DIR)), where D and IR represents depth and
infrared cameras.

## Results

Modality | Original Loss | Modified Loss
---|---|---
No Projection Head | Projection Head | No Projection Head | Projection Head
Original Labelling |  Manual Labelling |  Original Labelling |  Manual Labelling |  Original Labelling |  Manual Labelling |  Original Labelling |  Manual Labelling
Top (D) |  0.8713 |  0.8573 |  0.8876 |  0.8604 |  0.8983 |  0.9080 |  0.8722 |  0.9202
Top (IR) |  0.8456 |  0.8423 |  0.8406 |  0.8642 |  0.8118 |  0.8769 |  0.7952 |  0.8913
Top (DIR) |  0.8663 |  0.9007 |  0.8881 |  0.9021 |  0.8919 |  0.8969 |  0.8592 |  0.9273
Front (D) |  0.8503 |  0.9156 |  0.8577 |  0.9064 |  0.8759 |  0.8773 |  0.8461 |  0.8851
Front (IR) |  0.8367 |  0.8532 |  0.8324 |  0.8823 |  0.8854 |  0.8782 |  0.8448 |  0.9113
Front (DIR) |  0.8818 |  0.9302 |  0.8998 |  0.8945 |  0.8969 |  0.9250 |  0.8722 |  0.9268
Fusion (D) |  0.9229 |  0.9514 |  0.9082 |  0.9042 |  0.9449 |  0.9451 |  0.9154 |  0.9517
Fusion (IR) |  0.8946 |  0.8937 |  0.8941 |  0.9289 |  0.9240 |  0.9337 |  0.8932 |  0.9552
Fusion (DIR) |  0.9342 |  0.9536 |  0.9248 |  0.9451 |  0.9599 |  0.9619 |  0.9366 |  0.9738
Table 2: AUC of ROC curve results corresponding to different combinations of losses (original or modified), projection head (present or absent), and labelling (original or manual). The modality abbreviations used are: Depth (D), IR (IR), Depth and IR (DIR). Modality | Original Loss | Modified Loss
---|---|---
No Projection Head | Projection Head | No Projection Head | Projection Head
Original Labelling |  Manual Labelling |  Original Labelling |  Manual Labelling |  Original Labelling |  Manual Labelling |  Original Labelling |  Manual Labelling
Top (D) |  0.8925 |  0.8734 |  0.9097 |  0.8994 |  0.9253 |  0.9152 |  0.8979 |  0.9243
Top (IR) |  0.8764 |  0.9152 |  0.8985 |  0.9062 |  0.9226 |  0.9221 |  0.8512 |  0.9022
Top (DIR) |  0.8910 |  0.9154 |  0.9005 |  0.9019 |  0.9162 |  0.9115 |  0.8947 |  0.9182
Front (D) |  0.8900 |  0.9127 |  0.8922 |  0.9373 |  0.9274 |  0.9257 |  0.8952 |  0.9176
Front (IR) |  0.8997 |  0.9016 |  0.9090 |  0.9457 |  0.9232 |  0.9542 |  0.8931 |  0.9024
Front (DIR) |  0.9002 |  0.9218 |  0.9123 |  0.9304 |  0.9358 |  0.9607 |  0.9138 |  0.9284
Fusion (D) |  0.9321 |  0.9400 |  0.9312 |  0.9502 |  0.9498 |  0.9611 |  0.9439 |  0.9497
Fusion (IR) |  0.9020 |  0.9547 |  0.9442 |  0.9518 |  0.9254 |  0.9728 |  0.9209 |  0.9334
Fusion (DIR) |  0.9618 |  0.9578 |  0.9695 |  0.9544 |  0.9772 |  0.9721 |  0.9427 |  0.9591
Table 3: AUC of PR curve results corresponding to different combinations of losses (original or modified), projection head (present or absent), and labelling (original or manual). The modality abbreviations used are: Depth (D), IR (IR), Depth and IR (DIR). |  OL-NP-NL |  OL-NP-ML |  OL-PH-NL |  OL-PH-ML |  ML-NP-NL |  ML-NP-ML |  ML-PH-NL |  ML-PH-ML
---|---|---|---|---|---|---|---|---
OL-NP-NL |  n/a |  0.743 |  1.000 |  0.743 |  0.743 |  0.079 |  1.000 |  0.001
OL-NP-ML |  0.743 |  n/a |  0.743 |  1.000 |  1.000 |  1.000 |  0.150 |  0.210
OL-PH-NL |  1.000 |  0.743 |  n/a |  0.743 |  0.743 |  0.079 |  1.000 |  0.001
OL-PH-ML |  0.743 |  1.000 |  0.743 |  n/a |  1.000 |  1.000 |  0.210 |  0.150
ML-NP-NL |  0.743 |  1.000 |  0.743 |  1.000 |  n/a |  1.000 |  0.210 |  0.150
ML-NP-ML |  0.079 |  1.000 |  0.079 |  1.000 |  1.000 |  n/a |  0.008 |  1.000
ML-PH-NL |  1.000 |  0.150 |  1.000 |  0.210 |  0.210 |  0.008 |  n/a |  0.000
ML-PH-ML |  0.001 |  0.210 |  0.001 |  0.150 |  0.150 |  1.000 |  0.000 |  n/a
Table 4: Friedman’s post-hoc test with Bergmann and Hommel’s correction for different combinations of loss, projection head, and labelling across all the camera modalities. The test was performed on the AUC of ROC curve results. The abbreviations are: OL - Original Loss, NP - No Projection Head, NL - no labelling (use original video labels as is), ML - Modified Loss, PH - Projection Head, ML - Manual labelling. |  OL-NP-NL |  OL-NP-ML |  OL-PH-NL |  OL-PH-ML |  ML-NP-NL |  ML-NP-ML |  ML-PH-NL |  ML-PH-ML
---|---|---|---|---|---|---|---|---
OL-NP-NL |  n/a |  0.652 |  1.000 |  0.033 |  0.001 |  0.000 |  1.000 |  0.068
OL-NP-ML |  0.652 |  n/a |  1.000 |  1.000 |  0.242 |  0.161 |  0.917 |  1.000
OL-PH-NL |  1.000 |  1.000 |  n/a |  0.815 |  0.078 |  0.046 |  1.000 |  1.000
OL-PH-ML |  0.033 |  1.000 |  0.815 |  n/a |  1.000 |  1.000 |  0.068 |  1.000
ML-NP-NL |  0.001 |  0.242 |  0.078 |  1.000 |  n/a |  1.000 |  0.002 |  1.000
ML-NP-ML |  0.000 |  0.161 |  0.046 |  1.000 |  1.000 |  n/a |  0.001 |  0.917
ML-PH-NL |  1.000 |  0.917 |  1.000 |  0.068 |  0.002 |  0.001 |  n/a |  0.161
ML-PH-ML |  0.068 |  1.000 |  1.000 |  1.000 |  1.000 |  0.917 |  0.161 |  n/a
Table 5: Friedman’s post-hoc test with Bergmann and Hommel’s correction for
different combinations of loss, projection head, and labelling across all the
camera modalities. The test was performed on the AUC of PR curve results. The
abbreviations are: OL - Original Loss, NP - No Projection Head, NL - no
labelling (use original video labels as is), ML - Modified Loss, PH -
Projection Head, ML - Manual labelling.

Tables 2 and 3 show the AUC ROC and AUC PR results. The rows in the tables
show the camera modality and the columns show different combinations of loss
(original or modified), projection head (absent or present), and labelling
(original or manual). The highlighted cells indicate the highest AUC for that
modality for a given combination of loss, projection head, and labelling. The
baseline results for each modality corresponds to the combination of original
loss, no projection head, and original labelling. We observe that in terms of
AUC ROC, out of nine modalities, the modified loss with the presence of
projection head and manual labelling performed better in seven of them
(improved performance from $3.12\%$ for Fusion(D) to $8.91\%$ for Front(IR)).
In two cases, original loss with no projection head and manual labelling
performed better (Front (D) and Front (DIR)). However, all the nine best
performing models had manual labelling, further highlighting the role of good
quality of labels in supervised classification problems. Some other findings
are as follows:

  * •

Fusion of camera modalities (D and IR) for each top and front view improved
the performance in comparison to individual camera and modality type
irrespective of loss type and its various combinations.

  * •

Major performance improvement was achieved when top and front views were fused
for each of the depth and IR cameras (Fusion (D) AUC=$0.9517$, Fusion (IR)
AUC=$0.9552$) with modified loss, projection head and manual labelling.
Combining different views can help in capturing complimentary information,
which improved the performance.

  * •

The best performance was achieved when both the camera modalities and views
were combined (AUC=$0.9738$). This shows that apart from capturing top and
front views, combining depth and IR cameras further boost the performance in
detecting anomalous driving behaviours.

In terms of AUC PR (see Table 3), eight out of nine highest numbers are for
modified loss and distributed across different combinations of projection head
and manual labelling. This further ascertains that modifying the loss function
is important in a supervised contrastive learning task, and adding projection
head and manual labelling further improves the performance.

### Statistical Testing

To understand the overall performance improvement of the three modifications,
we performed the Friedman’s post-hoc test with Bergmann and Hommel’s
correction (Calvo and Santafé Rodrigo 2016; Calvo 2021; do Prado Lima 2018).
This test computes the p-value for each pair of algorithms corrected for
multiple testing using Bergman and Hommel’s correction. Table 4 shows the (AUC
ROC) comparison between different combinations of loss, projection head, and
labelling across nine modalities. The significant differences (p-value
$<0.05$) are shown in bold and highlighted cells. From Table 4, we infer the
following:

  * •

The performance of modified loss with the presence of the projection head and
manual labelling is statistically different than original loss and original
labelling (irrespective of projection head) and modified loss with projection
head and original labelling.

  * •

The modified loss with manual labelling (irrespective of projection head) is
statistically different than modified loss with projection head and original
labelling.

  * •

Within original loss, each combination performs statistically similar.

Table 5 shows the (AUC PR) comparison between different combinations of loss,
projection head, and labelling across nine modalities, we infer the following:

  * •

Modified loss with no projection head (irrespective of labelling) is
statistically different than modified loss with projection head and original
labelling and the baseline (original loss with no projection head and original
labelling).

  * •

Modified loss with no projection and manual labelling is statistically
different than original loss with projection head, and original labelling.

  * •

Original loss with projection head and manual labelling is statistically
different than original loss with no projection head and original labelling
(baseline) and statistically similar to other original loss combinations.

Therefore, combining the results from Tables 4 and 5, we conclude that within
a supervised classification framework, modified contrastive loss with manual
labelling is certainly superior and adding the projection head can also help
in detecting seen and unseen anomalous driving behaviours.

## Conclusions

In this paper, we presented modifications to the supervised contrastive loss
along with considerations for including projection head and refining labels to
detect anomalous driving behaviours. We support our results through rigorous
statistical testing. We found that combining video clips from different views
improves the performance drastically, with the best results achieved by
combining top and frontal views and depth and IR cameras. In the future, we
plan to extend this approach by collecting data with actual drivers on the
road.

## References

  * Borghesi et al. (2019) Borghesi, A.; Bartolini, A.; Lombardi, M.; Milano, M.; and Benini, L. 2019.  Anomaly detection using autoencoders in high performance computing systems.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 9428–9433.
  * Borghi et al. (2017) Borghi, G.; Venturelli, M.; Vezzani, R.; and Cucchiara, R. 2017.  Poseidon: Face-from-depth for driver pose estimation.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 5494–5503. IEEE.
  * Cai et al. (2021) Cai, R.; Zhang, H.; Liu, W.; Gao, S.; and Hao, Z. 2021.  Appearance-Motion Memory Consistency Network for Video Anomaly Detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 938–946.
  * Calvo (2021) Calvo, B. 2021.  scmamp: Statistical Comparison of Multiple Algorithms in Multiple Problems.  https://github.com/b0rxa/scmamp.  [Online; accessed 2021-08-27].
  * Calvo and Santafé Rodrigo (2016) Calvo, B.; and Santafé Rodrigo, G. 2016.  scmamp: Statistical comparison of multiple algorithms in multiple problems.  _The R Journal, Vol. 8/1, Aug. 2016_.
  * Chen et al. (2020) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , 1597–1607. PMLR.
  * Das, Ohn-Bar, and Trivedi (2015) Das, N.; Ohn-Bar, E.; and Trivedi, M. M. 2015.  On performance evaluation of driver hand detection algorithms: Challenges, dataset, and metrics.  In _2015 IEEE 18th international conference on intelligent transportation systems_ , 2953–2958. IEEE.
  * Diaz-Chito, Hernández-Sabaté, and López (2016) Diaz-Chito, K.; Hernández-Sabaté, A.; and López, A. M. 2016.  A reduced feature set for driver head pose estimation.  _Applied Soft Computing_ , 45: 98–107.
  * do Prado Lima (2018) do Prado Lima, J. A. 2018.  Friedman test with post-hoc using Bergmann-Hommel procedure.  https://gist.github.com/jacksonpradolima/b40aa19325787898fc5ddc3155752d94.  [Online; accessed 2021-08-27].
  * Girdhar et al. (2019) Girdhar, R.; Carreira, J.; Doersch, C.; and Zisserman, A. 2019.  Video action transformer network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 244–253.
  * Hadsell, Chopra, and LeCun (2006) Hadsell, R.; Chopra, S.; and LeCun, Y. 2006.  Dimensionality reduction by learning an invariant mapping.  In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , volume 2, 1735–1742. IEEE.
  * Ji et al. (2012) Ji, S.; Xu, W.; Yang, M.; and Yu, K. 2012.  3D convolutional neural networks for human action recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(1): 221–231.
  * Khan and Madden (2014) Khan, S. S.; and Madden, M. G. 2014.  One-class classification: taxonomy of study and review of techniques.  _The Knowledge Engineering Review_ , 29(3): 345–374.
  * Khan, Nogas, and Mihailidis (2021) Khan, S. S.; Nogas, J.; and Mihailidis, A. 2021.  Spatio-temporal adversarial learning for detecting unseen falls.  _Pattern Analysis and Applications_ , 24(1): 381–391.
  * Khosla et al. (2020) Khosla, P.; Teterwak, P.; Wang, C.; Sarna, A.; Tian, Y.; Isola, P.; Maschinot, A.; Liu, C.; and Krishnan, D. 2020.  Supervised contrastive learning.  _arXiv preprint arXiv:2004.11362_.
  * Köpüklü et al. (2020) Köpüklü, O.; Ledwon, T.; Rong, Y.; Kose, N.; and Rigoll, G. 2020.  Drivermhg: A multi-modal dataset for dynamic recognition of driver micro hand gestures and a real-time recognition framework.  In _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)_ , 77–84. IEEE.
  * Kopuklu et al. (2021) Kopuklu, O.; Zheng, J.; Xu, H.; and Rigoll, G. 2021.  Driver anomaly detection: A dataset and contrastive learning approach.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 91–100.
  * Martin et al. (2019) Martin, M.; Roitberg, A.; Haurilet, M.; Horne, M.; Reiß, S.; Voit, M.; and Stiefelhagen, R. 2019.  Drive&act: A multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2801–2810.
  * Mehta et al. (2021) Mehta, V.; Dhall, A.; Pal, S.; and Khan, S. S. 2021.  Motion and region aware adversarial learning for fall detection with thermal imaging.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , 6321–6328. IEEE.
  * Nogas, Khan, and Mihailidis (2020) Nogas, J.; Khan, S. S.; and Mihailidis, A. 2020.  Deepfall: Non-invasive fall detection with deep spatio-temporal convolutional autoencoders.  _Journal of Healthcare Informatics Research_ , 4(1): 50–70.
  * of Ontario (2021) of Ontario, G. 2021.  Distracted driving.
  * Ohn-Bar, Martin, and Trivedi (2013) Ohn-Bar, E.; Martin, S.; and Trivedi, M. 2013.  Driver hand activity analysis in naturalistic driving studies: challenges, algorithms, and experimental studies.  _Journal of Electronic Imaging_ , 22(4): 041119.
  * Ortega et al. (2020) Ortega, J. D.; Kose, N.; Cañas, P.; Chao, M.-A.; Unnervik, A.; Nieto, M.; Otaegui, O.; and Salgado, L. 2020.  DMD: A Large-Scale Multi-modal Driver Monitoring Dataset for Attention and Alertness Analysis.  _Lecture Notes in Computer Science_ , 387–405.
  * Pietrasik (2021) Pietrasik, T. 2021.  Road traffic injuries.
  * Roth and Gavrila (2019) Roth, M.; and Gavrila, D. M. 2019.  Dd-pose-a large-scale driver head pose benchmark.  In _2019 IEEE Intelligent Vehicles Symposium (IV)_ , 927–934. IEEE.
  * Schwarz et al. (2017) Schwarz, A.; Haurilet, M.; Martinez, M.; and Stiefelhagen, R. 2017.  Driveahead-a large-scale driver head pose dataset.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 1–10.
  * Vu et al. (2019) Vu, H.; Nguyen, T. D.; Le, T.; Luo, W.; and Phung, D. 2019.  Robust anomaly detection in videos using multilevel representations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 5216–5223.
  * Wang, Zou, and Zhang (2020) Wang, Z.; Zou, Y.; and Zhang, Z. 2020.  Cluster attention contrast for video anomaly detection.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , 2463–2471.
  * Wu et al. (2015) Wu, Z.; Wang, X.; Jiang, Y.-G.; Ye, H.; and Xue, X. 2015.  Modeling spatial-temporal clues in a hybrid deep learning framework for video classification.  In _Proceedings of the 23rd ACM international conference on Multimedia_ , 461–470.
  * Zheng et al. (2021) Zheng, Y.; Jin, M.; Liu, Y.; Chi, L.; Phan, K. T.; and Chen, Y.-P. P. 2021.  Generative and Contrastive Self-Supervised Learning for Graph Anomaly Detection.  arXiv:2108.09896.